Settings

Every Timelake feature has a setting that turns it on or off or chooses how it behaves. Settings are declared once, in the engine, and can be set at up to four levels. When a setting is set at more than one level, the most specific one wins:

Create: fixed when the table is created, by its options (--time-column, --series-key, --chunk, --no-dedup, --no-ingest-time on the CLI; time_column=, series_key=, chunk=, dedup=, ingest_time= in Python).
Table: a table property under the timelake. prefix, stored with the table and used by every reader and writer. Set it at creation (--prop, properties=) or later (timelake alter, Table.set_properties). Other engines ignore these properties.
Session: for one SQL session, overriding the table's value: SET timelake.read_dedup = false in SQL, --set read_dedup=false on the CLI, Lake(..., settings={...}) in Python (any spelling: read-dedup, timelake.read_dedup). SHOW timelake.read_dedup shows the session's value.
Call: for one append or maintenance run, through its options (AppendOptions, MaintainOptions, or the method's keyword arguments in Python).

A setting has one name. Table properties spell it with dashes (timelake.read-dedup); SQL needs identifiers, so it uses underscores (timelake.read_dedup). Both spellings, and the bare name, name the same setting. An unknown name, a bad value, or a setting set at a level it doesn't allow is an error. The table below gives the bare names; format-version is set by Timelake itself (at creation and by timelake upgrade). Tables written by a newer Timelake may carry settings this one doesn't know; reading ignores them. Storage options, which say how to reach the bucket and which requests to use, are given when a lake is opened; they are on the Storage page.

timelake settings                    # every setting: default, levels, description
timelake settings iot.readings       # the table's values; defaults in parentheses
timelake alter iot.readings --set read-dedup=false --set retention=90d
timelake alter iot.readings --unset retention    # back to the default

table.set_properties({"read_dedup": "false", "retention": None})
lake = timelake.Lake("s3://bucket/lake", settings={"read_dedup": "false"})

Setting	Default	Set at	Meaning
`time-column` column, since 0.1.0	none	create	The time column. Its presence makes the table a time-series table.
`series-key` columns, since 0.1.0	none	create	Comma-separated columns that identify a series.
`chunk-interval` hour, day, or month, since 0.1.0	day for time-series tables	create	Partition size: how much time one partition covers.
`dedup` boolean, since 0.1.0	true with a series key, else false	create	Rows with the same (series, time) count once; the latest write wins. Applied within an append, by the Timelake reader, and by compaction. Fixed at creation: files written with it off may hold copies, which the reader would then miss.
`ingest-time-column` column, since 0.5.0	_ingest_ts for time-series tables	create	The column holding each row's insertion time (`ingest_time=False` or `--no-ingest-time` leaves it out). Each append fills it with the time the append started, unless the input brings its own values. Among rows with the same (series, time), the latest insertion wins, then the later commit.
`retention` duration, since 0.1.0	forever	create table	Maintenance drops partitions that end before now minus this duration, e.g. 90d.
`auto-maintain` boolean, since 0.1.0	`true`	table call	Run size-tiered compaction, retention, and snapshot expiry inline after appends. One append can override it (`maintain=` in Python, `--no-maintain` on the CLI).
`compact-min-files` count, since 0.1.0	`8`	table	Inline compaction merges a size tier of a partition once it has this many files.
`compact-cold-after` duration or off, since 0.5.0	`1h`	table call	Inline maintenance rewrites a partition whole (sorted, duplicates removed) once the table holds data newer than the partition's end by this much and the partition has two small files, whether the append touched it or not. `off` leaves it to the size tiers.
`read-dedup` boolean, since 0.5.0	as dedup	table session	The Timelake reader returns only the latest write of each (series, time), before compaction has removed the older copies. Has no effect on tables with dedup off.
`read-dedup-strategy` merge or keys-first, since 0.5.0	`merge`	table session	How the reader removes copies: `merge` merges the files that may share keys in key order; `keys-first` lists the older copies from the key columns, then reads the files like any scan and leaves them out (slower on the industrial IoT benchmark).
`filter-pushdown` auto, true, or false, since 0.7.0	`auto`	table session	The Timelake reader applies filters while decoding Parquet pages, cheapest first, and decodes the other columns only for the rows that pass. Less decoding when filters keep few rows, but each row group's filter columns and other columns are fetched one after the other. `auto` is on for local files and off on object storage, where first queries wait for the extra round trip; `true` suits repeat queries served from the cache.
`series-order` auto, segments, or merge, since 0.8.0	`auto`	table session	How the signal aggregates (`time_weighted_mean`, `integral`, `counter_diff`, `counter_rate`) get each series' readings in time order. They sum runs of readings in order and join them; runs that overlap in time can't be joined. Files are read in time order, each partition a stretch of time. `auto`: files that overlap another on the series and time, or aren't sorted, are merged in order first (as dedup on read does), with the other files of the time they span. `segments`: only copies are merged (with dedup on read); overlaps are an error. `merge`: every file is merged.
`window-series-order` auto, true, or false, since 0.8.0	`auto`	table session	A window over each series in time order (`lead(…) OVER (PARTITION BY <series key> ORDER BY time)`) reads the series merged in order, already split by series, so the reader neither repartitions nor sorts the rows. Faster on one large file (a day of readings: 18% less), slower over many (a week: 50% more), with less memory either way. `auto`: when the files read are in one time partition.
`shared-signal-state` boolean, since 0.8.0	`true`	session	Signal aggregates of the same readings in one query (`time_weighted_mean`, `integral`, `counter_diff`, `counter_rate` of one column, by one method) compute their runs of readings once and each result from them, instead of each computing its own.
`newest-first` boolean, since 0.7.0	`true`	table session	A query sorted by the time column, newest first, with a limit (`ORDER BY time DESC LIMIT k`) reads the newest time partition first and older ones only while they can still hold one of the k newest rows. The same for the k newest buckets of a `GROUP BY time_bucket(…)` sorted by the bucket, newest first.
`in-list-pruning` boolean, since 0.7.0	`true`	table session	The Timelake reader skips row groups and pages whose range of a column holds none of the values of a `column IN (…)` filter with more than 20 values, which DataFusion doesn't prune with: lists in the query, and, on scans that don't filter rows while decoding (object storage by default), the key lists of a join's runtime filter.
`target-file-size-bytes` bytes, since 0.1.0	`268435456`	table	Size at which compaction output and writes roll to a new file (256 MiB).
`row-group-rows` count, since 0.1.0	`1048576`	table	Parquet row group size. Page indexes keep single-series reads selective inside large row groups.
`snapshot-retention` duration, since 0.1.0	`1h`	table	Expire snapshots older than this, keeping at least `min-snapshots` and at most `max-snapshots`.
`min-snapshots` count, since 0.1.0	`5`	table	Snapshots always kept, whatever their age.
`max-snapshots` count, since 0.1.0	`100`	table	Snapshots kept at most: older ones expire even if younger than `snapshot-retention`.
`head-format` codec name or none, since 0.3.0	`arrow`	table	Format of the head object (`metadata/timelake-head`): `arrow`, `json`, a registered codec, or `none` for no head.
`head-inline-bytes` bytes, since 0.3.0	`1048576`	table	Room in the head for the last-value index, then the Parquet footers of the newest files (1 MiB). What doesn't fit is read separately.
`keep-snapshots-until` milliseconds since the epoch, since 0.3.0	none	table	No snapshot expires before this time. `timelake upgrade` sets it so the upgrade can be rolled back.
`rollup-routing` boolean, since 0.5.0	`true`	table session	The Timelake reader answers a `GROUP BY time_bucket(…)` query on this table from a rollup that matches it. Buckets changed since the rollup's last refresh, and partial buckets at the edges of the time range, still come from this table, so the answer is the same either way. A table read without dedup (`read-dedup` off) isn't routed, since a rollup counts each reading once.
`rollup-refresh-interval` duration or off, since 0.5.0	`1m`	table	On a rollup: inline maintenance after an append to its raw table refreshes it once its last refresh is this old. `off` leaves it to `timelake maintain` and `REFRESH MATERIALIZED ROLLUP`.
`float-encoding` dictionary or byte-stream-split, since 0.8.0	`dictionary`	table	How data files encode float columns. `dictionary`: a dictionary of each column chunk's values, falling back to byte-stream-split once it outgrows its page. Sensor readings repeat values: a third of the size on the IIoT benchmark and up to twice as fast to scan on TSBS DevOps; random doubles cost 5% more space and 6% more scan time. `byte-stream-split`: byte-stream-split only. Applies to files written from then on.
`data-page-rows` count, since 0.6.0	20,000; 8,192 when written from WebAssembly	table	Rows per Parquet data page in the files written. Smaller pages make compaction from a Worker use less memory (it holds a page per column of each file it merges: 8,192 rows peaked at 77 MiB merging 24 TSBS files, 20,000 at 123 MiB) and page pruning finer, at a few percent more bytes (TSBS: +3.6%) and slower scans below about 8,000 rows.
`version-check-interval` duration, since 0.6.0	`1s`	table session	The Timelake reader (the engine, and the TypeScript client unless given `cacheMs`) uses a table version it loaded or checked this long before checking for a newer one, so queries in that time send no request for it and can miss commits by other processes. `0s` checks on every query. This process's own commits are seen at once, and writers always check.
`rollups` table names, since 0.5.0	none	Timelake	On a raw table: its rollups, comma-separated. `CREATE` and `DROP MATERIALIZED ROLLUP` keep it.
`rollup-source` table name, since 0.5.0	none	Timelake	On a rollup: the raw table it summarises.
`rollup-bucket` duration, since 0.5.0	none	Timelake	On a rollup: the width of its time buckets.
`rollup-aggregates` JSON, since 0.5.0	none	Timelake	On a rollup: its aggregates, each an output column, a function, and the column it reads.
`format-version` semantic version, since 0.3.0	the current format for new tables; 0.1.0 if absent	Timelake	The table format. Timelake refuses tables of a newer format it can't handle; `timelake upgrade` brings a table to the current format.

Durations are a number and a unit: s, m (minutes), h, d, w or y, or the long forms (90 days, 2 weeks). Examples: 30m, 36h, 90d, 1y.

Filter pushdown

With filter-pushdown on, a scan decodes the filter columns of a row group, evaluates the filters, and decodes the other columns only for the rows that pass. On local disk (TSBS, standard scale, repeat queries) it cut the geomean by 4–5% and selective group-bys by 15–40%, and made queries that keep most rows up to 20% slower. On object storage it costs a request round trip per row group: with 20 ms per request, first queries took 12–17% longer, while repeat queries served from the block cache were 5–8% faster. So auto turns it on for local files and off on object storage. Timelake's scans ignore DataFusion's own datafusion.execution.parquet.pushdown_filters; scans that remove duplicates use pushdown either way.

Iceberg properties set at creation

write.format.default = parquet and write.parquet.compression-codec = zstd. Timelake writes data files with the codec this names, at write.parquet.compression-level where the codec has levels (ZSTD 3 by default). Table creation and timelake alter reject an unknown codec, or a level the codec doesn't have such as high or a negative one for gzip; set by another engine, such a value fails writes, never reads. Codecs: zstd, lz4, snappy, gzip, brotli, or uncompressed (from JavaScript: zstd and snappy). Set it with timelake alter tsbs.readings --set write.parquet.compression-codec=lz4. On TSBS IoT, repeat queries on local disk took 11.5 ms with ZSTD 3, 11.1 ms with ZSTD 1, 12.3 ms with LZ4 and 10.8 ms with Snappy, while the data took 200, 204, 242 and 277 MB: reading doesn't wait on decompression, and on object storage bytes cost time and money, so ZSTD 3 stays the default.
write.metadata.previous-versions-max = 20: keeps metadata.json, which every query reads, small. Older metadata files are deleted by snapshot expiry.

Schema

Arrow types are mapped to the closest lossless Iceberg type when a table is created:

8- and 16-bit integers become int; UInt32 becomes long; UInt64 is rejected.
Timestamps of any unit are stored in microseconds; time-zone-aware timestamps become timestamptz (UTC).
Dictionary-encoded columns use their value type; large and view string and binary types become string and binary.
The time column is NOT NULL, and a text time column is stored as a UTC timestamp.

Schema evolution isn’t supported yet: the schema is fixed when the table is created.

Rollups Storage & configuration