Summary
The first version: open Iceberg tables on any object store with time-series layout and maintenance built in, and a reader with a last-value index, faster than naive S3 Tables or Lance for time series. Implemented 2026-10-12 with its exit criteria met, and released as 0.1.0.
Scope
Value proposition. "Better than naive S3 Tables or Lance for time series, with no Glue jobs." A team that today appends micro-batches to S3 Tables (Firehose, PyIceberg, Spark) and reads with DuckDB/Athena should get, by switching to Timelake v0:
- the same open Iceberg tables (plain DuckDB still reads them),
- on any object store (S3, R2, GCS, Azure, RustFS/MinIO, local disk),
- with time-series layout and maintenance built in (sort, compaction, dedup, retention, snapshot expiry), run inline by the writer or as one stateless
timelake maintaincommand, - and much faster time-series reads through the Timelake reader, including a last-value index.
In scope (v0):
| Area | v0 deliverable |
|---|---|
| Catalog | Object-store-only catalog: vN.metadata.json committed with put-if-absent (conditional write), version-hint.text for Hadoop-style readers (DuckDB iceberg_scan). No server. |
| Contract | time_column, series_key, chunk_interval (hour/day/month), dedup on/off, retention, all as timelake.* table properties. Tables without a time column (dimensions, e.g. device metadata) are allowed. |
| Writer | Append Arrow batches → align to table schema → sort by (series_key…, time) → split by time partition → Parquet (ZSTD, page index, time-series encodings). Idempotent batch IDs. |
| Commit layer (D13) | Own append / replace / delete commits on iceberg-rust public types, manifest merging, optimistic retry. Iceberg v2 by default. |
| Last-value index | A small Parquet sidecar per snapshot with the latest row per series (pointer in the snapshot summary). Updated on every append. |
| Maintenance | Compaction (merge, re-sort, dedup, latest write wins), retention (drop expired partitions), snapshot expiry + unreferenced-file cleanup. Inline auto-maintenance after appends (thresholds) or timelake maintain. |
| Reader | DataFusion TableProvider: file pruning on Iceberg column bounds, Parquet row-group/page pruning, time_bucket, and automatic rewrite of latest-row queries (ROW_NUMBER() … = 1, ORDER BY time DESC LIMIT 1) onto the last-value index. |
| Interfaces | timelake CLI (create, append/import Parquet+CSV, sql, latest, maintain, info, duckdb) and a Python package (PyO3): Lake, create_table, append, sql, latest, maintain with Arrow in and out. |
| Benchmark | Same ingest stream (TSBS IoT, 5-minute micro-batches) into naive Iceberg (PyIceberg appends, the S3 Tables write pattern), Iceberg after binpack compaction (S3 Tables' default maintenance), Lance (append, then compacted), and Timelake. Then TSBS IoT + single-device queries, local and on RustFS with 20 ms latency. |
Out of scope (v0): REST catalog server, hot tier / WAL, rollups, deletes/GDPR, backfill branches, TimescaleDB import, timelake-ingest service and rules, DuckDB extension, Vortex, schema evolution beyond add-column.
Exit criteria: On RustFS + 20 ms, Timelake's reader beats naive Iceberg + DuckDB and Lance on the TSBS IoT and single-device suites (geomean, first run), latest-value queries take tens of milliseconds, plain DuckDB reads every Timelake table with identical results, and no maintenance job outside Timelake is needed.
Results
Implemented 2026-10-12; exit criteria met (bench/results/V0.md).
- S3 (RustFS + 20 ms), first runs:
- Right after streaming ingest: 3.7× faster than Lance and 16× faster than naive Iceberg + DuckDB on TSBS IoT; 7.8–9× faster on single-truck queries.
- After each system's maintenance: 1.2–1.5× faster on TSBS IoT, 2.2–2.7× on single-truck queries.
- Local disk: 2.7–3.8× faster on TSBS IoT, 7.7–10× on single-truck queries.
- Latest-value queries: 12 requests instead of 1,700–3,600.
- Ingest on S3: 2× faster than Lance and 3.7× faster than PyIceberg, including inline compaction.
- Live size: 228 MB, vs 262 MB (Iceberg) and 449 MB (Lance).
- Compatibility: plain DuckDB reads all tables with identical results.
Reader lessons, now defaults:
- Load all of a query's tables concurrently before planning.
- Read each Parquet footer once.
- Use more scan partitions on object storage.
- Write 1M-row row groups (page indexes keep single-series reads selective).
- Declare the sort order only when there are enough files to keep scans parallel.
Open from v0:
- A reader data cache: DuckDB still won some repeat runs on its cached data. Built in 0.5.0 as the block cache (spec 22).
- Plain DuckDB on Timelake files is slower than on PyArrow-written files on S3 (0.52 vs 0.42 s), which needs investigation (#134).
- The REST catalog (spec 32).
History
- 2026-10-12: implemented, exit criteria met (bench/results/V0.md)
- 2026-10-19: was spec 31; a milestone on the roadmap since the specs were regrouped one per feature.