Summary
The Timelake reader (CLI, Rust, Python, and a WebAssembly reader for JavaScript) runs SQL on DataFusion over Timelake tables, and uses what a generic engine doesn't know: dedup on read, the last-value index, rollup routing, series-aware scans, time-series functions, and I/O tuned for object storage. Plain DuckDB still reads the same tables (spec 1).
Design
We don't try to beat DuckDB as a general engine. We win by using knowledge a generic engine doesn't have:
| What | Status |
|---|---|
| Pruning by Iceberg column bounds, Parquet row-group and page statistics | Built (v0) |
| Last-value index: "current state of every device" without scanning (spec 6) | Built (v0) |
| Dedup on read before compaction (spec 12) | Built (0.5.0) |
| Automatic rollup routing (spec 27) | Built (0.5.0) |
| I/O tuned for object storage: one-request table state (spec 7), a block cache (spec 22), the scan settings (spec 23, spec 24, spec 25) | Built (0.3.0–0.7.0) |
| Sort-order awareness: no re-sorting for per-series windows and signal aggregates (spec 26) | Built (0.7.0); ASOF joins planned |
| Series-level pruning from a series index, beyond min/max statistics (#36) | Planned |
| Hot-tier merge for instant freshness (spec 14) | Planned |
Engine: a DataFusion TableProvider per table, optimizer rules (rollup routing, latest-row rewrites, newest-first,
series order) and the function library (spec 28); why DataFusion is in
embedded engine, and why the provider is ours rather than iceberg-datafusion in its D24. v0's reader
lessons, now defaults: load all of a query's tables concurrently before planning, read each Parquet footer once, and use
more scan partitions on object storage.
Python:
import timelake as tl
lake = tl.Lake("s3://bucket/warehouse")
lake.sql("SELECT … time_weighted_mean(…) …") # full SQL via DataFusion, a pyarrow.Table
lake.latest("iot.sensor") # from the last-value indexResults are Arrow, so Polars, pandas and DuckDB take them zero-copy.
Open questions and issues
- #32 Query: read string columns as Utf8View
- #36 Reader: a series index that covers some files, with statistics pruning for the rest
- #56 Query: load each table once when concurrent queries need it
- #62 Permissions: re-apply row filters that the scan only uses for pruning
History
- 2026-10-11: reader design (SPEC.md §9)
- 2026-10-12: built in v0
- 2026-10-19: moved from SPEC.md into spec 17, then into this spec when the specs were regrouped one per feature.
- 2026-10-20: links D24, the reason the table provider is Timelake's own rather than
iceberg-datafusion.