v0 · open source, Apache 2.0

A serverless time-series lake on Apache Iceberg

Timelake stores time series as open Iceberg tables on any object store, keeps them sorted and compacted without external jobs, and reads them with a query engine that knows what a series is. There is no server and no database to run.

Quick start Benchmarks Explorer GitHub

The problem with time series on a lake today

Teams put telemetry in S3 Tables or self-managed Iceberg because it is cheap, durable and readable by every engine. Then they write custom jobs to make it fast.

Frequent appends make small files

Streaming 5-minute batches into an Iceberg table (the S3 Tables + Firehose pattern) leaves one file and one manifest per commit. In our test that was 864 files and 864 manifests per table after three days. Readers fetch all of them.

Maintenance is a separate system

Compaction, re-sorting, retention and snapshot expiry run as Glue or Spark jobs that someone schedules, sizes, pays for and debugs. Until they run, queries pay for the raw layout.

Generic engines don't know it's time series

"Latest value per device" scans the whole table. A query for one device in the last hour still opens files for every device, because nothing sorted or indexed the data by series.

How Timelake works

A Rust library with a CLI and a Python package. Writers, maintenance and readers are all stateless: everything durable is in object storage.

A catalog that is just files

Each commit writes vN.metadata.json with put-if-absent (S3 If-None-Match and equivalents). Exactly one concurrent writer wins; the others rebase and retry. No catalog server, no database.

A time-series contract

Declare a time column and a series key, like a TimescaleDB hypertable. Tables are partitioned by hour, day or month and every file is sorted by (series key, time).

Maintenance inside the writer

After an append, size-tiered compaction merges groups of similar-sized small files, and old snapshots expire in batches. timelake maintain rewrites each partition into one sorted, deduplicated file and applies retention.

A last-value index

Every append updates a small Parquet file with the newest row per series, referenced from the snapshot. "Latest per device" becomes one small read instead of a scan.

A reader that knows the layout

A DataFusion table provider prunes files on Iceberg column bounds, rewrites latest-row queries to the index, loads all of a query's tables concurrently and reads each Parquet footer once.

Plain Iceberg + Parquet on disk

Tables are Iceberg v2 with Parquet data files in the Hadoop-catalog layout (vN.metadata.json + version-hint.text). DuckDB's iceberg_scan reads them without Timelake installed.

Quick start

Point it at a local directory or an s3://, gs:// or az:// location.

shell

export TIMELAKE_WAREHOUSE=s3://my-bucket/lake

# Create a time-series table from a file's schema and load it
timelake import iot.readings readings.parquet --time-column time --series-key device_id

timelake sql "SELECT device_id, time_bucket(INTERVAL '1 hour', time) AS h, avg(temp)
              FROM iot.readings WHERE time > now() - INTERVAL '1 day' GROUP BY 1, 2"
timelake latest iot.readings     # newest row per device, from the index
timelake maintain                # compact, retention, expire (also runs inline)

Benchmark highlights

TSBS IoT data (23.3M rows) streamed as 1,728 five-minute micro-batches into Timelake, Lance and naive Iceberg (PyIceberg appends, the S3 Tables write pattern), then queried with the 12 TSBS IoT queries.

All results

3.7–16×

faster TSBS IoT queries on S3 right after streaming ingest

0.28 s vs 1.05–4.50 s, first run

1.2–1.5×

faster after every system's own compaction

0.29 s vs 0.36–0.42 s, first run

12 vs 1,733

requests for "latest position per truck" on S3 after streaming ingest

Timelake vs naive Iceberg (Lance: 3,614)

434 s

to stream 1,728 commits to S3, compaction included

Lance 817 s, PyIceberg 1,607 s

“S3” is RustFS on one laptop behind a proxy that adds 20 ms per request, not AWS. S3 Tables' managed compaction is simulated with its default (binpack, no sort). One dataset, one machine; see the methodology.

Open by default

Every byte is Iceberg metadata and Parquet. Plain DuckDB reads Timelake tables with identical results (checked on every benchmark query), and the last-value index is a Parquet file any engine can read.

Runs where your data is

S3 and S3-compatible stores, Google Cloud Storage, Azure Blob Storage, or a local directory. Commits need conditional writes (put-if-absent). Tested so far on local disk and RustFS.

Storage & configuration