timebench
The benchmark system behind every result on this site. It streams the same data into each system, runs each system's own maintenance, and times the same queries before and after. It is open source, lives in the Timelake repository for now, and is built to move to a repository of its own.
Principles
- Every system gets the same rows, in the same order, and the same queries.
- Each system is written, maintained, and read the way its own documentation suggests.
- Data arrives as it does in production, in small batches from a stream, not in one bulk load.
- Not only table formats: databases and plain files run too, so a result says what a team would otherwise use.
- Losses are published next to wins.
- Every run is kept, with the commit, the machine, and every version, and anyone can rerun it.
How a run works
- The workload's data is generated from a fixed seed, the same every time.
- Ingest. Each system creates its tables and commits the data batch by batch. Each commit is timed, and the files and requests it took are counted.
- Probes while ingesting. Every few seconds, a fresh reader runs the dashboard queries and asks for the newest row it can see. That gives query latency while data is being written, and whether a reader sees every commit that has finished.
- Queries right after ingest. What readers see before any cleanup: many small files and a long history of commits. Systems whose compaction runs later, as a scheduled job or a managed service, look like this until it has run.
- Maintenance. Each system's own: compaction, cleanup of old versions. For S3 Tables, the run waits for AWS's managed compaction, which takes hours.
- Queries after maintenance. The same queries on tidy tables, the best case for each system.
- Concurrent dashboards. 1, 4, and then 16 clients at once, each keeping a reader open like a dashboard backend, run the dashboard queries in turn for 30 seconds: queries per second over all clients, and how slow the slowest 5% were.
- Row counts are compared across systems. A mismatch is recorded with the run.
Reading the results
- First run. Each query runs in a fresh process, once, then twice more. The first run loads table metadata, file footers, and data from storage: the latency a new dashboard, job, or serverless function sees. The ranked lists show first runs; the results pages also show the best repeat.
- Geometric mean. A benchmark's time is the geometric mean over the queries every system completed, so one slow query doesn't hide the rest and every system answers the same set. The multiplier is relative to the fastest.
- System and reader. "Lance + DuckDB" means Lance's tables read by DuckDB. Timelake alone means Timelake's own reader.
- Failures are left out of the means and listed on the results page. A query that runs past its time limit isn't run again at that system's later checkpoints; it is listed as skipped.
- Requests and bytes. On S3, every request goes through a proxy that counts requests and bytes: what object storage bills for, and the same wherever the client runs.
- Cost. On AWS, requests by kind and the bytes stored are priced at S3's list prices for the region, recorded with each run so later price changes don't rewrite history. Ingest is scaled to a month of data arriving at the benchmark's rate. Machines aren't included yet, and S3 Tables' own storage and compaction fees aren't modeled.
- Database servers keep data in memory between queries, so their first run is warmer than a fresh process reading object storage.
Systems
| System | What it is | Written | Maintenance | Read by |
|---|---|---|---|---|
| Timelake | Timelake's own writer: Iceberg tables of sorted Parquet, compacted as it appends. | Table.append per commit, with inline maintenance (compaction, dedup, retention). | timelake maintain: each partition rewritten into one sorted file, rollups refreshed. | Timelake, DuckDB |
| Iceberg (PyIceberg) | Plain Iceberg tables written by PyIceberg appends: what a Firehose stream into S3 Tables writes. | PyIceberg append per commit, tables partitioned by day. | A bin-pack rewrite of small files, S3 Tables' default compaction. | DuckDB |
| S3 Tables (PyIceberg) | The same appends into Amazon S3 Tables, AWS's managed Iceberg. | PyIceberg append per commit, through S3 Tables' Iceberg REST endpoint. | S3 Tables' own managed compaction; the run waits for it, up to 6 hours. | DuckDB |
| Lance | Lance datasets, appended commit by commit. | write_dataset in append mode per commit. | compact_files, then cleanup_old_versions. | DuckDB, DataFusion |
| Parquet files | Parquet files without a table format: a new file per commit, as an S3 sink writes them. | A Parquet file per commit in a folder per day. | Each day rewritten into one sorted file. | DuckDB, DataFusion |
| TimescaleDB | TimescaleDB, a PostgreSQL database for time series, on its server's own disk. | COPY per commit into hypertables; rows that arrive twice are skipped. | Every chunk converted to its columnstore. | PostgreSQL |
| ClickHouse | ClickHouse MergeTree tables on its server's disk, ordered by series key and time. | Arrow inserts per commit; tables where rows can repeat use ReplacingMergeTree. | OPTIMIZE TABLE … FINAL merges every partition; repeated rows are read with FINAL until then. | ClickHouse |
| QuestDB | QuestDB WAL tables partitioned by day, on its server's disk. | Its wire protocol from Python, per commit; repeated rows are removed on write (DEDUP UPSERT KEYS). | None; a checkpoint waits until every commit is applied. | QuestDB |
| InfluxDB 3 Core | InfluxDB 3 Core with its query file limit raised, as its docs allow, so long time ranges can be queried. | Line protocol over HTTP, four requests at a time. | None: Core has no compaction. | InfluxDB |
| InfluxDB 3 Core, as shipped | InfluxDB 3 Core as shipped: a query reads at most 432 Parquet files, about 3 days of data, so longer ones fail. | Line protocol over HTTP, four requests at a time. | None: Core has no compaction. | InfluxDB |
Each benchmark's page lists the systems in its results, and its methodology what each does in it. Planned: Iceberg written the way an expert team would, with a streaming sink and scheduled compaction.
Where it runs, and the history
- Tables live on the machine's disk, or on S3. For development, a local object store with a fixed delay per request stands in for S3.
- Every week, the same benchmarks run on one fixed AWS machine in us-east-2 (8 Graviton cores, local SSD): TSBS on Mondays, on the disk and on S3 in the same region, and industrial IoT on Thursdays. Other systems' versions are pinned, so a change in the history comes from Timelake.
- Each run is one file in the repository, which the results pages chart over time. Each results page links the commit it measured.
Benchmarks
The public benchmarks for time-series data that we know of, our own, and why each runs or doesn't.
| Benchmark | Status | Why |
|---|---|---|
| TSBS IoT | Running | Our base workload. TSBS is the benchmark time-series databases publish numbers for. |
| TSBS DevOps | Running | The TSBS use case most databases publish numbers for. |
| ClickBench | Planned | We run a 10% subset during development. Comparable numbers need the full dataset on its reference machine. |
| RTABench | Planned | Close to fresh-data dashboards. Its pre-aggregated queries wait for rollups. |
| TSM-Bench | Planned | Its queries and online mode fit. Its code has no license yet; we're asking the authors. |
| IoT-benchmark | Planned | The best tool for mixed reads and writes. Its adapters talk to a server, so it waits for Timelake's server mode. |
| Longevity | Planned | Borrows LST-Bench's method (Microsoft, SIGMOD 2024), applied to our own time-series data. |
| Industrial IoT | Running | TSBS models trucks every 10 seconds. Plants look different, and that's where Timelake is used. |
| AI agents | Planned | People increasingly ask questions of their data through AI agents, and no public benchmark compares storage for that. |
| TS-Benchmark | Not planned | Unmaintained since 2022, and its 5 queries are covered by the benchmarks above. |
| TSDBBench | Not planned | Unmaintained since 2019; the databases it compares have moved on. |
| TPCx-IoT | Not planned | TPC rules allow only audited results to be published, and its key-value records don't suit columnar files. |
| STAC-M3 | Not planned | Closed to members, with paid audits. |
| TSRBench | Not planned | It pastes the series into the prompt, so storage plays no part. Our AI agents benchmark asks the question that matters for Timelake. |
Run it yourself
uv sync && cargo build --release -p timelake-cli uv run timebench list # benchmarks, sizes, systems uv run timebench run tsbs-iot --scale small # every lake system on your disk, a few minutes uv run timebench report latest
Setup, storage options, and how to add a benchmark or a system: bench/README.md.