Codebase tour

What lives where, which technologies are used and why, and how a write and a query flow through the code. Paths are relative to the repository root. For the design behind it, see the specs; for the on-disk format, see Architecture.

The big picture

            your code
   ┌───────────┼──────────────┬──────────────────┐
   │           │              │                  │
 timelake    Python         Rust             browser / Node / serverless
  (CLI)    (timelake-py)   (use the crate)       │
   └───────────┼──────────────┘           @timelake/client (TypeScript)
               │                                 │
        crates/timelake  (the engine)     crates/timelake-wasm (read-only)
               │                                 │
               └──────── crates/timelake-format ─┘   shared rules
                                 │
                 object storage: Iceberg metadata + Parquet files

• One engine. All writing and full SQL happen in crates/timelake. The CLI and the Python package are thin layers over it.
• A second, small reader for JavaScript. The engine depends on DataFusion and Tokio, which are too large for a browser. crates/timelake-wasm reimplements just the reading needed for time-series queries (latest values, time ranges, time buckets). The rules both must agree on live in crates/timelake-format.
• The website uses the client package to read the demo data, and the CLI for local SQL.

Repository layout

Technologies

The engine, file by file

crates/timelake/src, about 4,000 lines including unit tests. Read them in this order:

Tests live next to the code (#[cfg(test)] modules, with test storage in testing.rs that injects faults or latency) and in crates/timelake/tests: end to end on local disk (lifecycle.rs), concurrent commits (commits.rs), and SQL (sql.rs).

Follow an append through the code

timelake append iot.readings day1.csv, or t.append(batch) in Python, ends up in Table::append (append.rs):

1. sort_and_split (timelake_format::rows): match columns to the table, compute each row's day, sort by day, series key, and time, and cut the batch into one piece per day.
2. write_partition (write.rs): encode each day's rows as Parquet under data/time_day=…/, several days at a time, each row group's columns on threads of their own. Each file comes back with its path, size, row count, and min/max per column; its upload waits in an Uploads queue (a large file's earlier parts are already sent).
3. latest_of_sorted (timelake_format::rows): the newest row of each series in the new data.
4. commit (commit.rs): a SnapshotWriter prepares a manifest for the new files, the merged last-value index, and the manifest list, then uploads them and the data files at once. With more than eight manifests, the old ones are merged into one so reads stay quick.
5. publish (commit.rs): write_if_absent(vN+1.metadata.json). If the file already exists, another writer won: delete every file this attempt wrote, reload, and retry. On success, update version-hint.text and the head in the background (background.rs, head/).
6. Back in append: if inline maintenance is on, run size-tiered compaction on the days this append touched, and expire old snapshots in batches. If that fails, the append still succeeded: the error is returned in the result, not raised.

Follow a query through the code

timelake sql "…" or lake.sql("…") calls Session::sql (query/session.rs):

1. Parse the SQL and find every table it mentions.
2. prefetch: load all those tables in parallel. A table with a head comes from that one object (metadata, files, last-value index, footers), confirmed by a listing made at the same time. A table this process already has is checked for a newer version while planning goes on; the scan waits for that check. Others are read through the metadata chain.
3. DataFusion plans the query. It asks the lake catalog (query/catalog.rs) for each table and gets a TimelakeProvider (query/provider.rs).
4. The optimizer runs Timelake's rules. The latest-row rewrite (query/rewrite.rs): if the query asks for the newest row per series (or the newest row for some series), the table scan is replaced with a scan of the last-value index. Rollup routing (query/route.rs): a GROUP BY time_bucket answered from a rollup for the buckets it has right. Newest-first (query/newest.rs): ORDER BY time DESC LIMIT k reads time partitions newest first and stops early. Series order (query/series_order.rs): scans under a time_weighted_mean merge files that could give a series out of order.
5. Otherwise the provider drops files whose min/max values (query/stats.rs) can't match the WHERE clause, spreads the remaining files over parallel scan tasks, fetches every file footer once, and hands the files to DataFusion's Parquet reader, which skips row groups and pages. For IN lists longer than DataFusion prunes with, the provider works out the row groups and pages to read from the footers itself (query/in_list.rs) and passes them as each file's access plan. On scans without filter pushdown, a predicate adapter does the same for a join's runtime filter as each file is opened. With dedup on, files of one partition whose key bounds overlap may hold the same (series, time) (query/dedup.rs): they are pruned only on the key columns, split by a hash of the series, merged in key order slice by slice, and only the latest write of each key is kept.
6. DataFusion runs the rest (filters, joins, aggregates) and returns Arrow batches.

The CLI and the Python package

• crates/timelake-cli/src: main.rs defines the commands with clap and prints results as a table, CSV, or JSON; load.rs reads CSV and Parquet and appends them in commits; info.rs prints the engine's TableInfo as timelake info text; bench.rs is the hidden benchmark mode of timelake sql. tests/cli.rs runs the binary end to end.
• crates/timelake-py/src/lib.rs: classes Lake and Table. Each method runs the engine on a shared Tokio runtime and releases Python's GIL while it waits. Data comes in through the Arrow C stream interface, so pyarrow and polars objects work directly. Errors map to KeyError, FileExistsError, ValueError and timelake.ConflictError.
• crates/timelake-py/python/timelake: the Python side: __init__.py, the type stub _timelake.pyi, _types.py (typed result dicts), and py.typed. Tests are in crates/timelake-py/tests.

The WebAssembly reader and the TypeScript client

The WASM module never makes network requests. The TypeScript client fetches bytes and hands them in, which keeps the module small and identical in every JavaScript runtime:

1. The client fetches version-hint.text and vN.metadata.json; new Table(json) parses it (meta.rs, with the metadata reader of timelake-format).
2. The client fetches the manifest list and manifests; the module parses them (Avro, with timelake_format::iceberg, as the writers read them) into a list of files.
3. planFiles(query) drops files whose min/max values can't match (prune.rs).
4. For each file, the client fetches the end of the file (the footer). ParquetFile.plan (file.rs) returns the byte ranges of the row groups and columns needed. The client fetches just those ranges.
5. Query.push decodes and filters each file a batch at a time (filter.rs) and collects rows (with a limit, only those that can still be returned), time buckets (aggregate.rs), or the newest row per series (latest.rs); finish returns JSON rows (output.rs). The client validates them with Zod if a schema was given.

spec.rs describes the query a host sends; error.rs the errors it gets back. The crate has its own unit tests and a golden test on a small table written by the engine (crates/timelake-wasm/tests).

packages/timelake-client/scripts/build-wasm.sh compiles the crate, shrinks it with wasm-opt, and embeds it as base64 in src/wasm/inline.js, so the package loads without a separate .wasm file. The generated files are committed with a hash of their sources in the header; bun run check:wasm (and CI) fails when they are stale. The client's tests build a small warehouse with the CLI, serve it locally, and compare answers with the engine's, without network access.

The website

Vercel builds and deploys the site on every push to main (project root dashboard/).

The benchmarks

How to build and test each part is on Development.