20. Maintenance runs, retention and expiry

Summary

The jobs Timelake replaces Glue with (compaction, rollup refresh, retention, snapshot expiry and file cleanup) run inline after appends when thresholds are met, or as one stateless timelake maintain command. Nothing outside Timelake has to be scheduled. Orphan-file cleanup and sharded runs are open.

Design

timelake-maintain runs as a set of idempotent, independently schedulable tasks:

Task	What it does	Status
Compaction	spec 19	Built
Rollup refresh	Incremental, from the raw table's changed buckets (spec 27)	Built (0.5.0)
Retention	Per-table TTL (e.g. raw 90 d, 1-min rollup 2 y, 1-h rollup forever). Drops whole partitions (metadata-only when aligned)	Built
Snapshot expiry	Expires old snapshots and deletes the files only they reference, never one a branch or tag points at; `timelake.keep-snapshots-until` keeps them longer (0.3.0, for upgrades). What the kept snapshots use is read in full first: if a manifest list or manifest of one is missing, expiry stops with an error naming it and deletes nothing; one only expired snapshots used that is already gone is skipped	Built
Orphan cleanup	Deletes files no snapshot references (#50)	Planned
Manifest rewrite	Clusters manifests by time so planning stays fast as tables grow	Planned

Inline: after an append, auto-maintain runs the tasks whose thresholds are met (compact-min-files, compact-cold-after, rollup-refresh-interval, …), so small deployments need nothing else.
timelake maintain (Table.maintain()) runs a full pass: every partition rewritten into one file, every rollup refreshed (--no-rollups to skip), retention and expiry. Stateless: a cron job, a Kubernetes CronJob, a Lambda, or triggered by commit events (catalog events, S3 notifications).
Concurrency: safe through Iceberg optimistic commits and conflict-aware retries (spec 2).
Limits per run: a setting caps the bytes one run rewrites (needed on Workers, where a request has at most 5 minutes of CPU; useful anywhere inline maintenance shouldn't hold up appends). A run that stops at the cap commits what it rewrote; the next run continues. Built in the shared (WASM) maintenance (spec 13), and MaintainOptions::max_bytes in the engine (unreleased).
One plan for both writers: which files compaction merges, which partitions retention drops, which snapshots expire and which metadata files go are decided by timelake_format::maintain for the engine and the WASM writer alike, and both report the same fields. Only the engine refreshes rollups.

Open questions and issues

#13 Maintenance: wait a minimum age before deleting files of expired snapshots
#18 Maintenance: --shard i/n so concurrent runs don't repeat work
#50 Maintenance: delete orphan files

History

2026-10-11: maintenance tasks (SPEC.md §12)
2026-10-12: inline maintenance, retention and expiry built in v0
2026-10-14: keep-snapshots-until for upgrades
2026-10-17: rollup refresh in maintenance; byte cap per run (WASM)
2026-10-19: moved from SPEC.md into spec 15, then into this spec when the specs were regrouped one per feature.
2026-10-20: expiry reads what the kept snapshots use in full before it changes anything (#46).
2026-10-20: expiry keeps snapshots a branch or tag points at.
2026-10-20: one plan and one report for both writers; the engine skips manifests an interrupted expiry deleted, as the WASM writer did, and lists only metadata/v to prune old metadata files. The WASM writer's expiry retries with backoff and a limit, as commits do.

Summary

Design

timelake-maintain runs as a set of idempotent, independently schedulable tasks:

Task

What it does

Status

Compaction

spec 19

Built

Rollup refresh

Incremental, from the raw table's changed buckets (spec 27)

Built (0.5.0)

Retention

Per-table TTL (e.g. raw 90 d, 1-min rollup 2 y, 1-h rollup forever). Drops whole partitions (metadata-only when aligned)

Built

Snapshot expiry

Expires old snapshots and deletes the files only they reference, never one a branch or tag points at; timelake.keep-snapshots-until keeps them longer (0.3.0, for upgrades). What the kept snapshots use is read in full first: if a manifest list or manifest of one is missing, expiry stops with an error naming it and deletes nothing; one only expired snapshots used that is already gone is skipped

Built

Orphan cleanup

Deletes files no snapshot references (#50)

Planned

Manifest rewrite

Clusters manifests by time so planning stays fast as tables grow

Planned

Inline: after an append, auto-maintain runs the tasks whose thresholds are met (compact-min-files, compact-cold-after, rollup-refresh-interval, …), so small deployments need nothing else.

timelake maintain (Table.maintain()) runs a full pass: every partition rewritten into one file, every rollup refreshed (--no-rollups to skip), retention and expiry. Stateless: a cron job, a Kubernetes CronJob, a Lambda, or triggered by commit events (catalog events, S3 notifications).

Concurrency: safe through Iceberg optimistic commits and conflict-aware retries (spec 2).

Limits per run: a setting caps the bytes one run rewrites (needed on Workers, where a request has at most 5 minutes of CPU; useful anywhere inline maintenance shouldn't hold up appends). A run that stops at the cap commits what it rewrote; the next run continues. Built in the shared (WASM) maintenance (spec 13), and MaintainOptions::max_bytes in the engine (unreleased).

One plan for both writers: which files compaction merges, which partitions retention drops, which snapshots expire and which metadata files go are decided by timelake_format::maintain for the engine and the WASM writer alike, and both report the same fields. Only the engine refreshes rollups.

History

2026-10-11: maintenance tasks (SPEC.md §12)

2026-10-12: inline maintenance, retention and expiry built in v0

2026-10-14: keep-snapshots-until for upgrades

2026-10-17: rollup refresh in maintenance; byte cap per run (WASM)

2026-10-19: moved from SPEC.md into spec 15, then into this spec when the specs were regrouped one per feature.

2026-10-20: expiry reads what the kept snapshots use in full before it changes anything (#46).

2026-10-20: expiry keeps snapshots a branch or tag points at.

2026-10-20: one plan and one report for both writers; the engine skips manifests an interrupted expiry deleted, as the WASM writer did, and lists only metadata/v to prune old metadata files. The WASM writer's expiry retries with backoff and a limit, as commits do.