Roadmap & limitations
Known limitations in v0
- No REST catalog. Tables are found by path (Hadoop-catalog layout). Engines that only speak the Iceberg REST protocol can’t attach yet.
- Freshness is per commit. Data is visible when its append commits; there is no hot tier or write-ahead log for sub-commit freshness.
- No deletes. Row-level deletes, the delete audit trail and GDPR erasure are designed but not built. Retention drops whole partitions only.
- No schema evolution. The schema is fixed at creation.
- Duplicates until compaction. Dedup happens during compaction, not at read time.
- In-memory compaction. A partition is merged in memory; size the chunk interval so partitions fit.
- Planning cache. A process reuses a table version it loaded within the last second for query planning (its own commits are always visible).
- Replaced files linger. Files removed by compaction are deleted when the snapshots that reference them expire (after an hour by default).
- Benchmarks are narrow. One dataset (TSBS IoT), one machine, S3 simulated with RustFS plus a latency proxy.
- Tested storage. Local disk and RustFS. AWS S3, GCS and Azure use the same code paths but haven’t been benchmarked.
Planned
| Area | What |
|---|---|
| Catalog | An optional Iceberg REST endpoint over the same layout, so any REST-capable engine can attach. |
| Deletes | Immutable, audited deletes with deletion vectors (Iceberg v3), undelete, and GDPR purge with a 7-day maximum delay. |
| Backfill & import | Branch-staged backfill with atomic publish; import from Iceberg/S3 Tables and TimescaleDB. |
| Aggregates | Incremental continuous aggregates stored as Iceberg tables, and a time-series function library (gap filling, time-weighted averages, counters, percentiles). |
| Freshness | An optional hot tier (object-store WAL or Valkey) merged by the reader for second-level freshness. |
| Ingestion | A Telegraf/Firehose-style pipeline (MQTT, HTTP, Kafka) with in-stream rules and alarms, deployable as a library, Lambda or container. |
| DuckDB | A DuckDB extension with dedup on read, index-aware scans and the function library. |
| Reader | A data cache for repeat queries, and more query shapes served from indexes. |