Google Cloud
On Google Cloud, Timelake runs inside Cloud Run functions and keeps its tables in a Cloud Storage bucket. Data reaches the functions in one of several ways, depending on how your devices send it. Every design below appends to the same table from the same source folder, so you can start with one and add another later.
| Design | Devices send | Visible to queries after | Fits |
|---|---|---|---|
| Files to Cloud Storage | CSV, Parquet, or JSON files | each upload | gateways that buffer readings and upload every few minutes |
| Pub/Sub to Cloud Storage | Pub/Sub messages | the file interval (1 to 10 minutes) | many small senders; Pub/Sub does the batching |
| Pub/Sub, pulled every minute | Pub/Sub messages | about a minute | a steady message stream with no extra files in between |
Which to start with. If your gateways can write files, use Files to Cloud Storage: it has the fewest parts, and each file is one commit. If devices send single readings, use Pub/Sub to Cloud Storage, which turns a minute of messages into one file and so one commit. Use Pub/Sub, pulled every minute when you'd rather not keep landing files, or need data within about a minute.
Then scheduled checks read recent data and alert, and maintenance runs every hour. Other designs covers MQTT devices, per-message triggers, and backfills.
Timelake hasn't been tested on Cloud Storage yet (Limitations). It reaches Cloud Storage through the same storage library as S3, and commits use Cloud Storage's generation preconditions.
Cloud Storage or BigQuery tables for Apache Iceberg
Google Cloud has its own way to keep Iceberg tables in your bucket: BigQuery tables for Apache Iceberg.
| Cloud Storage bucket, with Timelake | BigQuery tables for Apache Iceberg | |
|---|---|---|
| Data | Parquet and Iceberg metadata in your bucket | Parquet in your bucket; BigQuery keeps the table's metadata |
| Writers | Timelake (CLI, Python, Rust) | BigQuery: DML, load jobs, the Storage Write API, Pub/Sub BigQuery subscriptions |
| Compaction | In the writer, after appends; maintain for a full pass | BigQuery's storage optimization, billed as compute |
| Readers | Timelake, DuckDB, and other engines that open Iceberg tables by path | BigQuery; other engines read an exported metadata snapshot, which can lag streamed data |
| Time-series features | Latest values, deduplication, rollups, retention by time | Plain Iceberg, queried with BigQuery SQL |
| Runs on | Any object store, or local disk | Google Cloud only |
Choose BigQuery tables if your queries already run in BigQuery. Choose Timelake on Cloud Storage for frequent small appends that need fast queries straight away, for time-series queries, or to keep the same setup on other clouds. Files written into a BigQuery-managed table by anything other than BigQuery are not tracked, so Timelake can't write into one.
Other designs
- Devices over MQTT. Google Cloud IoT Core was retired in 2023, so there is no managed MQTT service to put in front. Run an MQTT broker (on GKE or a VM) with a bridge that publishes to the Pub/Sub topic, then use Pub/Sub to Cloud Storage or Pub/Sub, pulled every minute.
- A function per message. An Eventarc or push trigger on the topic calls a function per message: simple, and fine for a few messages a minute. Beyond that, use Pub/Sub to Cloud Storage or Pub/Sub, pulled every minute.
- A long-running consumer. For a stream busy enough to keep a puller running all the time, run the
pullloop continuously on GKE or a VM, appending every few seconds. - Backfill. Load history from a VM in the same region:
gcloud storage cp -rthe old files to disk, thentimelake import iot.readings ./history --batch-id backfill-2025. Appends of history don't conflict with live ingestion.
Cost
Each commit sends about 7 writes and 2 reads to the lake bucket; Cloud Storage charges writes as Class A and reads as Class B operations. At one commit a minute, that is about 10,000 writes a day. So the batching settings above (upload interval, file interval, pull schedule) are the main cost control. Storage & configuration breaks the requests down. Pub/Sub charges per data volume, and Cloud Run bills for the time each function runs.