A folder and cron

Do the Local setup first.

Gateways drop CSV files into a folder. A script appends each file to a table on local disk and moves it to done/.

Gateways

CSV files

write

incoming/

every minute

ingest.sh

timelake append

commit

warehouse/

Iceberg tables

1. Write the ingest script

Save as ingest.sh and make it executable (chmod +x ingest.sh):

#!/bin/sh
# Append every file in incoming/ to iot.readings, then move it to done/.
set -eu
cd "$(dirname "$0")"
export TIMELAKE_WAREHOUSE=./warehouse
for f in incoming/*.csv; do
  [ -e "$f" ] || continue
  timelake append iot.readings "$f" --batch-id "$(basename "$f" .csv)"
  mv "$f" done/
done

The file name is the batch id. If the script stops after an append but before the move, the next run skips that file instead of loading it twice. So give every file a unique name, such as the gateway and the time it was written (gateway-1-20260105T1000.csv), and send a correction as a new file.

2. Run it

$ cat > incoming/gateway-1-20260105T1000.csv <<'EOF'
time,device_id,temp,humidity
2026-01-05T10:00:00Z,pump-1,61.2,40
2026-01-05T10:00:00Z,pump-2,58.9,42
EOF
$ cat > incoming/gateway-1-20260105T1005.csv <<'EOF'
time,device_id,temp,humidity
2026-01-05T10:05:00Z,pump-1,62.0,41
2026-01-05T10:05:00Z,pump-2,59.4,43
EOF
 
$ ./ingest.sh
  commit 0: 2 rows, 1 files, 0.0 MB, commit 7 ms [0.1s]
iot.readings: appended 2 rows in 1 commits, 0.08s (26 rows/s) → v2
  commit 0: 2 rows, 1 files, 0.0 MB, commit 3 ms [0.0s]
iot.readings: appended 2 rows in 1 commits, 0.01s (277 rows/s) → v3

A file that arrives again is skipped:

$ cp done/gateway-1-20260105T1005.csv incoming/ && ./ingest.sh
  commit 0: 0 rows, 0 files, 0.0 MB, commit 0 ms (duplicate batch, skipped) [0.0s]
iot.readings: appended 0 rows in 0 commits, skipped 1 duplicate batch, 0.00s (0 rows/s) → v3
 
$ timelake sql "SELECT device_id, count(*) AS n, max(temp) AS max_temp FROM iot.readings GROUP BY 1 ORDER BY 1"
+-----------+---+----------+
| device_id | n | max_temp |
+-----------+---+----------+
| pump-1    | 2 | 62.0     |
| pump-2    | 2 | 59.4     |
+-----------+---+----------+

Batch ids are remembered while their commits are kept: by default the last hour, and at most 100 commits (the snapshot-retention and max-snapshots settings). A file loaded again after that adds its rows again, and deduplication still keeps one row per pump and time.

3. Run it on a schedule

With cron, every minute (crontab -e):

* * * * * PATH=$HOME/.cargo/bin:$PATH $HOME/timelake-ingest/ingest.sh >> $HOME/timelake-ingest/ingest.log 2>&1

Run one copy of the script at a time. Each append also does the cheap part of maintenance: it merges small files and removes duplicates. To keep each day in one sorted file, add timelake maintain once an hour.

Setup Appending from Python