TSBS DevOps: methodology

What we ran

We use TSBS's data generator, unchanged, and its 15 query types, ported to SQL every system runs. We don't use TSBS's loaders or query runners: timebench loads the data and times the queries, so the numbers don't compare with published TSBS results.

The original

TSBS's DevOps use case simulates servers reporting metrics every 10 seconds. Its cpu-only variant keeps the CPU metrics, 10 per host, and that's the one most databases publish results for: all 15 DevOps query types read only the cpu table.

The queries range from one host's maximum CPU over an hour to averages per host and hour across all hosts, the moments a host's CPU passed 90%, and the latest reading of every host.

Data

• TSBS's own generator, tsbs_generate_data --use-case=cpu-only, seed 123, one reading every 10 seconds per host from 2026-01-01 to 2026-01-04.
• Standard scale: 4,000 hosts, 104M rows, the size published TSBS results use. Small scale: 100 hosts, for development.
• Converted once to Parquet with the schema TSBS loads into TimescaleDB: a cpu table with the host name in it, as TSBS's loader puts it, and a tags table describing each host.

Two ways the data arrives

• Streamed: 5-minute windows like TSBS IoT, 864 commits of 120,000 rows at standard scale. Queries run after this.
• Bulk load: 1-hour windows, 72 commits of 1.44M rows, as fast as each system commits. The rate in rows per second is the number closest to TSBS's own load results, though TSBS loads with many parallel workers and timebench with one writer per system.

Queries

• All 15 types, named as in TSBS: single-groupby-<metrics>-<hosts>-<hours> (6 combinations), cpu-max-all-1 and -8, double-groupby-1, -5 and -all, high-cpu-1 and -all, lastpoint, groupby-orderby-limit.
• Ported once from TSBS's TimescaleDB SQL to SQL every engine runs. Random hosts and time windows are fixed (hosts below 100, windows ending at 2026-01-03 23:00, TSBS's window lengths). TimescaleDB's DISTINCT ON (lastpoint) becomes ROW_NUMBER() … = 1.
• The dashboard queries, probed during ingest and run by concurrent clients, are the single-host ones: three single-groupby types, cpu-max-all-1, high-cpu-1, and groupby-orderby-limit.

Systems and storage

The same systems, readers, and storage as TSBS IoT (see its methodology), with the host name as the series key: Timelake sorts by it, and TimescaleDB indexes it.

Compared with published TSBS numbers

Published TSBS DevOps results come from a database server queried by many parallel workers, reporting queries per second at full load. Ours are single queries from a fresh process, so the query times can't be set side by side. The bulk-load rate and the concurrent-dashboard throughput are the closest to TSBS's terms.

Planned: a run with TSBS's own loader and query runner, once Timelake has a server mode that speaks the PostgreSQL protocol.