AI agents: methodology
PlannedAn AI agent answers questions about the data through tools. Is it right more often, and cheaper, from Timelake?
Not run yet. People increasingly ask questions of their data through AI agents, and no public benchmark compares storage for that. This page is the plan.
What we'll run
Our own benchmark. Later, OpenRCA's and AssetOpsBench's tasks, run with their own code.
The question
TSRBench and similar benchmarks paste a series into the prompt, so storage plays no part. Here the agent has to find and fetch the data itself. Benchmarks close to this exist for other data (OpenRCA, AssetOpsBench, o11y-bench); none compares where the data is stored.
How we'll run it
- The same agent, model, and questions against the same data in raw files, DuckDB, TimescaleDB, Timelake over SQL, and Timelake over its own agent tools.
- Questions generated from the data with events added on purpose, so every answer is known: values with units, comparisons between periods, anomalies and when they happened, a cause shared by many devices, gaps, late data, and questions that should be refused.
- Answers checked automatically. For each question: right or wrong, tokens, tool calls, time, and cost.
- Then OpenRCA's and AssetOpsBench's tasks against Timelake, as published comparisons.