30. Signal aggregates

Summary

time_weighted_mean, integral, counter_diff and counter_rate over irregular readings, exact, as fast as avg. They share one mergeable state (runs of readings), which rollups store and several aggregates of one column compute once; a condition counts as 1 while true, for time in a state and uptime. gap_count and gap_seconds, how often and how long readings stopped, are run aggregates of the same kind.

Design

time_weighted_mean(time, value [, method]) and integral(time, value [, method]) (value × seconds), method forward_fill (default, alias ffill) or linear; counter_diff(time, value) and counter_rate(time, value) (per second), where a drop is a reset to zero, so the reading after it counts whole. All over each group's first to last reading; readings at one time count once with the largest value; one reading time gives its value as the mean, 0 as the integral and growth, and no rate. Decided with the user on 2026-10-18 (exact always, both methods, first to last reading).

State: a list of segments, runs of a group's readings in time order: first and last reading, the values of the readings next to them, the time from the first to the next and into the last, the area and the counter's growth. Segments that follow each other join exactly (the gap's area and growth added; a shared time takes the larger value, corrected through the stored neighbours); partial results emit their runs unjoined, since another partition may hold readings between them. Overlapping segments are an error. The four aggregates share the state and differ in what they return from it. Layout measured on IIoT hourly means: neighbours as Option<f64> (96-byte segments) 2.30 s, as plain f64s (80 bytes) 2.17 s, the same as the mean alone before; the open segment kept inline in each group instead of behind the vector's pointer 2.95 s (the group array doubles and DataFusion moves it as it grows).
Series order: no partition may have readings of a series inside another partition's run of it. Scans under a signal aggregate read each series whole and in time order, merging only the files that need it (spec 26, setting series-order).
Rollups store the state (time_weighted_state); routing merges it with the query's aggregate (integral(state [, method])), at the rollup's width too. One stored state answers all four for its column and method; the counters from either method's state.
Cost: IIoT small, hourly per tag over 30 days: avg 1.98 s; each signal aggregate 2.18–2.20 s; all four in one query 3.47 s with a state each, 2.11 s sharing one: an optimizer rule after routing replaces the signal aggregates of the same readings and method (the counters join either method's) with one time_weighted_state and computes each result from it in a projection (integral_of_state(state)); setting shared-signal-state. Against the window form of the IIoT queries over 30 days: energy per machine (lag) 1.66 → 1.09 s, time-weighted means of a line's temperatures (lead) 166 → 87 ms.
Gaps (built 2026-10-19, #123): gap_count(time, threshold) and gap_seconds(time, threshold), how many times and for how long in all a group's readings stopped for longer than threshold (an interval literal without months). A gap is the time from a reading to the next; silences before the first reading or after the last aren't gaps. Their run is a reading time's first and last, the gaps so far and their length (32 bytes, a run of its own rather than fields added to the signal segment, which cost 6% before); gap_state returns the state. The run code is generic (query/runs.rs: a Run trait, the runs of a group and the accumulators), shared with the signal aggregates at no cost (hourly means 3.58 → 3.50 s, all four 4.80 → 4.68 s, interleaved runs on a loaded machine). IIoT gaps of 100 one-second tags, same answers as the benchmark's lead() window form: a day 226 → 77 ms, 30 days 17.8 → 1.36 s. Rollups store the gap state (spec 27): from an hourly rollup, 30 days of gaps in 44 ms.

Open questions and issues

None open.

History

2026-10-18: time_weighted_mean built
2026-10-19: integral, counter_diff, counter_rate, conditions, shared state, scans in stretches of time
2026-10-19: moved from SPEC.md into its own spec.
2026-10-19: was spec 22; specs regrouped one per feature, with milestones on the roadmap.
2026-10-19: gap_count and gap_seconds, on run code shared with the signal aggregates.

Summary

Design

State: a list of segments, runs of a group's readings in time order: first and last reading, the values of the readings next to them, the time from the first to the next and into the last, the area and the counter's growth. Segments that follow each other join exactly (the gap's area and growth added; a shared time takes the larger value, corrected through the stored neighbours); partial results emit their runs unjoined, since another partition may hold readings between them. Overlapping segments are an error. The four aggregates share the state and differ in what they return from it. Layout measured on IIoT hourly means: neighbours as Option<f64> (96-byte segments) 2.30 s, as plain f64s (80 bytes) 2.17 s, the same as the mean alone before; the open segment kept inline in each group instead of behind the vector's pointer 2.95 s (the group array doubles and DataFusion moves it as it grows).

Series order: no partition may have readings of a series inside another partition's run of it. Scans under a signal aggregate read each series whole and in time order, merging only the files that need it (spec 26, setting series-order).

Rollups store the state (time_weighted_state); routing merges it with the query's aggregate (integral(state [, method])), at the rollup's width too. One stored state answers all four for its column and method; the counters from either method's state.

Cost: IIoT small, hourly per tag over 30 days: avg 1.98 s; each signal aggregate 2.18–2.20 s; all four in one query 3.47 s with a state each, 2.11 s sharing one: an optimizer rule after routing replaces the signal aggregates of the same readings and method (the counters join either method's) with one time_weighted_state and computes each result from it in a projection (integral_of_state(state)); setting shared-signal-state. Against the window form of the IIoT queries over 30 days: energy per machine (lag) 1.66 → 1.09 s, time-weighted means of a line's temperatures (lead) 166 → 87 ms.

Gaps (built 2026-10-19, #123): gap_count(time, threshold) and gap_seconds(time, threshold), how many times and for how long in all a group's readings stopped for longer than threshold (an interval literal without months). A gap is the time from a reading to the next; silences before the first reading or after the last aren't gaps. Their run is a reading time's first and last, the gaps so far and their length (32 bytes, a run of its own rather than fields added to the signal segment, which cost 6% before); gap_state returns the state. The run code is generic (query/runs.rs: a Run trait, the runs of a group and the accumulators), shared with the signal aggregates at no cost (hourly means 3.58 → 3.50 s, all four 4.80 → 4.68 s, interleaved runs on a loaded machine). IIoT gaps of 100 one-second tags, same answers as the benchmark's lead() window form: a day 226 → 77 ms, 30 days 17.8 → 1.36 s. Rollups store the gap state (spec 27): from an hourly rollup, 30 days of gaps in 44 ms.

History

2026-10-18: time_weighted_mean built

2026-10-19: integral, counter_diff, counter_rate, conditions, shared state, scans in stretches of time

2026-10-19: moved from SPEC.md into its own spec.

2026-10-19: was spec 22; specs regrouped one per feature, with milestones on the roadmap.

2026-10-19: gap_count and gap_seconds, on run code shared with the signal aggregates.