Timeseries collections lose and duplicate committed points across graceful restarts

[mkhairi]

Tested against: origin/main @ 8e84501a (post-v0.3.0 main head, 2026-07-07)

Severity: High — silent loss and duplication of acknowledged writes on
the engine whose workloads (metrics ingestion) restart daemons routinely;
both faces corrupt every downstream aggregate.

Summary

Rows in a timeseries collection do not survive a graceful daemon
restart (SIGTERM, "Shutting down..." observed) intact. Across repeated
controlled restarts on fresh data directories, two corruption modes
appear, sometimes on consecutive restarts of the same collection:

1. Tail loss — the most recently inserted point(s) vanish after the
   first restart following the writes.
2. Duplication — surviving points are returned twice after a
   subsequent restart (also observed as every point duplicated on the
   first restart in an earlier data directory).

After the state has been corrupted once, further restarts leave it
stable. Document, KV, graph, and bitemporal data survive the identical
restarts intact on this build — the instability is specific to the
timeseries engine's flush/replay.

Repro

Session 1 (fresh data dir):

CREATE COLLECTION ts_r (ts TIMESTAMP TIME_KEY, value FLOAT) WITH (engine='timeseries');
INSERT INTO ts_r (ts, value) VALUES ('2026-07-07 10:00:00', 1.0);
INSERT INTO ts_r (ts, value) VALUES ('2026-07-07 10:05:00', 2.0);
INSERT INTO ts_r (ts, value) VALUES ('2026-07-07 10:10:00', 3.0);
SELECT * FROM ts_r;
--  1783418400000 | 1.0
--  1783418700000 | 2.0
--  1783419000000 | 3.0     -- 3 rows, correct

Graceful restart (same binary, same data dir), session 2:

SELECT * FROM ts_r;
--  1783418400000 | 1.0
--  1783418700000 | 2.0     -- 2 rows: the 10:10 point is GONE

Same experiment with a 2-point collection in the same data directory,
observed across consecutive restarts:

-- after write session: 2 rows (1.0, 2.0)     -- correct
-- after restart 1:     1 row  (1.0)          -- tail point lost
-- after restart 2:     2 rows (1.0, 1.0)     -- surviving point DUPLICATED
-- after restart 3:     unchanged (1.0, 1.0)

An earlier data directory on the same build showed the duplication face
directly on the first restart: 2 distinct points read back as 4 rows
(each point twice).

Expected

Committed timeseries points read back exactly once after any number of
graceful restarts — matching what the document/KV/graph/bitemporal
engines already deliver on this build.

Files (best-guess)

• Timeseries flush on shutdown / replay on boot (the "data-groups-replay"
  phase) — tail loss says the last in-memory chunk isn't flushed or
  replayed; duplication says a segment is applied twice on a subsequent
  replay.
• nodedb/src/data/executor/handlers/timeseries/raw_scan/scan.rs and the
  staged-insert overlay merge
  (nodedb/src/data/executor/handlers/transaction/overlay/timeseries_merge.rs)
  were recently touched for read-your-own-writes staging; if staged
  points reach the WAL and the flushed chunk both, that would double
  them on replay while an unflushed chunk alone would drop the tail.

Operational context

Timeseries workloads are exactly the ones that restart daemons routinely
(deploys, config reloads, host maintenance). Every restart silently
corrupts historical metrics — bucketed averages, sums, and counts are
wrong afterward, with no error at write time, shutdown, or boot.