[gareth-morgan-nv]

Description

Stacked on the egocap timing PR (#1459): this branch targets that PR's branch, so the diff shows only the recorder-side work.

Kept episodes are time-aligned in the background after a take is finalized. Each source is resampled onto one index camera's frame times and written as an aligned copy beside the raw session; the raw recording is never modified.

What changed

• Aligner and writer. Valid range, nearest-frame indices (earlier tie wins, frames may repeat), linear and spherical interpolation per frame type, and an aligned tree with valid_indices, valid_timestamps, aligned_frame_indices and <field>_interp arrays, copied attributes and sidecars, and hard-linked videos. A source that cannot be aligned (single sample, backwards timestamps, no common range) is refused with the reason rather than skipped.
• Qualification and continuity evidence. Per-source timing evidence says whether a saved time is a qualified event time; continuity evidence (clock-mapping epoch or host-clock connection) must show no break. A kept episode is bound to its manifest (identity, finalized/kept state, routed slot, written row counts) before the evidence is trusted, and a source with recorded rows that is missing from the episode is refused.
• Tracking time estimate. For the glove and controller, a configured per-source delay (alignment.tracking_latency_s in the station file) is taken off the host poll time once, when a sample is converted. The source's timing evidence records it as an estimate with the latency used. A source with no configured delay stays unqualified; zero is a valid delay. The station template ships no value, and the estimate is not written under the dataset's latency key. The aligner uses saved times unchanged.
• Publication. The aligned episode is built in a private staging directory and moved into place with one rename. An existing aligned episode is refused, never replaced.
• Automatic alignment. One background worker aligns each kept episode (including one kept by recovering an interrupted take), on when the station file sets alignment.index_source (there is no default; alignment.enabled: false turns it off). Each alignment runs as a low-priority child process of the command-line tool, so it does not share the recorder's interpreter and can be stopped. Alignment is skipped, with one logged warning, for a session whose index camera or any recorded camera lacks a sensor capture time (a LiveKit stream with capture_monotonic_us). Failures are logged with the reason; a refusal leaves a hidden marker beside the aligned copy so it is not retried at every start (delete it to retry), while a retryable failure is. At startup the worker scans, on a thread, for kept episodes with no aligned copy whose evidence allows alignment, and removes staging directories left by a run killed more than a day ago. Shutdown drains the queue after the recorder closes, with a bounded wait, then stops any alignment still running; nothing partial is published.
• Tooling. recording/tools/align_episode.py aligns one kept episode by hand (exit 0 published, 1 refused, 2 retryable failure). An aligned session keeps one index camera; publishing an episode on a different one is refused. The end-of-day move and task-description tools keep a session's _aligned copy in step.
• Mock. capture-server --mock accepts --controller-delay-ms, --glove-delay-ms and --delay-jitter-ms, which make a source's content that old at the poll (seeded, bounded jitter).

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Testing

Run on the final head from examples/egocap:

• tests/recording: 677 passed, 3 skipped; 100% line and branch coverage of recording/egocap.
• tests/backend: 740 passed.
• tests/capture_server: 801 passed.
• tests/recording_service: 92 passed, 100 skipped (the skipped tests need EGOCAP_REAL_EPISODE set to a recorded episode).
• ruff check and ruff format --check; SKIP=check-copyright-year pre-commit run --all-files.
• Alignment output compared against independently generated expected arrays and an expected tree for several scenarios.
• Not run: a real station (cameras, headset, gloves) and a mock run through LiveKit with synthetic delays. Neither is covered here.

Known limitations / follow-ups

• The tracking delay is a configured estimate, not a measured sensor time; set alignment.tracking_latency_s to a value measured on the station. Physical timing accuracy is not claimed.
• Cameras are aligned only when their time is a qualified capture event (the LiveKit path with a monotonic capture basis). The plain capture-server --mock cameras carry no such evidence, so automatic alignment refuses those takes and logs why; running the mock cameras through LiveKit is a follow-up.
• Startup recovery re-queues only episodes whose timing evidence allows alignment; older recordings without it are aligned on request with the command-line tool.
• Each alignment starts a child process; long-take timing and memory have not been measured.
• The mock's delay options apply to the glove and controller content, not to camera frames.

Checklist

• I have read and understood the contribution guidelines
• I have run the linter and formatter with SKIP=check-copyright-year pre-commit run --all-files
• I have made corresponding changes to the documentation
• I have added tests that prove my fix/feature works (or explained why not)
• I have signed off all my commits (git commit -s) per the DCO

PR description drafted with Claude Sonnet 5.5.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: NVIDIA/IsaacCapture
• Branch: gmorgan/EgocapTiming
• File: .coderabbit.yaml
• Commit: 0a4625a
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 911a2c05-b0a9-4129-8337-9f75f079cf80

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[NVdejung]

🤖 Automated first-pass review (Claude, for Dejun) — I'll follow up personally where needed.

Summary — This PR adds recorder-side time alignment of kept episodes. Each take now stores per-source qualification and continuity evidence. There is a resampling aligner and a staged writer that never replaces an existing episode, a background worker in the backend that is on by default, a CLI, and synthetic tracking delays in the mock. The aligner and writer are careful and well tested. The blockers are in how the tracking latency is switched on and how it is labelled.

Must fix

1. station_config.py:140 — _resolve_latency now reads numeric_sources[].latency.
   • The station template has always filled that key with 0.05 s (ego_stations.example.yaml:87, :93). Its header called the value reference data "not consumed here", and the PR description calls it a placeholder.
   • After the upgrade, every station file copied from the template saves raw tracking capture_time as poll time − 50 ms, marked qualified, without anyone asking for it.
   • The key already means "capture-to-receive latency" in the dataset schema (data_recording/config.yaml:69). recording_service.py:744 writes the poll-based value back under that same key in the session config.yaml, so a reader that follows the schema would apply it a second time, or against receive time.
   • Fix: read the estimate from a new, explicit key (for example under alignment:). Remove the template values or mark them as unmeasured. Don't write the estimate as numeric_sources[].latency; the timing evidence already holds latency_s.
2. livekit_recording.py:515 — LiveKit takes still record "tracking_time": "host_poll", and the comment at L514 says the same. Whenever a latency is applied, this contradicts the merged event_time.<source>.meaning: host_poll_minus_latency. tracking_semantics.recorded_time was updated, but this key next to it was not. Fix: drop tracking_time from the media evidence (tracking evidence now comes from tracking_event_time), or derive it from the latency.

Suggestions

• Default index source (station_config.py:155): as far as I can tell, the default can never align a take.
  • The default is the first station video_sources entry, which is a CaptureServer camera. Those cameras never get an event_time entry, and every configured camera is a required source, so such a station refuses every take.
  • A LiveKit-only station gets no aligner unless index_source is set.
  • LiveKit streams only qualify with capture_monotonic_us, but LIVEKIT_PREVIEW.md:36, which the README now points to, documents monotonic_us as the default.
  • Consider turning alignment on only when index_source names a recordable stream, checked once at session open.
• Recovery (auto_align.py:77, app.py:443):
  • recover() runs synchronously at startup. It reads every manifest ever recorded and opens every unaligned kept episode.
  • It re-queues episodes refused for any reason other than qualification at every start, although the docstring (L80–81) says history is not retried. Run it in a thread and record refusals.
  • Kept results from recover_episode (recording_session.py:680) never go through _emit_result, so they are only aligned after a restart.
• Staging folders (aligned_writer.py:271): a killed run leaves a hidden _aligned.staging-* folder holding a full copy of the episode, and nothing removes it. The analytics skip only folder names ending in _aligned (count_daily_trajectories.py:167), so they count that copy, both after a crash and while an alignment is running. Remove stale staging folders in recover(), and have the analytics skip hidden folders.
• Aligned copies can go out of date:
  • The root index_source (aligned_writer.py:112) is rewritten on every publish, which relabels episodes that were aligned on a different index.
  • The end-of-day move and rename tools (e.g. eod_studio_session_move.py:85) update the raw group but not its _aligned sibling.
• Stale docs:
  • recording/README.md:130, :146 and episode_layout.py:185 still say config.yaml holds only meta.
  • alignment.py:22 and :69 still describe aligning on receive time − latency, which no caller does any more.
• Shared process: alignment runs in the recorder's own process, next to a live take. It uses plain Python loops (alignment.py:130) and holds every array of the episode in memory (aligned_writer.py:212). Please check this with a long take. The 30 s limit bounds the app's shutdown step, but the process likely still waits for the alignment thread to finish before it exits.
• Nits:
  • The _digest comment (recording_service.py:145) doesn't hold for the backend, which now sets frame types. A take left SEALED or CHECKED across the upgrade will be reported as recorded with a different configuration.
  • write_aligned_episode is exported and still replaces an existing episode (aligned_writer.py:226), which goes against the new AGENTS.md rule.
  • align_episode(epochs=…) is only used by tests.
  • In the mock, jitter is also applied to a source with zero delay and then clamped at 0, which biases that source late (mock_isaac_capture.py:763).
  • A bounded queue plus an unbounded backlog behaves exactly like one unbounded queue.
• Internal names (no action needed): none found in added lines, file names or commit text, and the fixtures are synthetic. The scan does not cover the 12 binary .npz fixtures; a pattern search over them found nothing.

Conflicts

• With gmorgan/TimestampAlignment: the PR is 0 commits behind and a local 3-way merge has no conflicts, so no rebase is needed.
• With open PRs: #1437 (8 shared files) and #1426 (10 shared files) are stack parents by the same author, with no textual conflicts.
  • Qualification here depends on #1437's timing_evidence() shape, its "sensor_capture" basis string and its clock-mapping epochs. Rebase this PR and re-run its tests after every change to #1437.
  • The stack also includes #1428 and #1429, both titled as drafts, so this PR cannot reach main before they do.

Change-by-change (59 files)

File / area	What changed	Makes sense?
AGENTS.md	Rule: derived output lives apart from the raw session; an existing derived episode is refused	✅ (exported write_aligned_episode still replaces)
README.md	Mock note: delay flags; plain mock cameras are kept but refused by alignment	⚠️ the LiveKit doc it points to doesn't give the basis alignment needs
backend/egocap_web/app.py	Passes the station latency into ServiceConfig; starts AutoAligner, chains on_episode_result, recovers at startup, waits for queued alignments on shutdown	⚠️ synchronous recover(); "bounded so shutdown is too" only covers the app's shutdown step
backend/egocap_web/auto_align.py (new)	One worker, queue plus backlog, startup recovery, bounded close	⚠️ retries refused episodes; never removes staging folders
backend/egocap_web/livekit_recording.py	tracking_event_time(); continuity from the mapping epoch on each frame; event_time per stream	⚠️ tracking_time is now stale (must-fix 2)
backend/egocap_web/recording_adapter.py	Frame types on the tracking specs; continuity on samples; latency taken off the poll time	⚠️ where the latency comes from (must-fix 1)
backend/egocap_web/recording_session.py	Latency read per session and passed into conversion; tracking timing evidence on every take, merged with the media evidence	✅ (imports the private _convert_raw_item)
backend/egocap_web/station_config.py	Resolves latency and alignment_index	⚠️ must-fix 1; default index source
capture_server/launch.py, server.py, mock/mock_isaac_capture.py	Delay and jitter flags in milliseconds; the mock reads controller and glove poses at their own delayed instants	✅ (jitter nit)
recording/egocap/recording/alignment.py (new)	Valid range, nearest-frame walk, pose/linear/nearest resampling, continuity and qualification checks	⚠️ stale docstrings; epochs only used by tests
recording/egocap/recording/aligned_writer.py (new)	Writer, validation, staged publish with one rename, binding to the kept episode, qualification read from timing	⚠️ leftover staging folders; root index_source
recording/egocap/recording/episode_layout.py	Manifest schema 4 with per-source continuity	✅ (SESSION_CONFIG_FILE comment now stale)
recording/egocap/recording/episode_store.py	NumericSourceSpec.frame_type	✅
recording/egocap/stations/ego_stations.example.yaml	Header now says latency is read; adds an alignment: section	⚠️ placeholder latencies now take effect (must-fix 1)
recording/recording_service/models.py	continuity on NumericSample and VideoSample	✅
recording/recording_service/recording_service.py	Continuity validated and recorded per take in the manifest; digest skips an unset frame type; session config.yaml always written with source declarations	⚠️ latency written under the schema's key; digest comment
recording/tools/align_episode.py (new)	CLI that aligns one kept episode	✅ (usage line omits --manifest-episode)
tests/backend/* (9 files)	Aligner worker, recovery and shutdown; app startup/shutdown wiring; latency and continuity on samples; evidence merge; station config	✅
tests/capture_server/* (3 files)	Delay flags, mock delay and jitter, source-factory wiring	✅
tests/recording/unit/*, tests/recording_service/* (9 files)	Aligner, writer, publication, binding to the kept episode, reader contract, continuity, session config	✅ skimmed, not read line by line
tests/recording/fixtures/* (19 files)	Synthetic raw inputs, expected trees and scenarios (JSON plus 12 binary .npz)	✅ JSON spot-checked for names only; binary files not read

_{Reviewed at c24fae9 · 59 files · 2026-11-09 13:23 PST}

[gareth-morgan-nv]

🤖 Agent-generated (Claude · claude-sonnet-5-5)
Re: @NVdejung's review — #1457 (review)

Thanks for the review. Addressed in b3fdc2d and 61e516b.

Must fix

1. ✅ The tracking estimate is now read only from alignment.tracking_latency_s (per source, finite and ≥ 0; an unknown source name or wrong type is a station-config error). numeric_sources[].latency is reference data again, the template ships no value, and the estimate is no longer written under the dataset's latency key (the timing evidence keeps latency_s).
2. ✅ The stale tracking_time entry is removed from the LiveKit timing evidence.

Suggestions

• ✅ Recovery: the scan runs on a thread at worker start, and a refusal leaves a hidden marker beside the aligned copy so it is not retried every start (a retryable failure writes none). Kept results from recover_episode now reach the aligner through a new on_episode_kept hook.
• ✅ Staging folders: the analytics skip hidden folders, and the scan removes staging directories older than a day.
• ✅ Aligned copies: an aligned session refuses an episode on a different index camera; the end-of-day move moves the _aligned sibling; the task-description tool updates it.
• ✅ Docs: alignment.py, recording/README.md, episode_layout.py, the _digest comment and the CLI usage line.
• ✅ Shared process: each alignment now runs as a low-priority child process of the command-line tool, so it does not share the recorder's interpreter and is killed at shutdown. Not yet measured on a long take.
• ✅ Nits: the mock no longer jitters a zero-delay source; the bounded queue plus backlog is now one unbounded queue.
• 🔍 write_aligned_episode still replaces an existing episode: it is the low-level formatter used by tests and the staged publish, which is the only path that refuses. Left as is.
• 🔍 align_episode(epochs=…) is only used by tests; left for now.

Tests, lint and the full pre-commit run pass on the pushed head. Still not covered: a long-take measurement, a real station, and the mock cameras over LiveKit.

[gareth-morgan-nv]

🤖 Agent-generated (Claude · claude-sonnet-5-5)
Re: @NVdejung's review — #1457 (review)

Correction to my earlier reply: it left out three items. They are fixed in 3fc413d.

• ✅ Default index source: the app tells the aligner which cameras each session records, once when the session opens. While the configured index camera is not among them, kept episodes are not queued for alignment and one warning is logged; it resumes when a later session records it.
• ✅ LiveKit doc: LIVEKIT_PREVIEW.md now says automatic alignment needs capture_monotonic_us; with the default monotonic_us (frame-ready time) the take is recorded but not aligned.
• ✅ _digest: a source's frame type is no longer part of the configuration digest (it says how fields are resampled, not how data is laid out), so a take left SEALED or CHECKED across the upgrade is not reported as recorded with a different configuration.

Still open and stated in the description: no long-take timing or memory measurement, no real-station run, and the mock cameras over LiveKit.

[gareth-morgan-nv]

🤖 Agent-generated (Claude · claude-sonnet-5-5)
Re: @NVdejung's review — #1457 (review)

Second correction: re-reading the review against the code turned up more gaps. Fixed in 1b80c17.

• ✅ Default index source. My earlier check only confirmed the index camera was recorded, which a plain CaptureServer camera passes, so the default could still never align. Now there is no default index camera: alignment is on only when alignment.index_source is set. At each session open it is skipped, with one warning, unless the index camera and every recorded camera have a sensor capture time (a LiveKit stream with capture_monotonic_us).
• ✅ recording/README.md:146 still said config.yaml carries no source fields; corrected.
• ✅ write_aligned_episode no longer replaces an existing episode; it refuses, like the staged publish.
• ✅ align_episode(epochs=…) removed (only tests used it).
• ✅ recording_session.py no longer imports a private name (convert_raw_item).

Not done: the long-take timing and memory measurement of the child-process aligner. Also still open, as stated in the description: a real-station run and the mock cameras over LiveKit.