Ingest progress & status: per-phase progress in the UI and CLI (#64)

[gangtao]

Closes #64.

What & why

Ingest gave almost no live feedback — the UI's Ingest jobs panel sat on an indeterminate bar + running…, and tpk ingest printed one line then went silent through minutes of extraction, so both surfaces looked stuck. This threads one progress mechanism through the pipeline and lets both surfaces consume it.

How it works

• graphify tee (graphify_runner.py) — run_graphify switched from subprocess.run to Popen, reading graphify's stdout line-by-line and forwarding each line to an on_line callback (the live heartbeat for the long extract phase). PYTHONUNBUFFERED=1 in the child env forces line-flushing over the pipe. stderr is merged into stdout and a bounded tail is kept for the error detail.
• Progress callback (ingest.py) — IngestProgress(phase, message, nodes, edges) + ProgressFn, threaded through ingest_repo as on_progress. Phases: fetch (github repos) → extract (one event per graphify line) → parse → upsert → done; node/edge counts from parse onward. Default None = no-op.
• JobManager / API (api.py) — the in-memory job record gains phase/message/updated_at and updates nodes/edges live, all under the existing lock; /api/jobs exposes them (additive, no endpoint change).
• UI (Manage.tsx) — active jobs show a phase label (extracting…/saving…/…), live node/edge counts, elapsed via a local 1s ticker (advances between the 5s polls), and a "stalled?" hint when updated_at is older than 60s. Indeterminate bar and finished line unchanged.
• CLI (cli.py) — a refreshing single status line on a TTY ([i/N] key · extracting… · 2m30s · <last graphify line>), throttled plain lines when piped, and --verbose still streams graphify's full output.

Design decisions (honest-indeterminate)

graphify is an opaque subprocess with no reliable per-file total, so there's no fabricated determinate bar and no processed/total — phase + elapsed + live heartbeat + counts. The heartbeat is driven by graphify's own stdout, so a genuine hang stops updated_at and the UI honestly reads "stalled" (a timer would report false liveness). Jobs stay in-memory (no persistence), as before.

Behavior change to call out

tpk ingest --verbose now line-tees graphify's output (piped) rather than inheriting the terminal. Consequence: graphify likely disables color and any \r-updated progress bar collapses to line-by-line output. In exchange, the error path now captures graphify's real output tail even under --verbose. This is inherent to the tee design.

Testing

• DB-free suites: 21 passed, 2 skipped (the 2 skips are DB-gated CLI integration tests). Covers: graphify Popen tee (line forwarding, PYTHONUNBUFFERED, non-zero-exit tail), the ingest_repo phase sequence, the JobManager progress sink (tested directly, no DB), and the CLI _progress_line formatter.
• Frontend tsc -b && vite build clean.

Deferred to manual verification (not runnable headlessly)

• graphify buffering / live heartbeat — a real tpk ingest --repo <small> to confirm graphify's lines stream live rather than batching at the end (validates the PYTHONUNBUFFERED=1 assumption). If graphify batches on a pipe, the heartbeat degrades to phase-boundary granularity — still an improvement, no code path breaks.
• UI layout eyeball — the running-status line and the (now left-aligned) queued row, across various count/elapsed lengths. Build is green; logic and CSS-token reuse were statically reviewed.

🤖 Generated with Claude Code