export buildkit history and structured lifecycle telemetry at teardown

[piob-io]

Summary

At teardown, the action exports the job's BuildKit build history and ships it — with structured lifecycle telemetry — to the vm-agent's ReportDockerBuild RPC. The guest ships raw bytes; all semantic parsing lives host-side in the fa agent so BuildKit format drift is absorbed by an agent deploy, not an action release.

What ships

• BuildKit history records: Control.ListenBuildHistory({EarlyExit:true}) streams the job's BuildHistoryRecords. Each record is decoded as a typed protobuf-es message only to read Ref (for post-export deletion), CompletedAt (incomplete flag), and the trace descriptor, and to clear logs; it is then re-serialized and shipped opaquely (protobuf-es preserves unknown fields across the round trip, so fields newer than our vendored proto still reach the agent). Records are pruned via Control.UpdateBuildHistory(Delete:true) after export.
• Solve trace attachments: fetched from buildkitd's content store via Content.Read(digest), size-capped (2 MiB per trace; oversized → shipped without trace, marked truncated).
• Runner step timeline: newest _diag/Worker_*.log read verbatim, tail-capped at 1 MiB; parsing is agent-side.
• Lifecycle facts: builder mode + fallback reason, commit decision + skip reason, integrity outcome/duration, buildctl du -v cache breakdown (per-mount sizes), fs used/size, prune decision/bytes, hotload/buildkitd-ready/shutdown timers, SIGKILL flag, export-timeout/prune-failed flags.

Chunked reporting

A whole-job report can exceed the vm-agent's default 4 MiB gRPC MaxRecvMsgSize (staging: a 24-build job produced a 6.3 MB message rejected with resource_exhausted). reportDockerBuild therefore splits builds across multiple unary ReportDockerBuildRequest calls via chunkBuilds, with a ~3 MiB per-request budget covering timeline + record + trace bytes:

• the runner step timeline ships in every chunk (workflow-step attribution is per-request server-side);
• the lifecycle message ships only in the first chunk (the agent emits lifecycle metrics per request carrying it);
• every chunk carries at least one build, so a single build at the per-record/trace caps (~2.5 MiB) can slightly exceed the budget while staying under the 4 MiB limit;
• each chunk gets its own report timeout; the whole path stays fail-soft (telemetry never fails the customer job).

Protos

BuildKit control.proto (+ deps) and containerd content.proto are vendored verbatim from pinned upstream releases (buildkit v0.32.2, containerd v2.3.4, googleapis pinned commit) via scripts/fetch-protos.sh; stickydisk.proto comes from buf.build/blacksmith/vm-agent. Generation is dev-time only: buf generate → src/gen/ → pnpm build → checked-in dist/. Nothing is fetched or compiled during an action run, and no hand-authored proto content or hand-rolled wire parsing remains.

Ordering

Post-step teardown: du snapshot → history export → history prune → buildkitd shutdown → integrity check → fs usage → unmount → commit/skip decision → report.

Identity

The guest sends only expose_id, vm_id, git sha/branch, and build facts; keys, entity, run/job attribution are stamped host-side by the vm-agent.

Companion PRs: agent-side parsing/insertion FastActions/fa#5232 (merged) + OTLP JSON trace-parser fix FastActions/fa#5277; ClickHouse schema useblacksmith/web#11965 (merged).

Link to Devin session: https://app.devin.ai/sessions/0c6ebc2a6de54933bebe821e21ab8b76
Open in Devin Desktop: https://app.devin.ai/desktop/session/0c6ebc2a6de54933bebe821e21ab8b76?variant=devin
Requested by: @piob-io

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[ajwerner]

is this valid? The sticky disk as a block io device doesn't know about which logical files have been deleted, right?

[devin-ai-integration]

Right — the block device has no notion of file deletion, so residual bytes can remain in freed blocks after commit; we don't claim physical scrubbing. What the delete does guarantee is filesystem-level: it happens via buildkitd's UpdateBuildHistory{Delete} API (BoltDB record delete + content-store blob unref) before buildkitd shutdown → integrity check → unmount → commit, so the committed filesystem no longer references those records. The purpose is (1) history.db doesn't grow unboundedly across runs and (2) the next job's export doesn't re-export prior jobs' records — not data erasure. I've reworded the comment to say exactly that (was misleadingly phrased as "never get committed").

[ajwerner]

@devin do we bound the size of the report we generate in terms of bytes? I think we need a way to hard bound it somewhere around 10s of MiB and under 128 MiB

[devin-ai-integration]

Yes — the report is hard-bounded to ~10 MiB worst case, enforced at assembly time in src/build-telemetry.ts:

• MAX_RECORD_BYTES = 512 KiB per history record (oversized records ship without payload, marked truncated)
• MAX_TRACE_BYTES = 2 MiB per solve-status trace (dropped + truncated if over)
• MAX_TOTAL_PAYLOAD_BYTES = 8 MiB running cap across all records+traces combined — once hit, remaining builds ship metadata-only
• MAX_TIMELINE_BYTES = 1 MiB tail-cap on the runner _diag step timeline

Everything else in the report (lifecycle facts, du totals, per-cache-mount rows) is small fixed-size fields, so the whole ReportDockerBuild request tops out around 9–10 MiB and is typically a few hundred KiB. Well under the 128 MiB ceiling and inside the "10s of MiB" target.