HydraFusion: emit per-phase model, verdict and credit attributes to OpenTelemetry

[samueltauil]

Describe the feature or problem you'd like to solve

A HydraFusion turn can run several models, but from the outside I only get one answer and one credit figure. The CLI already writes every routing decision down in ~/.copilot/session-state/<id>/events.jsonl. None of it reaches OpenTelemetry or the UI, so the only way to see which model did what is to read session state, which carries no compatibility promise.

Proposed solution

What the CLI emits today

I measured this on 1.0.84-2 with COPILOT_OTEL_FILE_EXPORTER_PATH, no collector involved. Every turn arrives as one invoke_agent span, several chat hydrafusion spans, and execute_tool spans. Every one of those chat spans carries gen_ai.request.model=hydrafusion.

That is correct. The GenAI semantic conventions define gen_ai.request.model as the model the client asked for, and the client did ask for hydrafusion. The conventions have no notion of a router that expands one request into several model calls, so there is nowhere for the per-phase detail to sit. This is a spec that has not caught up with the architecture, not a defect.

The metric side has the same shape. Six instruments, no token metric, no credit metric, and gen_ai.request.model is always hydrafusion. A turn that spent 51 credits across two models from two vendors is indistinguishable from a cheap single-model turn except by duration.

Where the data already lives

All of it is in the session log, fully structured. One event per phase:

{"phaseKind":"judge","role":"judge","model":"gpt-5.6-sol","status":"succeeded",
 "verdict":"reject","durationMs":4522,
 "usage":{"requestCount":1,"inputTokens":12500,"outputTokens":7,
          "cachedTokens":7680,"cacheWriteTokens":0,"totalNanoAiu":2249200000}}

Plus session.fusion_resolved for the routing decision (pattern, policy, routeSource, routingLatencyMs, the phase plan), session.fusion_handoff for escalations, and session.fusion_completed for the turn rollup. Per-phase credits sum to the turn total exactly, to the nanounit, across every turn I checked.

So the instrumentation work is basically done. The values are computed, named and written to disk. They just never leave the process through a supported channel.

1. One span per fusion phase

Emit a child span per phase under the turn root, carrying the model that actually served it:

invoke_agent                        github.copilot.fusion.pattern = cascade
└─ chat hydrafusion                 gen_ai.request.model  = hydrafusion
   ├─ phase 1                       gen_ai.response.model = mai-code-1.1-flash
   │                                github.copilot.fusion.phase.kind = primary
   │                                github.copilot.nano_aiu = 575436000
   ├─ phase 2                       gen_ai.response.model = gpt-5.6-sol
   │                                github.copilot.fusion.phase.kind = judge
   │                                github.copilot.fusion.phase.verdict = reject
   │                                github.copilot.nano_aiu = 2457200000
   └─ phase 3                       gen_ai.response.model = gpt-5.6-sol
                                    github.copilot.fusion.phase.kind = repair
                                    github.copilot.fusion.phase.committed = true
                                    github.copilot.nano_aiu = 11686960000

gen_ai.response.model needs no spec change. The conventions already define it as the model that generated the response, and per phase that is unambiguous. Everything router-specific goes under a github.copilot.fusion.* namespace, which keeps it out of the way if the WG later standardises orchestration attributes. If GitHub wanted to take that shape to the semconv group as a routing extension, there is now a working implementation and 24 turns of data behind it.

Suggested attributes:

Attribute	Source field
gen_ai.response.model	model on the phase event
github.copilot.fusion.pattern	pattern
github.copilot.fusion.phase.kind	phaseKind
github.copilot.fusion.phase.role	role
github.copilot.fusion.phase.verdict	verdict
github.copilot.fusion.phase.committed	derived from finalSourceModel
github.copilot.fusion.route_source	routeSource
github.copilot.fusion.routing_latency_ms	routingLatencyMs
github.copilot.nano_aiu	usage.totalNanoAiu

Metadata only. No prompt or response content needs to cross this boundary.

2. Token and credit metrics dimensioned by serving model

gen_ai.client.token.usage is already a standard semconv histogram and the numbers are all in the phase events. A credit metric with the same dimensions would let people answer cost questions without running a trace backend at all.

3. Something in the CLI itself

Not everyone will stand up a collector. A /usage view, or an expandable breakdown on the turn that just finished, showing the phases, who drafted, what the reviewer said and where the credits went. The announcement explains that intermediate drafts are held back so unfinished work does not read as final, which is the right call while the turn is running. After it finishes, I would like to be able to ask.

4. An official reference dashboard

Once the attributes exist, ship a Grafana dashboard JSON and a collector config alongside the docs. Right now anyone who wants this reverse-engineers their own, and they will all diverge. One maintained dashboard in-repo settles the question of what the panels are supposed to mean.

Why I think this is worth doing

Three of these were surprises I could only get at by reading the log, and they are the kind of thing I would expect an org paying for credits to ask about:

• Across 24 turns, the phases that did not supply the answer came to 2.5% of spend. I would not have guessed that low, and I could not have found it from the CLI output.
• One model took 42% of total credits from three phases, all of them heavy refactors. Per-model attribution is invisible today because every span says hydrafusion.
• routeSource was capi_plan on all 24 turns, meaning routing policy is served remotely and can change without a CLI release. Without per-phase telemetry, a policy change lands as an unexplained shift in the credit line, with nothing to diff against.

I built a working version to prove the data supports it: samueltauil/hydrafusion-traces. It tails the session log and rebuilds each turn as a trace. It is roughly 400 lines of stdlib Python, and it will break the first time the log format moves, which is exactly the argument for doing this properly upstream.

Example prompts or workflows

1. "Why did this turn cost 51 credits?" Today the answer is one number. With per-phase spans it is a draft phase at 51.11 and a critic phase at 0.69, and you can see immediately that the review was cheap and the draft was not.
2. Finance asks which models an org's credits went to this month. Currently unanswerable from telemetry, because every span reports hydrafusion.
3. A routing policy changes server-side and credit spend jumps 30%. With github.copilot.fusion.pattern and route_source on spans, you can see the mix shift. Without them you can only see the bill.
4. "Is the quality gate paying for itself?" Judge rejection rate needs phase.kind=judge and a verdict. My sample had 2 rejections out of 3, which is far too small to mean anything, and that is the point: nobody can gather this at scale right now.
5. A turn returns a weak answer and you want to know which model committed it. phase.committed answers it in one click.

Additional context

The dashboard I ended up with, for reference on what the data supports:

One cascade turn as three phases. mai-code-1.1-flash drafted for 22 seconds and 0.58 credits, gpt-5.6-sol reviewed it, returned reject, then redid the work itself in 42 seconds and 11.69 credits. The CLI showed one answer and the figure 14.72:

The whole ask sits on two attribute rows. gen_ai.request.model is hydrafusion, per the conventions. gen_ai.response.model is gpt-5.6-sol, which my tailer added from the session log. Both are true, and only the first one is available today:

Credits split by phase kind, which is the question I actually cared about and could not answer any other way:

Caveats on my end. This is one machine, one operator, 24 turns, one small Python project. It describes what the telemetry contains and shows the dashboard works. It is not a benchmark and says nothing about model quality. HydraFusion is a research preview and is expected to change. Model identifiers above are routing labels I observed, not product names.

Measurements and method: docs/SPIKE.md for what the CLI actually emits, docs/FINDINGS.md for the 24-turn breakdown, and a write-up of how I got there.

Possibly related:

• Desktop app: enterprise managed settings ignored for telemetry, and exported spans omit GitHub repo/org/branch attributes #4782, desktop app spans omitting GitHub context attributes. Different surface, same general theme of telemetry not carrying enough to attribute work.
• OTel: post-reload resumed turn emits invoke_agent root without input messages #4726, invoke_agent roots missing input on resumed turns.

Happy to split this into separate issues for the span work, the metrics, the in-product view and the dashboard if that suits triage better. I kept them together because the last three depend on the first.