Add model-context-graph skill: autopilot studies on new model/framework releases - #3
Open
buildwithtalia wants to merge 7 commits into
Open
Add model-context-graph skill: autopilot studies on new model/framework releases#3buildwithtalia wants to merge 7 commits into
buildwithtalia wants to merge 7 commits into
Conversation
…rk releases New /devrel-skills:model-context-graph skill that fires within an hour of a new AI model or coding framework/harness drop, benchmarks it against Postman's context graph on API + coding + agentic-autonomy suites, generates a study with charts and token-optimization data, posts to all Postman social accounts, and regenerates the AI harness config to use the new model. Co-Authored-By: Claude <noreply@anthropic.com>
Adds a fourth benchmark suite (downstream-update-tasks) that measures how the context graph finds the exact set of consumers affected by an upstream API change, and generates correct patches for each. Includes recall, precision, and patch-test pass rate; a scatter chart of recall vs precision; and a stacked bar showing tokens split into discovery vs patching so the discovery-cost delta is legible. Co-Authored-By: Claude <noreply@anthropic.com>
Renames the skill directory, frontmatter name, slash-command references, output directory, and README entry from model-context-graph to model-context-graph-comparison. Co-Authored-By: Claude <noreply@anthropic.com>
The skill talks to the context graph through a forthcoming Postman API that is not GA yet (expected mid-August 2026). Adds: - A dedicated "Context Graph API — placeholder" section documenting the expected base URL, four endpoints, and a swap-in checklist for the day the API ships. - POSTMAN_CONTEXT_GRAPH_API_KEY and POSTMAN_CONTEXT_GRAPH_API_URL env vars in the required-env table, both flagged as placeholders. - Stub behavior: until GA, the client returns fixture data from references/fixtures/context_graph/ so the pipeline still runs end-to-end. - Harness tool schemas expanded to include context_graph.query, context_graph.consumers, and context_graph.diff, each pinned to the postman.context_graph.v1.stub schema ref until release. Co-Authored-By: Claude <noreply@anthropic.com>
Resolve README.md conflict by keeping both new skill rows (model-context-graph-comparison and content-metrics).
The prior Stage 2 said "run the model on four task suites" and jumped
straight into per-suite tables and delta compute lists, but never said
what the skill actually calls or where the run happens. Two missing
pieces addressed:
1. How the skill invokes the harness. New subsections 2.1 (invoke),
2.2 (what runs today), 2.3 (what comes back), 2.4 (compute the
tagline deltas from that return shape).
- Preferred invocation: repository_dispatch on
buildwithtalia/ai-harness with client_payload
{model, adapter, releaseUrl, dispatchedBy}. Concrete `gh api
dispatches` snippet included.
- Alternative: workflow_dispatch equivalent.
- Payload field table.
- Note that on-model-release.yml also commits the adapter bump
back to main — that's what completes our Stage 5.
2. Correct the four-suite framing. The harness ships one suite
(`agent-benchmark`, 12 cases across build / find / ask), not the
four-suite api-tasks / coding-tasks / autonomy-tasks /
downstream-update-tasks design that only existed on paper.
- Section 2.2 documents the actual current 12-case shape with
categories, difficulty tiers, capability-axis tags, and the
deterministic + LLM-judge scoring model.
- Section 2.5 preserves the four-suite table as "planned
expansion (v2 of this pipeline)" with a clear note that the
harness has scaffolding for it (groundTruth.checks[], Zod
structured-output) but hasn't populated the tasks yet.
Section 2.3 also spells out the results/skill-input.json shape the
skill consumes — including that every providerDeltas row is keyed on
(agent, model, providerId) so the tagline can be attributed per
triple — and section 2.4 maps success_delta / token_delta_pct /
cost_per_success_delta / autonomy_delta to specific fields in that
JSON (or in cases.jsonl in the artifact).
Co-Authored-By: Claude <noreply@anthropic.com>
The old chart list and post plan referenced suites and metrics that don't exist in the harness today (chart 5 was recall vs precision on downstream-update-tasks; chart 6 was migration-cost stacked bar; tweet 4 embedded chart 5). None of those metrics come out of the 12-case agent-benchmark suite. Rewrite around the fields the harness actually returns in results/skill-input.json and runs/<id>/cases.jsonl: - Stage 3 required charts are now six deliverables all derivable from the current shape: 1. Bar — pass rate per (agent, model) baseline vs +cg 2. Grouped bar — output tokens per passed case 3. Bar — cost per passed case 4. Grouped bar — mean score by category (build / find / ask) 5. Scatter — quality (meanScore) vs latency (p50LatencyMs) 6. Radar — judge dimensions from cases.jsonl (optional) Every chart cites its source field so the chart and the number cannot disagree. - Multi-provider runs (harness supports +orbit alongside +cg) fan out charts 1–5 into one panel per provider. - Charts blocked on the v2 four-suite pipeline are moved into a separate "blocked" list at the end of Stage 3: cumulative goal completion over tool-call budget, consumer recall/precision scatter, discovery-vs-patching stacked bar. All annotated with their v2 dependency. - Stage 4 study skeleton reorganised around the three actual categories (build / find / ask) and the delta-attribution model (per (agent, model, provider) triple). The old "four benchmarks" section (API / coding / autonomy / downstream) is gone. New "Latency cost" and "Judge dimensions" sections consume chart 5 and chart 6. - Stage 6 Twitter thread rewrites the shot list around the actual charts (1 → pass rate, 2 → tokens, 4 → category, 5 → tradeoff) and points the outro tweet at triggerContext.workflowRunUrl so anyone can see the raw run. - Guardrails: "three suites" → "any category (build/find/ask)" and the halt condition points at meanScoreDelta / passRateDelta from providerDeltas, which is where the numbers actually live. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/devrel-skills:model-context-graphskill that fires within an hour of a new AI model or coding framework/harness release.data.csvso numbers and pictures can't disagree.context_graph.query,http_request,collection_run), and guardrails (sandboxing, tool-call budget, secrets redaction).blog-wordpress-stage, and Discord#announcements. Missing credentials skip a channel rather than blocking the run.Design points worth calling out
model-context-graph-output/seen.json.supercharge,unlock,revolutionize,leverage,game-changing,revolutionary)./devrel-skills:content-metricsand appendsimpact_scoreto the run log. Feeds directly into deciding whether the next drop gets the same treatment.Files
skills/model-context-graph/SKILL.md(new)README.md— one-row addition in the Other Skills tableTest plan
ANTHROPIC_API_KEY,OPENAI_API_KEY,GOOGLE_API_KEY,POSTMAN_API_KEY, Twitter creds, LinkedIn creds, WordPress creds)/devrel-skills:model-context-graph --dry-runand confirm all 6 stages complete without publishing/devrel-skills:model-context-graph --now "Claude 5.1 Opus"and inspectstudies/<date>-<slug>.md+ chartsharness/<date>-<slug>.jsonis emitted with sane sampling params derived from the benchmark sweepseen.jsonprevents duplicate postsGenerated with Claude Code