[docs-agent] Add Try It method audit (DOCS-101)#1340
Closed
alchemy-bot wants to merge 1 commit into
Closed
Conversation
Live-probes every method's canonical example payload across the 91 OpenRPC specs in src/openrpc/, classifies the failures, and surfaces three single-edit quick wins. Catalog lives at audits/2026-05-29-try-it-audit/. Headline numbers: 3,749 probed, 2,604 ok, 842 failed (24.4%), 303 skipped (websocket-only or no example). Refs DOCS-101 Requested-by: @0xmts
🔗 Preview Mode
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c4d6cb10b0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Collaborator
|
good report but we don't wanna merge it, we can mark this task done! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a categorized catalog of every failing "Try It" example across the JSON-RPC OpenRPC specs under
src/openrpc/(chain specs + Alchemy product specs).This was requested in the #dx-docs-feedback Slack thread where Caleigh flagged that
trace_*anddebug_*Try It examples were failing for ~50% of methods. Rather than fix in isolation, this catalog probes every method's literal example payload against the documented endpoint and groups the failures so cluster fixes can be prioritized.Headline numbers
Three single-edit quick wins called out in the report
eth_getProofshared example has an invalid hex character —0x283s34c8e2b1...includes a literals, breaking the example on ~29 EVM chains. One edit insrc/openrpc/chains/_components/custom/methods.yaml.eth_submitWorkis referenced from ~64 chain specs that aren't PoW — every one returnsdoes not exist/is not available.eth_getAccount(Erigon-only) is referenced from ~44 chain specs that don't run Erigon — same shape.The full categorized breakdown, per-spec failure rates, and methods-failing-on-N+-chains tables are in
audits/2026-05-29-try-it-audit/report.md.failures.csvhas one row per failure for spreadsheet pivot work. Scripts to reproduce are checked in alongside.What this PR does NOT do
src/openapi/). Needs a per-spec adapter; flagged as follow-up.Linear
DOCS-101 — Full Try It audit: catalog failing examples across Node API + REST specs
Requested by
@0xmts (via Slack thread)