Add a CI check for the Python code samples on the docs pages - #2195
Open
GWeale wants to merge 1 commit into
Open
Add a CI check for the Python code samples on the docs pages#2195GWeale wants to merge 1 commit into
GWeale wants to merge 1 commit into
Conversation
Nothing currently checks the code in the documentation. python-lint.yaml and python-tests.yaml are both filtered to samples/python/**, and no samples directory exists, so on pull requests neither can run. Samples that do not parse, and samples importing names google-adk no longer provides, have shipped as a result. This adds a checker that reads the fenced Python blocks straight out of the markdown and asks two things of each: does it parse, and do its google.adk imports resolve against the installed package. No sample is executed. They build agents and call live endpoints, so running one in CI would be slow and billable; importing the library is enough to answer both questions. Samples are frequently excerpts rather than whole modules, so a plain parse failure is not automatically a defect. Method bodies, blocks that only make sense indented under something shown above them, pseudocode using ... as a placeholder, and signature listings with no body are all recognised and allowed through. The checker ships with tests, half of which plant a defect of each class it claims to catch. A checker that reports nothing because it is broken looks exactly like one that reports nothing because the samples are fine, and only the planted cases tell those apart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Nothing currently checks the code in the documentation.
python-lint.yamlandpython-tests.yamlare both filtered tosamples/python/**, and there is nosamples/directory in the repository, soon a pull request neither workflow can ever run. Java and TypeScript snippets
have no check either. Only the Go and Kotlin snippet files under
examples/are built, and those are a small fraction of the code a reader actually sees.
The consequence is samples that cannot work reaching the site. Run against
maintoday, this check reports eight, including an unindented class body onthe Plugins page that raises
IndentationErroron the first line a readerwould paste, and two pages importing
CodeExecutionInputfrom a module thatdoes not contain it.
What it does
Reads the fenced Python blocks straight out of the markdown and asks two
questions of each:
google.adkimports resolve? This is what rots quietly when thelibrary renames or moves something, because the prose keeps reading correctly
while the code stops working.
No sample is executed. They build agents and call live Gemini and Vertex
endpoints, so running one in CI would be slow, billable and flaky. Importing
the library is enough to answer both questions, and that is all it does.
Not every parse failure is a defect
Samples are frequently excerpts, and treating those as errors would make the
check unusable. Four shapes are recognised and allowed through:
await,breakorcontinuefrom inside a loop...or<...>standing in for omitted code, or!pip installdeforclassshown with no body, to describe an interfaceOn
mainthat leaves 8 reports out of 623 Python blocks, and I believe all 8are genuine. Seven are fixed by #2194.
Trusting the result
Half the tests plant a defect of each class the checker claims to catch, and
assert it is caught. A checker that reports nothing because it is broken looks
exactly like one that reports nothing because the samples are fine, and only
the planted cases tell those apart. The other half assert the excerpt shapes
above are not reported, which is what keeps it from crying wolf.
There are also parser tests for the two cases that caused real trouble: a
sample quoting fenced markdown inside a string, and a fence closed at the wrong
indentation.
Ordering
This check fails on
mainuntil the samples are fixed, so it should landafter #2194, which fixes all eight. I am happy to hold it, or to fold it
into that pull request, whichever you prefer.
Scope
Python only, since that is the largest body of samples and the one with no
coverage at all. The structure takes another language by adding a check
function; I did not do that here to keep the change reviewable. Java, Kotlin
and Go would each need their toolchain installed in the workflow, which is a
larger conversation about CI minutes.