Skip to content

Add a CI check for the Python code samples on the docs pages - #2195

Open
GWeale wants to merge 1 commit into
google:mainfrom
GWeale:check-code-samples
Open

Add a CI check for the Python code samples on the docs pages#2195
GWeale wants to merge 1 commit into
google:mainfrom
GWeale:check-code-samples

Conversation

@GWeale

@GWeale GWeale commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Why

Nothing currently checks the code in the documentation.

python-lint.yaml and python-tests.yaml are both filtered to
samples/python/**, and there is no samples/ directory in the repository, so
on a pull request neither workflow can ever run. Java and TypeScript snippets
have no check either. Only the Go and Kotlin snippet files under examples/
are built, and those are a small fraction of the code a reader actually sees.

The consequence is samples that cannot work reaching the site. Run against
main today, this check reports eight, including an unindented class body on
the Plugins page that raises IndentationError on the first line a reader
would paste, and two pages importing CodeExecutionInput from a module that
does not contain it.

What it does

Reads the fenced Python blocks straight out of the markdown and asks two
questions of each:

  • Does it parse? A sample a reader cannot paste into a file is a bug.
  • Do its google.adk imports resolve? This is what rots quietly when the
    library renames or moves something, because the prose keeps reading correctly
    while the code stops working.

No sample is executed. They build agents and call live Gemini and Vertex
endpoints, so running one in CI would be slow, billable and flaky. Importing
the library is enough to answer both questions, and that is all it does.

Not every parse failure is a defect

Samples are frequently excerpts, and treating those as errors would make the
check unusable. Four shapes are recognised and allowed through:

Shape Example
fragment a method body shown without its class
loop excerpt a block using await, break or continue from inside a loop
pseudocode ... or <...> standing in for omitted code, or !pip install
signature listing a def or class shown with no body, to describe an interface

On main that leaves 8 reports out of 623 Python blocks, and I believe all 8
are genuine. Seven are fixed by #2194.

Trusting the result

Half the tests plant a defect of each class the checker claims to catch, and
assert it is caught. A checker that reports nothing because it is broken looks
exactly like one that reports nothing because the samples are fine, and only
the planted cases tell those apart. The other half assert the excerpt shapes
above are not reported, which is what keeps it from crying wolf.

There are also parser tests for the two cases that caused real trouble: a
sample quoting fenced markdown inside a string, and a fence closed at the wrong
indentation.

python -m pytest tools/code_samples/check_test.py   # 17 passed
python tools/code_samples/check.py                  # whole tree
python tools/code_samples/check.py docs/plugins/index.md   # one page

Ordering

This check fails on main until the samples are fixed, so it should land
after #2194, which fixes all eight. I am happy to hold it, or to fold it
into that pull request, whichever you prefer.

Scope

Python only, since that is the largest body of samples and the one with no
coverage at all. The structure takes another language by adding a check
function; I did not do that here to keep the change reviewable. Java, Kotlin
and Go would each need their toolchain installed in the workflow, which is a
larger conversation about CI minutes.

Nothing currently checks the code in the documentation. python-lint.yaml and
python-tests.yaml are both filtered to samples/python/**, and no samples
directory exists, so on pull requests neither can run. Samples that do not
parse, and samples importing names google-adk no longer provides, have shipped
as a result.

This adds a checker that reads the fenced Python blocks straight out of the
markdown and asks two things of each: does it parse, and do its google.adk
imports resolve against the installed package. No sample is executed. They
build agents and call live endpoints, so running one in CI would be slow and
billable; importing the library is enough to answer both questions.

Samples are frequently excerpts rather than whole modules, so a plain parse
failure is not automatically a defect. Method bodies, blocks that only make
sense indented under something shown above them, pseudocode using ... as a
placeholder, and signature listings with no body are all recognised and
allowed through.

The checker ships with tests, half of which plant a defect of each class it
claims to catch. A checker that reports nothing because it is broken looks
exactly like one that reports nothing because the samples are fine, and only
the planted cases tell those apart.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant