Skip to content

feat: documentation snippet testing with literalinclude and Pydantic validation#704

Draft
raballew wants to merge 74 commits into
jumpstarter-dev:mainfrom
raballew:497-doc-snippet-testing-literalinclude
Draft

feat: documentation snippet testing with literalinclude and Pydantic validation#704
raballew wants to merge 74 commits into
jumpstarter-dev:mainfrom
raballew:497-doc-snippet-testing-literalinclude

Conversation

@raballew
Copy link
Copy Markdown
Member

Summary

  • Extract inline code snippets from 44 driver READMEs into standalone files under each driver's examples/ directory, referenced via {literalinclude} directives
  • All YAML configs use full Kubernetes-style syntax (apiVersion, kind, metadata) and are validated against Pydantic models (ExporterConfigV1Alpha1, HookConfigV1Alpha1, etc.)
  • Python examples are syntax-checked via compile() and import-checked via importlib
  • Shared test infrastructure in jumpstarter.testing.checks (file discovery, unused/inline detection) and jumpstarter.testing.examples (model validation)
  • Each driver's examples_test.py is a thin 30-line file that imports shared logic -- changing check rules requires editing one file, not 44
  • Every examples_test.py enforces:
    • test_example -- validates each example file against the appropriate Pydantic model
    • test_no_unused_examples -- fails if config*.yaml or usage*.py files are not referenced in README
    • test_no_inline_code_blocks -- fails if README has bare ```yaml or ```python fences instead of {literalinclude} or {code-block}
  • Driver creation template (create_driver.sh) updated to generate examples/config.yaml and examples_test.py for new drivers
  • Sphinx doctest removed -- replaced by standalone example files with per-driver pytest validation
  • make docs-test simplified to run pytest on example tests (no more Sphinx doctest runner)
  • CI documentation workflow updated with docs-test job

Test plan

  • make pkg-test-<driver> passes for representative drivers (e.g., yepkit, pyserial, mitmproxy)
  • make docs-test runs documentation example tests
  • make docs builds without literalinclude errors
  • Verify test_no_inline_code_blocks catches bare ```yaml/ ```python fences
  • Verify test_no_unused_examples catches unreferenced example files
  • Create a new driver with create_driver.sh and verify it includes examples/config.yaml and examples_test.py

🤖 Generated with Claude Code

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 26, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9dc06a21-c580-40ca-aa7c-edc12bba86f9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@raballew raballew force-pushed the 497-doc-snippet-testing-literalinclude branch from 7643f3d to 0ccaf49 Compare May 27, 2026 13:21
raballew and others added 28 commits May 27, 2026 21:20
Create the infrastructure for extracting inline code snippets from
documentation into standalone files with corresponding e2e tests.
Adds examples/tests/ with conftest fixtures and a docs-snippet-test
Makefile target.

Generated-By: Forge/20260526_125902_2796304_6b066394
Extract the inline driver implementation example from introduction/drivers.md
into a standalone Python file at examples/introduction/driver_example.py.
Replace the inline code block with a literalinclude directive. Add e2e test
that imports and runs the driver example using serve() for real validation.

Generated-By: Forge/20260526_125902_2796304_6b066394
Extract YAML exporter configuration snippets from introduction/exporters.md
and introduction/drivers.md into standalone files. Replace inline code blocks
with literalinclude directives. Add tests that validate each YAML file against
the ExporterConfigV1Alpha1 Pydantic model for real structural validation.

Generated-By: Forge/20260526_125902_2796304_6b066394
…ests

Extract all YAML hook configs and the Python hook example from
introduction/hooks.md into standalone files under examples/introduction/.
Replace inline code blocks with literalinclude directives. Add tests that
validate each YAML file against HookConfigV1Alpha1 and the full
ExporterConfigV1Alpha1 Pydantic models, and verify the Python example
compiles successfully.

Generated-By: Forge/20260526_125902_2796304_6b066394
Document the pattern for converting inline code blocks to literalinclude
directives with corresponding tests. Covers Python, YAML, and bash snippets,
test fixtures, and the docs-snippet-test Makefile target.

Generated-By: Forge/20260526_125902_2796304_6b066394
Ruff F401 (unused imports) would fail CI. The three test files imported
Path from pathlib but only used the examples_root fixture which already
returns a Path object.

Generated-By: Forge/20260526_125902_2796304_6b066394
The test only imports the module (does not execute the __main__ block),
so rename from test_driver_example_executes_successfully to
test_driver_example_imports_successfully.

Generated-By: Forge/20260526_125902_2796304_6b066394
…ample

Separate `import os` (stdlib) from `from jumpstarter.utils.env import env`
(third-party) with a blank line to satisfy ruff I001 import sorting rule.

Generated-By: Forge/20260526_125902_2796304_6b066394
Remove YAML comments from driver_exporter_config.yaml, remove redundant
file-existence tests superseded by validation tests, add trailing newline
to drivers.md, and fix grammar in contributor guidelines.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…r tests

Convert inline code blocks in 41 driver READMEs to literalinclude
directives pointing at standalone files under each driver's examples/
directory. Generate examples_test.py per driver that validates extracted
YAML configs (yaml.safe_load) and Python snippets (compile). Tests run
as part of existing make pkg-test-<driver> infrastructure.

104 example tests across 41 drivers. Doctest blocks and invalid YAML
are left inline. Install commands, CLI sessions, and console output
remain as inline code blocks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…t YAML syntax

Upgrade all 41 driver examples_test.py files to validate YAML configs
against ExporterConfigV1Alpha1DriverInstance.model_validate() (for export
sections) or ExporterConfigV1Alpha1.model_validate() (for full exporter
configs). Non-config YAML (scenarios, method definitions) falls back to
yaml.safe_load() validation. Remove one-shot conversion scripts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move jumpstarter.config.exporter imports from module level into test
functions using pytest.importorskip so tests skip gracefully when
jumpstarter is not installed rather than failing at collection time.
Remove incorrectly extracted doctest file from tftp driver and invalid
YAML config from ridesx driver.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ples

These files contained doctest >>> blocks that were incorrectly extracted
during conversion. They are not valid standalone Python and had no
corresponding test. The doctest blocks remain inline in the READMEs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move YAML config files from docs/source/reference/package-apis/drivers/
into each driver's examples/ directory. Update literalinclude paths in
READMEs. Remove hidden doctest blocks and replace with examples_test.py
that validates configs against ExporterConfigV1Alpha1DriverInstance.

44/45 drivers now have example tests. The remaining driver (uds) is a
base package with no config of its own.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ocs-test

Convert the last 2 doctest blocks (opendal, tftp) to standalone example
files with literalinclude. Move opendal.yaml from docs/ to driver
examples/. Simplify Makefile: docs-test now runs pytest on examples/
instead of Sphinx doctest (no doctest blocks remain). docs-snippet-test
kept as alias for backwards compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The documentation CI ran build, check-warnings, and linkcheck but never
ran the snippet validation tests. Add a docs-test job so documentation
examples are validated on every PR that touches python/docs/ or
python/packages/.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add jumpstarter.testing.examples module with validate_yaml_example and
validate_python_example functions. YAML validation dispatches by kind
(ExporterConfig, ClientConfig, UserConfig) or section (hooks, export)
to the appropriate Pydantic model. Python validation checks syntax and
imports.

Wrap all 67 config fragments in full Kubernetes-style YAML with
apiVersion, kind, and metadata so the kind discriminator drives model
selection automatically.

Replace 44 hand-written examples_test.py files and 3 central docs test
files with a single parametrized pattern that calls the shared module.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… tests

- README template now uses {literalinclude} for config instead of inline YAML
- Add examples/config.yaml.tmpl with full Kubernetes-style ExporterConfig
- Add examples_test.py.tmpl using shared jumpstarter.testing.examples
- Update creating-new-drivers rules with new directory structure and
  post-creation steps mentioning config.yaml and examples_test.py

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rename 49 example files from auto-generated heading slugs to concise,
descriptive names. Update literalinclude paths in 17 READMEs.

Examples: config_exporterconfig_example_1.yaml -> config_avh.yaml,
config_example_configuration_for_shelly_smart_p.yaml -> config_shelly.yaml,
usage_flash_with_compressed_images.py -> usage_compressed_flash.py

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each driver's examples_test.py now checks:
- test_no_unused_examples: fails if config*.yaml or usage*.py files in
  examples/ are not referenced in README.md
- test_no_inline_code_blocks: fails if README.md contains bare
  ```yaml or ```python fences that should use {literalinclude}

Convert 9 remaining inline fragments to {code-block} directives. Fix
stale literalinclude paths in someip and ssh-mitm READMEs. Update driver
creation template with the same checks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move discover_example_files, find_unused_examples, and
find_inline_code_blocks into jumpstarter.testing.checks (no model
dependencies). Remove duplicate implementations from 44 examples_test.py
files. Each test file is now 30 lines importing from the shared modules.

Changing the skip directive list, naming convention, or check logic now
requires editing one file instead of 44.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Follow pytest's recommended test_*.py naming convention. Update driver
creation template and rules documentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The docs build generates CRD reference pages from
controller/deploy/operator/config/crd/bases/. Without this path in the
trigger, CRD changes would not rebuild the documentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix test_exporter_configs.py to use discover_example_files from
  checks module instead of nonexistent make_example_test_params
- Update guidelines.md: fix stale docs-snippet-test reference to
  docs-test, clarify that Python usage files are syntax-checked
  fragments not standalone scripts
- Add make sync before make docs-test in CI for consistency with
  other documentation workflow jobs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
raballew and others added 30 commits May 28, 2026 10:31
When a YAML file has a kind field not found in KIND_TO_MODEL, raise a
ValueError instead of silently falling through to the warning path.
This makes the validation contract explicit: if kind is specified, it
must be recognized.

Generated-By: Forge/20260528_101328_15756_6af5bbd0
…nfig

Verify that validate_yaml_example raises a ValidationError when given a
YAML file with kind: ExporterConfig but missing required fields like
endpoint and metadata. This covers the Pydantic rejection path that was
previously untested.

Generated-By: Forge/20260528_101328_15756_6af5bbd0
The jumpstarter.testing.examples module is part of the core jumpstarter
package and should always be importable in the test environment. Using
importorskip can silently skip all parametrized validation tests if the
module fails to import for any reason, masking broken test infrastructure.

Generated-By: Forge/20260528_101328_15756_6af5bbd0
Consistent with the fix already applied to test_example_files.py,
replace pytest.importorskip with a direct module-level import of
validate_example from jumpstarter.testing.examples. This prevents
tests from being silently skipped if the import fails.

Generated-By: Forge/20260528_101328_15756_6af5bbd0
Adds test_validate_yaml_example_rejects_invalid_export_entry which
verifies that validate_yaml_example raises ValidationError when an
export section entry is not a valid dict for model validation.

Generated-By: Forge/20260528_101328_15756_6af5bbd0
Extract the inline code-block YAML configuration from the ridesx
README into examples/config.yaml as a full ExporterConfig, and
replace it with a literalinclude directive for testability.

Generated-By: Forge/20260528_101328_15756_6af5bbd0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move proc.stderr.close() immediately after reading stderr content,
before any branching to retry or exception raising. This prevents
file descriptor leaks on non-CPython runtimes where the garbage
collector may not promptly finalize the pipe object.

Generated-By: Forge/20260528_132121_291040_eab068b7
Add norecursedirs = ["examples"] to the ssh-mount package pytest
configuration to match the convention used by all other driver
packages that have an examples directory.

Generated-By: Forge/20260528_132121_291040_eab068b7
Add direct unit tests for path_with_query covering both branches:
URLs with query parameters and URLs without query parameters.
Previously this function only had indirect coverage through
integration tests.

Generated-By: Forge/20260528_132121_291040_eab068b7
…tion files

Add test_docs_no_inline_code_blocks to scan all non-JEP documentation markdown
files for inline YAML/Python code blocks that should use literalinclude. Known
violations are marked xfail(strict=True) so they track migration progress and
new violations cause immediate failures.

Add test_docs_no_unused_examples with find_unused_examples_in_docs to verify
example files under docs/source/examples/ are referenced from documentation.

Generated-By: Forge/20260528_145540_381322_936cce04
Add docs/source/examples/ subdirectories to _example_file_params so that
documentation example files (YAML configs and Python scripts) are validated
alongside driver package examples in test_example_validates and
test_example_instantiates.

Generated-By: Forge/20260528_145540_381322_936cce04
Generated-By: Forge/20260528_145540_381322_936cce04
…k detection

Inline bash/shell code blocks in README files were not being detected
because EXTRACTABLE_LANGUAGES only contained yaml, python, and py.
Add bash and shell to fulfill the spec requirement. Existing inline
bash/shell blocks across 24 driver packages are added to xfail sets
to be converted in follow-up work.

Generated-By: Forge/20260528_145540_381322_936cce04
…te_path

Bash evaluates command substitutions ($(...)) and backtick expressions
in PS1 on every prompt display. Zsh additionally interprets percent
sequences. Add _escape_for_bash_ps1 and _escape_for_zsh_ps1 methods
that neutralize $, backticks, backslashes, and % before embedding
remote_path into the prompt string.

Generated-By: Forge/20260529_102529_1255558_c89788a1
Replace unconditional StrictHostKeyChecking=no and
UserKnownHostsFile=/dev/null with StrictHostKeyChecking=accept-new
and a real known_hosts file (~/.ssh/known_hosts). Add --insecure
CLI flag for users who explicitly want to disable host key
verification.

Generated-By: Forge/20260529_102529_1255558_c89788a1
Generated-By: Forge/20260529_102529_1255558_c89788a1
…ecure flag

The README documented StrictHostKeyChecking=no and UserKnownHostsFile=/dev/null
as the defaults, but the code now defaults to StrictHostKeyChecking=accept-new
with UserKnownHostsFile=~/.ssh/known_hosts. Updated the documentation to match
the actual behavior and document the --insecure flag.

Generated-By: Forge/20260529_102529_1255558_c89788a1
The docs-test Makefile target no longer runs Sphinx doctests, leaving
this testcode block unexecuted. Since this is illustrative pseudocode
that cannot run standalone, convert it to a code-block directive.

Generated-By: Forge/20260529_102529_1255558_c89788a1
The loop over SECTION_TO_MODEL returned after validating the first
matching section. A YAML file containing both hooks and export would
only have the first section validated while the second was silently
skipped. Remove the early return so all matching sections are checked.

Generated-By: Forge/20260529_102529_1255558_c89788a1
Add tests covering empty markdown file lists, references found across
multiple markdown files, unreferenced example files, and missing
examples directory.

Generated-By: Forge/20260529_102529_1255558_c89788a1
validate_example silently did nothing for kinds other than yaml or
python (e.g. bash, shell). Add an else branch that raises ValueError
to surface unsupported kinds explicitly rather than silently skipping
validation.

Generated-By: Forge/20260529_102529_1255558_c89788a1
A non-dict hooks value (e.g. a list) was silently skipped without
validation. Now raises TypeError with a descriptive message. Extracted
section validation into _validate_section helper to stay under C901
complexity limit.

Generated-By: Forge/20260529_160937_1659706_722b0319
Convert inline bash/shell code blocks in driver READMEs to either
literalinclude directives (for standalone examples) or code-block
directives (for output/placeholder/interactive snippets).

Drivers converted: adb, androidemulator, composite, doip, dut-network,
energenie, esp32, flashers.

Generated-By: Forge/20260529_164302_1718250_e7b18c65
Convert inline bash/shell code blocks in driver READMEs for http-power,
mitmproxy, noyito-relay, pyserial, and snmp.

Generated-By: Forge/20260529_164302_1718250_e7b18c65
Convert inline bash/shell code blocks in driver READMEs for someip,
ssh, ssh-mitm, ssh-mount, and stlink-msd.

Generated-By: Forge/20260529_164302_1718250_e7b18c65
Convert inline bash/shell code blocks in driver READMEs for tmt,
uds-can, uds-doip, vnc, xcp, and yepkit.

Generated-By: Forge/20260529_164302_1718250_e7b18c65
…-block

Convert inline python/yaml/bash/shell blocks in getting-started docs
and CI/CD guides. Create external example files for standalone Python
test examples; use code-block directives for yaml config fragments,
CI workflow snippets, and contextual code blocks.

Generated-By: Forge/20260529_164302_1718250_e7b18c65
…sions

All 23 driver READMEs and 31 docs pages have been converted from
inline code blocks to either literalinclude directives (for standalone
examples) or code-block directives (for output/placeholder/interactive
snippets). Both xfail sets are now empty.

Generated-By: Forge/20260529_164302_1718250_e7b18c65
Align 13 test files with the project convention of *_test.py suffix
naming. test_utils.py is kept as-is since it contains test utilities,
not tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align with project convention of underscores in Python filenames.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant