feat: documentation snippet testing with literalinclude and Pydantic validation#704
Draft
raballew wants to merge 74 commits into
Draft
feat: documentation snippet testing with literalinclude and Pydantic validation#704raballew wants to merge 74 commits into
raballew wants to merge 74 commits into
Conversation
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
7643f3d to
0ccaf49
Compare
Create the infrastructure for extracting inline code snippets from documentation into standalone files with corresponding e2e tests. Adds examples/tests/ with conftest fixtures and a docs-snippet-test Makefile target. Generated-By: Forge/20260526_125902_2796304_6b066394
Extract the inline driver implementation example from introduction/drivers.md into a standalone Python file at examples/introduction/driver_example.py. Replace the inline code block with a literalinclude directive. Add e2e test that imports and runs the driver example using serve() for real validation. Generated-By: Forge/20260526_125902_2796304_6b066394
Extract YAML exporter configuration snippets from introduction/exporters.md and introduction/drivers.md into standalone files. Replace inline code blocks with literalinclude directives. Add tests that validate each YAML file against the ExporterConfigV1Alpha1 Pydantic model for real structural validation. Generated-By: Forge/20260526_125902_2796304_6b066394
…ests Extract all YAML hook configs and the Python hook example from introduction/hooks.md into standalone files under examples/introduction/. Replace inline code blocks with literalinclude directives. Add tests that validate each YAML file against HookConfigV1Alpha1 and the full ExporterConfigV1Alpha1 Pydantic models, and verify the Python example compiles successfully. Generated-By: Forge/20260526_125902_2796304_6b066394
Document the pattern for converting inline code blocks to literalinclude directives with corresponding tests. Covers Python, YAML, and bash snippets, test fixtures, and the docs-snippet-test Makefile target. Generated-By: Forge/20260526_125902_2796304_6b066394
Ruff F401 (unused imports) would fail CI. The three test files imported Path from pathlib but only used the examples_root fixture which already returns a Path object. Generated-By: Forge/20260526_125902_2796304_6b066394
The test only imports the module (does not execute the __main__ block), so rename from test_driver_example_executes_successfully to test_driver_example_imports_successfully. Generated-By: Forge/20260526_125902_2796304_6b066394
…ample Separate `import os` (stdlib) from `from jumpstarter.utils.env import env` (third-party) with a blank line to satisfy ruff I001 import sorting rule. Generated-By: Forge/20260526_125902_2796304_6b066394
Remove YAML comments from driver_exporter_config.yaml, remove redundant file-existence tests superseded by validation tests, add trailing newline to drivers.md, and fix grammar in contributor guidelines. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…r tests Convert inline code blocks in 41 driver READMEs to literalinclude directives pointing at standalone files under each driver's examples/ directory. Generate examples_test.py per driver that validates extracted YAML configs (yaml.safe_load) and Python snippets (compile). Tests run as part of existing make pkg-test-<driver> infrastructure. 104 example tests across 41 drivers. Doctest blocks and invalid YAML are left inline. Install commands, CLI sessions, and console output remain as inline code blocks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…t YAML syntax Upgrade all 41 driver examples_test.py files to validate YAML configs against ExporterConfigV1Alpha1DriverInstance.model_validate() (for export sections) or ExporterConfigV1Alpha1.model_validate() (for full exporter configs). Non-config YAML (scenarios, method definitions) falls back to yaml.safe_load() validation. Remove one-shot conversion scripts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move jumpstarter.config.exporter imports from module level into test functions using pytest.importorskip so tests skip gracefully when jumpstarter is not installed rather than failing at collection time. Remove incorrectly extracted doctest file from tftp driver and invalid YAML config from ridesx driver. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ples These files contained doctest >>> blocks that were incorrectly extracted during conversion. They are not valid standalone Python and had no corresponding test. The doctest blocks remain inline in the READMEs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move YAML config files from docs/source/reference/package-apis/drivers/ into each driver's examples/ directory. Update literalinclude paths in READMEs. Remove hidden doctest blocks and replace with examples_test.py that validates configs against ExporterConfigV1Alpha1DriverInstance. 44/45 drivers now have example tests. The remaining driver (uds) is a base package with no config of its own. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ocs-test Convert the last 2 doctest blocks (opendal, tftp) to standalone example files with literalinclude. Move opendal.yaml from docs/ to driver examples/. Simplify Makefile: docs-test now runs pytest on examples/ instead of Sphinx doctest (no doctest blocks remain). docs-snippet-test kept as alias for backwards compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The documentation CI ran build, check-warnings, and linkcheck but never ran the snippet validation tests. Add a docs-test job so documentation examples are validated on every PR that touches python/docs/ or python/packages/. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add jumpstarter.testing.examples module with validate_yaml_example and validate_python_example functions. YAML validation dispatches by kind (ExporterConfig, ClientConfig, UserConfig) or section (hooks, export) to the appropriate Pydantic model. Python validation checks syntax and imports. Wrap all 67 config fragments in full Kubernetes-style YAML with apiVersion, kind, and metadata so the kind discriminator drives model selection automatically. Replace 44 hand-written examples_test.py files and 3 central docs test files with a single parametrized pattern that calls the shared module. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… tests
- README template now uses {literalinclude} for config instead of inline YAML
- Add examples/config.yaml.tmpl with full Kubernetes-style ExporterConfig
- Add examples_test.py.tmpl using shared jumpstarter.testing.examples
- Update creating-new-drivers rules with new directory structure and
post-creation steps mentioning config.yaml and examples_test.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rename 49 example files from auto-generated heading slugs to concise, descriptive names. Update literalinclude paths in 17 READMEs. Examples: config_exporterconfig_example_1.yaml -> config_avh.yaml, config_example_configuration_for_shelly_smart_p.yaml -> config_shelly.yaml, usage_flash_with_compressed_images.py -> usage_compressed_flash.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each driver's examples_test.py now checks:
- test_no_unused_examples: fails if config*.yaml or usage*.py files in
examples/ are not referenced in README.md
- test_no_inline_code_blocks: fails if README.md contains bare
```yaml or ```python fences that should use {literalinclude}
Convert 9 remaining inline fragments to {code-block} directives. Fix
stale literalinclude paths in someip and ssh-mitm READMEs. Update driver
creation template with the same checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move discover_example_files, find_unused_examples, and find_inline_code_blocks into jumpstarter.testing.checks (no model dependencies). Remove duplicate implementations from 44 examples_test.py files. Each test file is now 30 lines importing from the shared modules. Changing the skip directive list, naming convention, or check logic now requires editing one file instead of 44. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Follow pytest's recommended test_*.py naming convention. Update driver creation template and rules documentation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The docs build generates CRD reference pages from controller/deploy/operator/config/crd/bases/. Without this path in the trigger, CRD changes would not rebuild the documentation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix test_exporter_configs.py to use discover_example_files from checks module instead of nonexistent make_example_test_params - Update guidelines.md: fix stale docs-snippet-test reference to docs-test, clarify that Python usage files are syntax-checked fragments not standalone scripts - Add make sync before make docs-test in CI for consistency with other documentation workflow jobs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a YAML file has a kind field not found in KIND_TO_MODEL, raise a ValueError instead of silently falling through to the warning path. This makes the validation contract explicit: if kind is specified, it must be recognized. Generated-By: Forge/20260528_101328_15756_6af5bbd0
…nfig Verify that validate_yaml_example raises a ValidationError when given a YAML file with kind: ExporterConfig but missing required fields like endpoint and metadata. This covers the Pydantic rejection path that was previously untested. Generated-By: Forge/20260528_101328_15756_6af5bbd0
The jumpstarter.testing.examples module is part of the core jumpstarter package and should always be importable in the test environment. Using importorskip can silently skip all parametrized validation tests if the module fails to import for any reason, masking broken test infrastructure. Generated-By: Forge/20260528_101328_15756_6af5bbd0
Consistent with the fix already applied to test_example_files.py, replace pytest.importorskip with a direct module-level import of validate_example from jumpstarter.testing.examples. This prevents tests from being silently skipped if the import fails. Generated-By: Forge/20260528_101328_15756_6af5bbd0
Adds test_validate_yaml_example_rejects_invalid_export_entry which verifies that validate_yaml_example raises ValidationError when an export section entry is not a valid dict for model validation. Generated-By: Forge/20260528_101328_15756_6af5bbd0
Extract the inline code-block YAML configuration from the ridesx README into examples/config.yaml as a full ExporterConfig, and replace it with a literalinclude directive for testability. Generated-By: Forge/20260528_101328_15756_6af5bbd0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move proc.stderr.close() immediately after reading stderr content, before any branching to retry or exception raising. This prevents file descriptor leaks on non-CPython runtimes where the garbage collector may not promptly finalize the pipe object. Generated-By: Forge/20260528_132121_291040_eab068b7
Add norecursedirs = ["examples"] to the ssh-mount package pytest configuration to match the convention used by all other driver packages that have an examples directory. Generated-By: Forge/20260528_132121_291040_eab068b7
Add direct unit tests for path_with_query covering both branches: URLs with query parameters and URLs without query parameters. Previously this function only had indirect coverage through integration tests. Generated-By: Forge/20260528_132121_291040_eab068b7
…tion files Add test_docs_no_inline_code_blocks to scan all non-JEP documentation markdown files for inline YAML/Python code blocks that should use literalinclude. Known violations are marked xfail(strict=True) so they track migration progress and new violations cause immediate failures. Add test_docs_no_unused_examples with find_unused_examples_in_docs to verify example files under docs/source/examples/ are referenced from documentation. Generated-By: Forge/20260528_145540_381322_936cce04
Add docs/source/examples/ subdirectories to _example_file_params so that documentation example files (YAML configs and Python scripts) are validated alongside driver package examples in test_example_validates and test_example_instantiates. Generated-By: Forge/20260528_145540_381322_936cce04
Generated-By: Forge/20260528_145540_381322_936cce04
…k detection Inline bash/shell code blocks in README files were not being detected because EXTRACTABLE_LANGUAGES only contained yaml, python, and py. Add bash and shell to fulfill the spec requirement. Existing inline bash/shell blocks across 24 driver packages are added to xfail sets to be converted in follow-up work. Generated-By: Forge/20260528_145540_381322_936cce04
…te_path Bash evaluates command substitutions ($(...)) and backtick expressions in PS1 on every prompt display. Zsh additionally interprets percent sequences. Add _escape_for_bash_ps1 and _escape_for_zsh_ps1 methods that neutralize $, backticks, backslashes, and % before embedding remote_path into the prompt string. Generated-By: Forge/20260529_102529_1255558_c89788a1
Replace unconditional StrictHostKeyChecking=no and UserKnownHostsFile=/dev/null with StrictHostKeyChecking=accept-new and a real known_hosts file (~/.ssh/known_hosts). Add --insecure CLI flag for users who explicitly want to disable host key verification. Generated-By: Forge/20260529_102529_1255558_c89788a1
Generated-By: Forge/20260529_102529_1255558_c89788a1
…ecure flag The README documented StrictHostKeyChecking=no and UserKnownHostsFile=/dev/null as the defaults, but the code now defaults to StrictHostKeyChecking=accept-new with UserKnownHostsFile=~/.ssh/known_hosts. Updated the documentation to match the actual behavior and document the --insecure flag. Generated-By: Forge/20260529_102529_1255558_c89788a1
The docs-test Makefile target no longer runs Sphinx doctests, leaving this testcode block unexecuted. Since this is illustrative pseudocode that cannot run standalone, convert it to a code-block directive. Generated-By: Forge/20260529_102529_1255558_c89788a1
The loop over SECTION_TO_MODEL returned after validating the first matching section. A YAML file containing both hooks and export would only have the first section validated while the second was silently skipped. Remove the early return so all matching sections are checked. Generated-By: Forge/20260529_102529_1255558_c89788a1
Add tests covering empty markdown file lists, references found across multiple markdown files, unreferenced example files, and missing examples directory. Generated-By: Forge/20260529_102529_1255558_c89788a1
validate_example silently did nothing for kinds other than yaml or python (e.g. bash, shell). Add an else branch that raises ValueError to surface unsupported kinds explicitly rather than silently skipping validation. Generated-By: Forge/20260529_102529_1255558_c89788a1
A non-dict hooks value (e.g. a list) was silently skipped without validation. Now raises TypeError with a descriptive message. Extracted section validation into _validate_section helper to stay under C901 complexity limit. Generated-By: Forge/20260529_160937_1659706_722b0319
Convert inline bash/shell code blocks in driver READMEs to either literalinclude directives (for standalone examples) or code-block directives (for output/placeholder/interactive snippets). Drivers converted: adb, androidemulator, composite, doip, dut-network, energenie, esp32, flashers. Generated-By: Forge/20260529_164302_1718250_e7b18c65
Convert inline bash/shell code blocks in driver READMEs for http-power, mitmproxy, noyito-relay, pyserial, and snmp. Generated-By: Forge/20260529_164302_1718250_e7b18c65
Convert inline bash/shell code blocks in driver READMEs for someip, ssh, ssh-mitm, ssh-mount, and stlink-msd. Generated-By: Forge/20260529_164302_1718250_e7b18c65
Convert inline bash/shell code blocks in driver READMEs for tmt, uds-can, uds-doip, vnc, xcp, and yepkit. Generated-By: Forge/20260529_164302_1718250_e7b18c65
…-block Convert inline python/yaml/bash/shell blocks in getting-started docs and CI/CD guides. Create external example files for standalone Python test examples; use code-block directives for yaml config fragments, CI workflow snippets, and contextual code blocks. Generated-By: Forge/20260529_164302_1718250_e7b18c65
…sions All 23 driver READMEs and 31 docs pages have been converted from inline code blocks to either literalinclude directives (for standalone examples) or code-block directives (for output/placeholder/interactive snippets). Both xfail sets are now empty. Generated-By: Forge/20260529_164302_1718250_e7b18c65
Align 13 test files with the project convention of *_test.py suffix naming. test_utils.py is kept as-is since it contains test utilities, not tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align with project convention of underscores in Python filenames. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
examples/directory, referenced via{literalinclude}directivesapiVersion,kind,metadata) and are validated against Pydantic models (ExporterConfigV1Alpha1,HookConfigV1Alpha1, etc.)compile()and import-checked viaimportlibjumpstarter.testing.checks(file discovery, unused/inline detection) andjumpstarter.testing.examples(model validation)examples_test.pyis a thin 30-line file that imports shared logic -- changing check rules requires editing one file, not 44examples_test.pyenforces:test_example-- validates each example file against the appropriate Pydantic modeltest_no_unused_examples-- fails ifconfig*.yamlorusage*.pyfiles are not referenced in READMEtest_no_inline_code_blocks-- fails if README has bare```yamlor```pythonfences instead of{literalinclude}or{code-block}create_driver.sh) updated to generateexamples/config.yamlandexamples_test.pyfor new driversmake docs-testsimplified to run pytest on example tests (no more Sphinx doctest runner)docs-testjobTest plan
make pkg-test-<driver>passes for representative drivers (e.g., yepkit, pyserial, mitmproxy)make docs-testruns documentation example testsmake docsbuilds without literalinclude errorstest_no_inline_code_blockscatches bare```yaml/```pythonfencestest_no_unused_examplescatches unreferenced example filescreate_driver.shand verify it includesexamples/config.yamlandexamples_test.py🤖 Generated with Claude Code