Open
Conversation
Each assertion type (Resolves, DoesNotResolve, ResolvesWith, ResolvesWithType, SearchByName, Needed) is now a self-contained class in src/babel_validation/assertions/, grouped by which service it targets (nodenorm.py, nameres.py, common.py). A central ASSERTION_HANDLERS registry in __init__.py maps lowercase assertion names to handler instances, and NodeNormAssertion / NameResAssertion marker base classes allow isinstance() checks for applicability. GitHubIssueTest.test_with_nodenorm() and test_with_nameres() are now 3-line dispatchers. A README.md documents all supported assertion types with examples in both wiki and YAML syntax. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pendently. Bumps pytest to >=9.0 (which includes built-in subtests support). Each TestResult is now evaluated in its own subtest block, so a failure no longer short-circuits the rest. Adds post-loop state-consistency subtests: a closed issue with failing tests fails with a "consider reopening" message, and an open issue where all tests pass emits an xfail "consider closing" hint. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously, get_github_issue_id() and GitHubIssueTest.__str__() both resolved the org/repo name via github_issue.repository.organization.name, which triggers lazy PyGitHub API calls. Parse org/repo from html_url instead (always present in the issue JSON, no extra round-trip needed). Also resolves the TODO comment in get_test_issues_from_issue(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Add `curie_params()` to `NodeNormTest` so handlers can declare which params are CURIEs (default: all params) - Prewarm NodeNorm cache only for CURIE params instead of all params - Validate CURIE format (PREFIX:LOCAL_ID) before calling test_param_set, yielding a clear failure for malformed CURIEs - Override `curie_params()` in HasLabelHandler (params[:1]) and ResolvesWithTypeHandler (params[1:]) to exclude non-CURIE params Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
tests/pytest.ini takes precedence over pyproject.toml when running tests under tests/, so the [tool.pytest.ini_options] block was never read. Register the `unit` mark in tests/pytest.ini and remove the now-redundant section from pyproject.toml. Also fixes python-dotenv dependency name (was incorrectly listed as `dotenv`, a thin wrapper package). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove unreachable duplicate block in ResolvesWithTypeHandler.test_param_set - Use github_issue_id fixture param directly instead of re-parsing the URL - Replace unused *args with *_ in SearchByNameHandler Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If the cache file exists but contains invalid JSON (e.g. from a killed process mid-write), delete it and re-fetch rather than crashing with an unhelpful JSONDecodeError. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The full NameRes results dump belongs at DEBUG level, not on stdout, where it would bypass pytest's output capture and could produce large noise in test runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a GitHub-issue-driven testing framework to babel-validation by introducing a reusable src/babel_validation/ library (assertion handlers, service wrappers, and test-case sources) and wiring in unit-test CI.
Changes:
- Added an assertions framework (
src/babel_validation/assertions/) with auto-generated documentation and a unit test to keep it in sync. - Added GitHub Issues integration to discover/parse
{{BabelTest|...}}and YAMLbabel_tests:blocks and execute them against NodeNorm/NameRes. - Added caching/parallelism support (FileLock + xdist) and a GitHub Actions workflow to run unit tests on PRs.
Reviewed changes
Copilot reviewed 26 out of 36 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
pyproject.toml |
Adds new runtime/test dependencies for GitHub + caching + parallelism. |
uv.lock |
Locks new dependencies required by GitHub issue framework and xdist. |
src/babel_validation/core/testrow.py |
Introduces shared TestRow / TestResult types used across sources/assertions. |
src/babel_validation/services/nodenorm.py |
Adds cached wrapper for NodeNorm normalization calls. |
src/babel_validation/services/nameres.py |
Adds cached wrapper for NameRes lookup calls. |
src/babel_validation/sources/google_sheets/google_sheet_test_cases.py |
Moves Google Sheet loader into library + adds local CSV caching. |
src/babel_validation/sources/google_sheets/blocklist.py |
Extracts blocklist sheet loader into library module. |
src/babel_validation/sources/github/github_issues_test_cases.py |
Implements GitHub issue search + parsing of embedded BabelTest definitions. |
src/babel_validation/assertions/* |
Adds assertion handler base classes, NodeNorm/NameRes assertion implementations, and README generator. |
tests/github_issues/* |
Adds GitHub-issue-driven tests and fixtures, including caching of discovered issue IDs. |
tests/test_environment/test_assertions_docs.py |
Unit test to enforce auto-generated assertions README is current. |
tests/* (various) |
Updates imports to use new library modules; minor test data structure cleanup. |
tests/pytest.ini |
Adds unit marker + timeout configuration (but see comment about config discovery). |
.github/workflows/tests.yaml |
Adds PR CI job running unit tests via uv run pytest -m unit. |
.gitignore |
Ignores root .env. |
CLAUDE.md |
Updates architecture docs to reflect new src/babel_validation/ library and GitHub issues tests. |
Comments suppressed due to low confidence (1)
src/babel_validation/sources/google_sheets/google_sheet_test_cases.py:47
- The Google Sheet CSV download is cached to a temp file, but the HTTP response is never validated. If Google returns a non-200 (or an HTML error page), that content will be cached and can make subsequent runs fail in confusing ways. Call response.raise_for_status() (and ideally set a reasonable timeout) before persisting response.text to the cache file.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
tests/pytest.ini was never picked up when running pytest from the repo root (pytest searches upward, not into subdirectories). The timeout and unit marker registration were silently ignored. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The list of repositories to scan for BabelTest assertions is now configuration rather than code. It lives in [DEFAULT] so it applies globally and can be overridden per-section if ever needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
5c97356 to
552d53a
Compare
Add module docstring to github_issues_test_cases.py defining assertion, param_set, and param_sets; document GitHubIssueTest class and __init__ params; add inline comments showing how Wiki/YAML syntax maps to param_sets. Add param_sets/:param params: notes to base class methods in __init__.py. Add inline comments to HasLabelHandler, ResolvesWithTypeHandler, and SearchByNameHandler clarifying special element positions. Add "Param Sets" section to gen_docs.py INTRO and regenerate README.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both handlers previously silently discarded extra parameters. They now yield a failed TestResult when more than 2 params are supplied, making malformed assertions visible rather than silently tolerated. Added unit tests to cover the new behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers the unhandled path in get_test_issues_from_issue() where a YAML block matches the detection regex but contains invalid YAML. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…/null babel test content Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds TestIssueHasTests (5 tests) and TestGetIssuesWithTests (4 tests) covering the regex gate, deduplication logic, and search false-positive filtering in GitHubIssuesTestCases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces a full GitHub-issue-driven test framework for babel-validation. Key additions: