Skip to content

Add GitHub issue tests#67

Open
gaurav wants to merge 81 commits intomainfrom
add-github-issue-tests
Open

Add GitHub issue tests#67
gaurav wants to merge 81 commits intomainfrom
add-github-issue-tests

Conversation

@gaurav
Copy link
Collaborator

@gaurav gaurav commented Jan 8, 2026

This PR introduces a full GitHub-issue-driven test framework for babel-validation. Key additions:

  • Assertion framework (src/babel_validation/assertions/) with 8 handler types
  • GitHub integration via PyGitHub — parses {{BabelTest|...}} wiki syntax and YAML blocks embedded in issue bodies
  • Restructure of shared code from tests/common/ into src/babel_validation/ (library pattern)
  • CI workflow running unit tests on PRs
  • pytest-xdist parallelism with FileLock-based cache coordination

gaurav and others added 10 commits February 15, 2026 02:39
Each assertion type (Resolves, DoesNotResolve, ResolvesWith, ResolvesWithType,
SearchByName, Needed) is now a self-contained class in
src/babel_validation/assertions/, grouped by which service it targets
(nodenorm.py, nameres.py, common.py). A central ASSERTION_HANDLERS registry
in __init__.py maps lowercase assertion names to handler instances, and
NodeNormAssertion / NameResAssertion marker base classes allow isinstance()
checks for applicability. GitHubIssueTest.test_with_nodenorm() and
test_with_nameres() are now 3-line dispatchers. A README.md documents all
supported assertion types with examples in both wiki and YAML syntax.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gaurav and others added 2 commits February 19, 2026 19:03
…pendently.

Bumps pytest to >=9.0 (which includes built-in subtests support). Each
TestResult is now evaluated in its own subtest block, so a failure no longer
short-circuits the rest. Adds post-loop state-consistency subtests: a closed
issue with failing tests fails with a "consider reopening" message, and an open
issue where all tests pass emits an xfail "consider closing" hint.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously, get_github_issue_id() and GitHubIssueTest.__str__() both resolved
the org/repo name via github_issue.repository.organization.name, which triggers
lazy PyGitHub API calls. Parse org/repo from html_url instead (always present
in the issue JSON, no extra round-trip needed). Also resolves the TODO comment
in get_test_issues_from_issue().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gaurav and others added 14 commits March 15, 2026 02:56
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Add `curie_params()` to `NodeNormTest` so handlers can declare which
  params are CURIEs (default: all params)
- Prewarm NodeNorm cache only for CURIE params instead of all params
- Validate CURIE format (PREFIX:LOCAL_ID) before calling test_param_set,
  yielding a clear failure for malformed CURIEs
- Override `curie_params()` in HasLabelHandler (params[:1]) and
  ResolvesWithTypeHandler (params[1:]) to exclude non-CURIE params

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
tests/pytest.ini takes precedence over pyproject.toml when running
tests under tests/, so the [tool.pytest.ini_options] block was never
read. Register the `unit` mark in tests/pytest.ini and remove the
now-redundant section from pyproject.toml.

Also fixes python-dotenv dependency name (was incorrectly listed as
`dotenv`, a thin wrapper package).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove unreachable duplicate block in ResolvesWithTypeHandler.test_param_set
- Use github_issue_id fixture param directly instead of re-parsing the URL
- Replace unused *args with *_ in SearchByNameHandler

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If the cache file exists but contains invalid JSON (e.g. from a killed
process mid-write), delete it and re-fetch rather than crashing with an
unhelpful JSONDecodeError.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The full NameRes results dump belongs at DEBUG level, not on stdout,
where it would bypass pytest's output capture and could produce large
noise in test runs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a GitHub-issue-driven testing framework to babel-validation by introducing a reusable src/babel_validation/ library (assertion handlers, service wrappers, and test-case sources) and wiring in unit-test CI.

Changes:

  • Added an assertions framework (src/babel_validation/assertions/) with auto-generated documentation and a unit test to keep it in sync.
  • Added GitHub Issues integration to discover/parse {{BabelTest|...}} and YAML babel_tests: blocks and execute them against NodeNorm/NameRes.
  • Added caching/parallelism support (FileLock + xdist) and a GitHub Actions workflow to run unit tests on PRs.

Reviewed changes

Copilot reviewed 26 out of 36 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
pyproject.toml Adds new runtime/test dependencies for GitHub + caching + parallelism.
uv.lock Locks new dependencies required by GitHub issue framework and xdist.
src/babel_validation/core/testrow.py Introduces shared TestRow / TestResult types used across sources/assertions.
src/babel_validation/services/nodenorm.py Adds cached wrapper for NodeNorm normalization calls.
src/babel_validation/services/nameres.py Adds cached wrapper for NameRes lookup calls.
src/babel_validation/sources/google_sheets/google_sheet_test_cases.py Moves Google Sheet loader into library + adds local CSV caching.
src/babel_validation/sources/google_sheets/blocklist.py Extracts blocklist sheet loader into library module.
src/babel_validation/sources/github/github_issues_test_cases.py Implements GitHub issue search + parsing of embedded BabelTest definitions.
src/babel_validation/assertions/* Adds assertion handler base classes, NodeNorm/NameRes assertion implementations, and README generator.
tests/github_issues/* Adds GitHub-issue-driven tests and fixtures, including caching of discovered issue IDs.
tests/test_environment/test_assertions_docs.py Unit test to enforce auto-generated assertions README is current.
tests/* (various) Updates imports to use new library modules; minor test data structure cleanup.
tests/pytest.ini Adds unit marker + timeout configuration (but see comment about config discovery).
.github/workflows/tests.yaml Adds PR CI job running unit tests via uv run pytest -m unit.
.gitignore Ignores root .env.
CLAUDE.md Updates architecture docs to reflect new src/babel_validation/ library and GitHub issues tests.
Comments suppressed due to low confidence (1)

src/babel_validation/sources/google_sheets/google_sheet_test_cases.py:47

  • The Google Sheet CSV download is cached to a temp file, but the HTTP response is never validated. If Google returns a non-200 (or an HTML error page), that content will be cached and can make subsequent runs fail in confusing ways. Call response.raise_for_status() (and ideally set a reasonable timeout) before persisting response.text to the cache file.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

gaurav and others added 4 commits March 15, 2026 22:49
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
tests/pytest.ini was never picked up when running pytest from the repo
root (pytest searches upward, not into subdirectories). The timeout
and unit marker registration were silently ignored.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The list of repositories to scan for BabelTest assertions is now
configuration rather than code. It lives in [DEFAULT] so it applies
globally and can be overridden per-section if ever needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gaurav gaurav force-pushed the add-github-issue-tests branch from 5c97356 to 552d53a Compare March 16, 2026 04:54
gaurav and others added 6 commits March 16, 2026 00:54
Add module docstring to github_issues_test_cases.py defining assertion,
param_set, and param_sets; document GitHubIssueTest class and __init__
params; add inline comments showing how Wiki/YAML syntax maps to
param_sets. Add param_sets/:param params: notes to base class methods in
__init__.py. Add inline comments to HasLabelHandler,
ResolvesWithTypeHandler, and SearchByNameHandler clarifying special
element positions. Add "Param Sets" section to gen_docs.py INTRO and
regenerate README.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both handlers previously silently discarded extra parameters. They now
yield a failed TestResult when more than 2 params are supplied, making
malformed assertions visible rather than silently tolerated. Added unit
tests to cover the new behaviour.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers the unhandled path in get_test_issues_from_issue() where a YAML
block matches the detection regex but contains invalid YAML.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…/null babel test content

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds TestIssueHasTests (5 tests) and TestGetIssuesWithTests (4 tests)
covering the regex gate, deduplication logic, and search false-positive
filtering in GitHubIssuesTestCases.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants