Skip to content

feat: add date-window and validator-version filters to rebuild_missing_validation_reports #1773

Open
cka-y wants to merge 3 commits into
mainfrom
feat/1792
Open

feat: add date-window and validator-version filters to rebuild_missing_validation_reports #1773
cka-y wants to merge 3 commits into
mainfrom
feat/1792

Conversation

@cka-y

@cka-y cka-y commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary:

Extends rebuild_missing_validation_reports (functions-python/tasks_executor) by adding optional filters so we can re-validate a historical window against the prod validator. Closes #1762.

  • filter_downloaded_after / filter_downloaded_before: ISO-date window on downloaded_at.
  • filter_validator_version_prefix: target datasets lacking any report matching the prefix (e.g. "8."), not just a non-exact version.
  • include_all_datasets: consider all datasets of a feed, not just the latest.

All default to preserve existing behavior. README + unit tests updated.

Expected behavior:

Dry run reports the count of published feeds' datasets in the window missing an 8.* report; non-dry run triggers a validation workflow per candidate against prod and writes the reports. Response params echo the applied filters.

Testing tips:

Fields must be nested under "payload". Dry run first:

{
  "task": "rebuild_missing_validation_reports",
  "payload": {
    "dry_run": true,
    "filter_downloaded_after": "2026-03-01",
    "filter_downloaded_before": "2026-06-01",
    "filter_validator_version_prefix": "8.",
    "include_all_datasets": true,
    "validator_endpoint": "https://gtfs-validator-web-mbzoxaljzq-ue.a.run.app"
  }
}

Check params echo your filters and review total_candidates, then flip dry_run to false to trigger.

  • include_all_datasets: true is what covers all datasets in the window (else latest-only).
  • Count filters by the feed's operational_status (default published) with an exclusive upper bound, so it won't match a raw downloaded_at BETWEEN query.
  • Idempotent: already-triggered and already-8.* datasets are skipped. Try "limit": 10 for a small batch first.
    Notes for testers:
  • The count reflects each feed's latest dataset only, operational_status = 'published', and an exclusive upper bound — so it won't match a raw SELECT ... FROM gtfsdataset WHERE downloaded_at BETWEEN ... count.
  • Re-running is idempotent: already-triggered datasets are skipped, and any dataset that already has an 8.* report is excluded.
  • Try omitting the new fields to confirm the original behavior is unchanged, and try "limit": 10 for a small end-to-end batch first.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@cka-y cka-y changed the title feat: rebuilding validation report between dates for a version feat: add date-window and validator-version filters to rebuild_missing_validation_reports Jul 23, 2026
@cka-y
cka-y marked this pull request as ready for review July 23, 2026 15:18

@davidgamez davidgamez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-run Validator 8.0.1 on datasets from March - May

2 participants