Skip to content

Simplify the indexed repository dashboard and verify nightly sync - #21

Merged
ewlarson merged 3 commits into
developfrom
feature/index-dashboard-nightly
Oct 3, 2026
Merged

ewlarson merged 3 commits into
developfrom
feature/index-dashboard-nightly

Conversation

@ewlarson

@ewlarson ewlarson commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

The dashboard showed all discovered GitHub repositories, including 27 never harvested here, and mixed database counts with search availability. It now lists only repositories with documents in the live Elasticsearch index and shows three totals: indexed repositories, searchable records, and sources scheduled nightly. Remove the Never Harvested Here card, redundant Aardvark/source-count columns, and stale explanatory text. Keep GitHub and filtered Search API links. Report index unavailability explicitly instead of showing misleading zero counts.

The nightly workflow previously succeeded immediately after enqueueing work. It now waits for every child harvest and requires both zero import errors and a successful search-index synchronization. Discovery failures abort before changing source settings; empty, skipped, incomplete, failed, or timed-out batches fail the workflow. The daily 07:15 UTC schedule remains in place, with a five-hour completion deadline inside the six-hour workflow limit.

Validation:

  • 29 focused tests pass, including PostgreSQL harvest integration, actual trigger-to-wait wiring, index pagination, index unavailability, and import/index failure cases.
  • Dashboard rendered and visually checked with production counts (10 repositories and 318,408 searchable records).
  • Ruff and diff checks pass. Full backend CI: 1,956 passed, 82 skipped, 1 expected-failure test unexpectedly passed.
  • Production end-to-end verification completed: https://github.com/OpenGeoMetadata/ogm-api/actions/runs/37098707615 . The new completion verifier confirmed all 10 child harvests and search synchronizations succeeded with zero import/index errors, including 177,789 USGS and 40,703 geobtaa records. The deployed workflow itself still reports only enqueue success; this PR fixes that monitoring gap.

Deployment order: deploy the updated application scripts before the next scheduled workflow uses --wait. No deployment is included in this PR.

@ewlarson
ewlarson merged commit b2828b7 into develop Oct 3, 2026
3 checks passed
@ewlarson
ewlarson deleted the feature/index-dashboard-nightly branch October 3, 2026 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant