Skip to content

fix(monitoring): bound state-age query in metrics endpoints - #313

Merged
danielvm-git merged 5 commits into
mainfrom
fix/dashboard-metrics-hang
Nov 6, 2026
Merged

danielvm-git merged 5 commits into
mainfrom
fix/dashboard-metrics-hang

Conversation

@danielvm-git

@danielvm-git danielvm-git commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

User description

Summary

Fixes BUG-2026-11-06-dashboard-metrics-hang — the v2.93.0 regression that left the admin dashboard stuck on "Loading dashboard..." ("the page wasn't loading") and forced the earlier rollback.

Root cause

06f86d39d made /api/monitoring/metrics run the new state-age report against deployments + cici_runs using the unbounded request context. The DashboardPage awaits that endpoint before setLoading(false), so a slow or write-locked DB left the page loading forever.

Fix

Apply the existing stateAgeQueryTimeout (2s) to both metrics endpoints — rename stateAgeReportForAlerts → stateAgeReportTimed and use it in handleMetrics and handlePrometheusMetrics.

Test plan

  • go test -run TestStateAgeReportTimedBoundsBlockedQuery ./components/monitoring — PASS (blocks QueryContext, asserts bounded return)
  • go test ./components/monitoring — PASS
  • go test -race ./components/monitoring — PASS
  • go test ./... — PASS
  • Preflight (go vet + go test + gosec + build + codeql + gitleaks) — green
  • trace-stories --strict — exit 0; 24 gate-script tests pass

Note

Merging is a fix: commit, so big-release will publish a patch (v2.94.1) and the Deploy workflow will fire. That deploy is still blocked by the separate VPS-side SCP issue (see #312), tracked independently of this fix.


PR Type

Bug fix, Tests


Description

  • Bound state-age DB queries in metrics endpoints with 2s timeout

  • Replace unbounded request-context call with stateAgeReportTimed

  • Add regression test blocking QueryContext to assert bounded return

  • Prevent admin dashboard hang caused by v2.93.0 regression


Diagram Walkthrough

flowchart LR
  A["handleMetrics"] -- "uses" --> B["stateAgeReportTimed"]
  C["handlePrometheusMetrics"] -- "uses" --> B
  D["stateAgeMetricValue"] -- "uses" --> B
  B -- "context.WithTimeout(stateAgeQueryTimeout)" --> E["bounded DB query"]
  E -- "returns unavailable report promptly" --> F["no dashboard hang"]
Loading

File Walkthrough

Relevant files
Bug fix
monitoring.go
Switch metrics endpoints to timed state-age report             

components/monitoring/monitoring.go

  • Update handleMetrics to call m.stateAgeReportTimed() instead of
    m.stateAgeReport(r.Context())
  • Update handlePrometheusMetrics to call m.stateAgeReportTimed() in
    writeStateAgeGauges
+2/-2     
state_age.go
Rename and document bounded state-age report                         

components/monitoring/state_age.go

  • Rename stateAgeReportForAlerts to stateAgeReportTimed
  • Add comment explaining bounded query prevents dashboard hang
  • Update stateAgeMetricValue to call renamed function
+7/-3     
Tests
state_age_test.go
Add regression test for blocked state-age query                   

components/monitoring/state_age_test.go

  • Add blockingDBer test double that blocks QueryContext until context
    cancellation
  • Add TestStateAgeReportTimedBoundsBlockedQuery to verify bounded query
    in blocked scenario
  • Assert report returns within timeout and marks all state machines
    unavailable
+52/-0   

BigBase Admin added 5 commits November 6, 2026 19:52
Reviewed recent deploy-file changes (Nov 3-4) as the likely broken-build source; SCP upload failure is at the SSH layer.
The v2.93.0 state-age gauge (06f86d3) made /api/monitoring/metrics
query deployments and cici_runs with the unbounded request context. The
admin dashboard awaits that endpoint before clearing its loading state,
so a slow or locked DB left it showing 'Loading dashboard...' forever
(the 'page not loading' regression).

Apply the existing 2s stateAgeQueryTimeout to the metrics endpoints too
(rename stateAgeReportForAlerts to stateAgeReportTimed and use it in
handleMetrics and handlePrometheusMetrics). Adds a regression test that
blocks QueryContext and asserts the report returns bounded.
Preflight green baseline; no active registry bugs; one code work item
(BUG-2026-11-06-dashboard-metrics-hang, already fixed with a regression
test) and one external item (deploy SCP failure, VPS-side).
@danielvm-git
danielvm-git merged commit c68b0de into main Nov 6, 2026
17 checks passed
@danielvm-git
danielvm-git deleted the fix/dashboard-metrics-hang branch November 6, 2026 23:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant