Extend the baseline into a versioned discovery and navigation evaluation corpus #409

Before you start

• Searched current issues and discussions; no equivalent task was found. Related work is linked below.

Objective

Make the later discovery, enrichment and navigation work measurable with reviewed expectations and reproducible source/configuration metadata.

Scope

Extend the completed small visual baseline, existing search ranking fixtures, Overture quality gates and navigation replays into a documented shared evaluation protocol. Begin with existing online/browser capabilities; add native/offline cases as those capabilities become available. This is not another baseline screenshot-only task or the station ranking fix itself.

Verified current state

Rechecked against main at a8e03bc3efc49207d91f7c1855402c6cf149b8fd on October 7, 2026.

• Baseline guide contains 20 maps, four search views and two sheets with explicit deployment/data limits; it is intentionally small.
• Search recorder records API outputs for deterministic ranking evaluation; these outputs are not necessarily raw upstream payloads.
• Overture evaluation already provides conflation/search quality tooling and reviewed regional release gates.
• Mobile navigation replay tests already exercise deterministic navigation fixtures. The task coordinates and expands these tools rather than inventing evaluation from scratch.

Completion criteria

• Define versioned cases for urban/rural browsing, stations, aliases/multilingual names, chain branches, malls/tenants, closed/missing businesses and place-sheet partial states.
• Record expected entities/coordinates and assessor judgments independently of provider ordering; keep raw upstream responses, adapted API results and final UI ranking separate where diagnosing provider behavior.
• Pin app/style/source revisions, region/extract dates, provider capabilities and non-secret configuration; explicitly record unavailable deployment/data revisions.
• Define cold/warm cache isolation and synonym/query-order cases, including the regression tracked in Station searches prefer unrelated POIs and depend on synonym cache order #389 without duplicating its repair scope.
• Reuse GPS/navigation replays for route alternatives, mode transitions, recovery/GPS loss and off-route behavior; add network-denial offline search/routing cases only when supported and report unavailable cases explicitly.
• Agree metrics and budgets before comparing changes: useful visible labels/crowding, relevance/duplicates, branch correctness, requests/latency, partial loading and navigation correctness.
• Provide repeatable commands, fixtures and a before/after report format; meaningful regressions run in appropriate CI without live paid-provider dependency.
• Document evidence licensing/redaction and operator-controlled live capture; visual snapshots complement behavior assertions rather than serving as ground truth.

Verification approach

Run existing suites and new independently judged fixtures with pinned inputs. Demonstrate a deliberately changed result/order or navigation event is detected; verify replay determinism and redaction. Require a sample comparison report that separates data, provider, normalization, presentation and runtime outcomes.

Dependencies

#393 is the completed starting baseline. #389 remains the dedicated station-search repair. #297 is a semantic-recall experiment, not the shared evaluation protocol. The corpus supports #396/#397/#399 now and #398/#403/#404 as capabilities arrive; it need not wait for all those features.

This is backlog planning, not authorization to expand the implementation beyond this scope. Resolve named decisions during triage before scheduling work.