Fixes #16408: Scope usage entity resolution to each run

[IceS2]

Describe your changes

Fixes #16408

Prevent repeated Usage workflows in the same Python process from publishing usage and queries to a deleted table ID after the same FQN is recreated.

Type of change

• Bug fix

High-level design

• Add a reusable entity-resolution engine and a table-specific plan builder under metadata.entity_resolution.
• Own one bounded, positive-only resolver cache across every staged file and join lookup in a Usage run; close it in finally without closing the borrowed OpenMetadata client.
• Resolve exact FQNs through REST. For case-insensitive and wildcard lookups, fetch uncached FQN candidates, enforce a per-candidate limit, hydrate current entities through REST, and validate their identity and include policy.
• Keep table parsing and service-specific normalization outside the generic engine. Other lineage callers, lower SDK caches, existing Usage error reporting, and backend validation are unchanged.

Per-run ownership retains reuse across staged batches without carrying IDs into another workflow. Successful results remain cached within the run until eviction; misses and failures are retried. No schema migration or configuration change is required.

Validation

• Resolver and Usage unit tests: 53 passed; 97% coverage of the new entity-resolution package. Additional SDK endpoint, restore, and table tests bring the focused total to 114 passed.
• PostgreSQL Usage integration: 3 passed. The regression hard-deletes only table A, recreates B with the same FQN, starts another UsageWorkflow in the same process, and asserts a different ID, successful publication, the expected count/date, and the exact regression-query association on B.
• Elasticsearch integration: 3 passed for mixed-case names, quoted dots, and literal wildcard characters, including delete/recreate hydration.
• Negative control using the pre-change lookup helper fails with publication to the deleted ID; the new path passes without clearing caches.
• Ruff lint/format and repository-configured type checks pass.
• A local PostgreSQL/OpenMetadata/Elasticsearch demo was exercised through real workflows and API reads: ingest metadata, publish usage to A, hard-delete A, ingest metadata again, then publish usage to B in the same process. B receives count 1 and the same SQL association.

No backend or UI changes; Java integration tests, Playwright, and screenshots are not applicable. OpenSearch was not exercised.

Checklist

• Linked to the issue with the required title format.
• Added behavioral unit and integration coverage, including the original failure scenario.
• Documented lookup and lifecycle contracts in code.
• No schema changes or migrations required.
• No UI changes requiring screenshots or recordings.

Confidence Score: 5/5

The PR appears safe to merge; the previously reported join-lookup failure isolation issue is fully addressed.

Findings

1.  Join Failures Abort Enrichment ▶

Summary

This PR replaces process-scoped Usage entity lookup state with a bounded resolver owned by each Usage run.

• Adds exact, case-insensitive, and wildcard FQN resolution with REST hydration.
• Shares resolver state across staged files and joined-table lookups, then clears it without closing the borrowed OpenMetadata client.
• Adds connector-specific table-reference normalization, including ClickHouse handling.
• Preserves per-record enrichment failure isolation and adds delete/recreate regression coverage.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    S[Usage source] --> T[Staged usage files]
    T --> B[MetadataUsageBulkSink]
    B --> R[Run-scoped EntityResolver]
    R --> E[Exact REST lookup]
    R --> Q[Candidate search]
    Q --> H[REST hydration and validation]
    E --> P[Usage, query, lifecycle, and join publication]
    H --> P
    B --> C[Close resolver and clear bounded cache]

Loading

_{Reviews (3) · Last reviewed commit: "Refactor usage table naming and resolver..."}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Issue Running usage ingestion on a service that was deleted and recreated results in a corrupted database #16408 is missing Shipping → Source.
• Issue Running usage ingestion on a service that was deleted and recreated results in a corrupted database #16408 is missing Shipping → Priority.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[greptile-apps]

Join Failures Abort Enrichment

If one joined-table lookup fails or exceeds the candidate limit, _resolve_usage_tables now raises instead of treating that join as unresolved. The exception reaches get_table_usage_and_joins, marks the entire source table as failed, and prevents its query and lifecycle data from being published. The previous lookup path caught these failures and skipped only the affected join target, so this boundary should preserve that failure isolation.

Knowledge Base Used: Metadata ingestion

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions]

✅ Playwright Results — workflow succeeded

Validated commit 2dd0fd5e96b6766e039f7c7ac3859b004f91206f in Playwright run 35349687337, attempt 1.

✅ 109 passed · ❌ 0 failed · 🟡 1 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 50m 26s

⏱️ Max setup 5m 25s · max shard execution 14m 49s · max shard-job elapsed before upload 19m 57s · reporting 6s

🌐 237.98 requests/attempt · 1.81 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

• Browser traffic was 237.98 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 1.81 per UI scenario (228 boots / 126 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	46	0	0	0	0	0
🟡 Shard ingestion-01	28	0	1	0	0	0
✅ Shard ingestion-02	35	0	0	0	0	0

🟡 1 flaky test(s) (passed on retry)

• Features/IncidentManager.spec.ts › Complete Incident lifecycle with table owner (shard ingestion-01, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gitar-bot]

Code Review ✅ Approved 1 closed / 1 findings

🟡 Medium risk

Run-scoped entity resolver for Usage workflows now prevents stale table ID reuse after recreate by hydrating with current IDs within each run. Joins resolution failure that previously suppressed query and lifecycle publication has been resolved.

✅ 1 closed

✅ Bug: Join resolution failure suppresses query & lifecycle publish

📄 ingestion/src/metadata/ingestion/bulksink/metadata_usage.py:275-287 📄 ingestion/src/metadata/ingestion/bulksink/metadata_usage.py:302-307 📄 ingestion/src/metadata/ingestion/bulksink/metadata_usage.py:366-380 📄 ingestion/src/metadata/ingestion/bulksink/metadata_usage.py:389-403
Previously, joined-table lookups went through get_table_entities_from_query → search_table_entities, which caught except Exception per service and returned None, so a failed join lookup was isolated and never aborted the source table's enrichment. The new _resolve_usage_tables (lines 389-416) only wraps resolution in try/finally for close(), so resolver errors — e.g. ValueError ("FQN candidate search exceeds the configured limit" / "Incomplete FQN candidate search" from engine _lookup), TypeError on an invalid search response, or a transient non-404 APIError — propagate out of __get_column_fqn → __get_table_joins. That call runs before ingest_entity_queries_data and _get_table_life_cycle_data inside the single try in get_table_usage_and_joins, so any join-target failure is caught at line 302, recorded as a failure, and skips query and lifecycle publication for an otherwise valid source table. Restore per-lookup isolation by swallowing resolution errors in __get_column_fqn (return None), mirroring the prior behavior.

Review coverage

Rules No rules evaluated

Functional validation Not enabled · Set up

Auto-approval Not enabled · Set up

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
12 New issues
0 Accepted issues

Measures
0 Security Hotspots
88.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud