Fixes 33544: derive datalake validation fixtures from EXECUTION_DATE

[harshach]

Describe your changes:

Fixes #33544. Salvaged from the closed PR #33248 (tracking: #33528).

The fixture rows in test_validations_datalake.py were built at module import time from the wall
clock, while the test case handed to the validator declares a fixed execution date:

EXECUTION_DATE = datetime.strptime("2021-07-03", "%Y-%m-%d")     # line 39
...
DL_DATA = ([... 30, datetime.utcnow() - timedelta(days=1), ...], ...)   # eight rows
...
execution_date=EXECUTION_DATE.timestamp()                        # line 1246

So the data and the declared execution date sat about five years apart, by a margin that changed on
every run. This derives the eight timestamps from EXECUTION_DATE and runs the parametrized test
under @freeze_time(EXECUTION_DATE), so anything that reads the clock during validation sees the
instant the data was built around.

Scope, stated plainly: no test in the file asserts on inserted_date today — it appears only in
the DataFrame column list — so nothing is failing now and I could not reproduce a red run. This is
determinism hardening: it removes the wall-clock dependency before an assertion or a validator change
inherits it, most visibly around a UTC day boundary where utcnow() at import and the assertion can
land on different days.

freezegun is already declared in ingestion/setup.py's test extras (line 526).

Type of change:

• Improvement

High-level design:

N/A — test fixture change, 10 lines.

Tests:

Use cases covered

• The datalake data-quality validators are exercised against fixture data whose timestamps have a
  fixed, known relationship to the declared execution date.

Unit tests

No new tests; the existing 57 parametrized cases now run deterministically.

57 passed in 42.45s

Backend integration tests

Not applicable — ingestion only.

Ingestion integration tests

Not applicable — this is a unit-test fixture.

Playwright (UI) tests

Not applicable.

Manual testing performed

1. python -m pytest tests/unit/observability/data_quality/test_validations_datalake.py — 57 passed.
2. Same run against the unmodified file from main — also 57 passed, which is why this is described
   as hardening rather than a fix.
3. Confirmed datetime.utcnow() no longer appears in the file, and that inserted_date is referenced
   only by the DataFrame column list.

UI screen recording / screenshots:

Not applicable.

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #33544.
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: not applicable.
• For UI changes: not applicable.
• I have added tests: existing tests cover this; no behaviour change.

🤖 Generated with Claude Code

Confidence Score: 4/5

The PR is not safe to merge until freezegun is available in the supported unit-test environments or the new dependency is avoided.

Findings

1.  Missing unit-test dependency ▶

Summary

This PR anchors datalake validation fixture timestamps to the fixed execution date and freezes the validator clock for deterministic inserted-row-count testing.

• Replaces eight import-time wall-clock timestamps with offsets from EXECUTION_DATE.
• Freezes the parametrized validation test at the same execution date.
• Introduces a unit-test dependency that is absent from the repository's supported local unit-test environments.

_{Reviews (1) · Last reviewed commit: "Fixes 33544: derive datalake validation ..."}

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Linked issue Datalake validation fixtures are built from the wall clock, not the declared execution date #33544 is not added to the Shipping project.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[gitar-bot]

Code Review ✅ Approved

🟢 Low risk

Derives datalake validation fixture timestamps from EXECUTION_DATE instead of wall-clock time and runs the parametrized test under @freeze_time, eliminating a time-dependent test dependency that could cause flakiness around UTC day boundaries. All 57 existing test cases pass without behavioral changes.

Review coverage

Rules No rules evaluated

Functional validation Not enabled · Set up

Auto-approval Not enabled · Set up

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[greptile-apps]

Missing unit-test dependency

This top-level import makes the module uncollectable in supported local unit-test environments. Both nox -s unit and make install_dev_env install the test-unit extra, but freezegun is declared only in the separate test extra. As a result, these documented workflows fail with ModuleNotFoundError before running any tests. Add freezegun to the unit-test dependency set or avoid requiring it here.

[mohittilala]

this should fine as nox -s unit-tests does install all the deps, and tests are green on this pr.

[greptile-apps]

You're right that nox -s unit-tests installs the test extra, so that workflow is covered and the green run is expected. My concern is the other supported unit-test path: nox -s unit and make install_dev_env install test-unit, where freezegun is not included. Since this test lives under tests/unit, those workflows can still fail during collection with ModuleNotFoundError. Please either add freezegun to test-unit or avoid the import; otherwise, if unit-tests is the only intended workflow, the dependency finding can be dismissed.

[mohittilala]

nox -s unit is not strictly recommended for all unit tests. Unit tests do include many other packages that aren't being installed with test-unit extras. test-unit seems bare minimum and I couldn't find use of it in any workflows or recommendations.

[greptile-apps]

That's fair. I overstated this as a supported-environment blocker. nox -s unit is a minimal/legacy session whose test-unit extra already omits dependencies needed by various unit tests, while the full nox -s unit-tests workflow installs .[test], which includes freezegun. make install_dev_env is likewise a development setup and does not guarantee that every unit test is runnable. This import therefore does not introduce a uniquely actionable dependency regression, so I’m withdrawing the finding; no change is needed.

[github-actions]

✅ Playwright Results — workflow succeeded

Validated commit 53066eadb667704f0e6d0d01d021ae542da0142c in Playwright run 35293269214, attempt 1.

✅ 110 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 25m 1s

⏱️ Max setup 5m 17s · max shard execution 12m 42s · max shard-job elapsed before upload 18m 50s · reporting 3s

🌐 235.47 requests/attempt · 1.79 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

• Browser traffic was 235.47 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 1.79 per UI scenario (216 boots / 121 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	46	0	0	0	0	0
✅ Shard ingestion-01	29	0	0	0	0	0
✅ Shard ingestion-02	35	0	0	0	0	0

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud