Skip to content

De-flake three unit tests (async ordering, sync timeout, off-main Core Data) - #25978

Open
jkmassel wants to merge 3 commits into
trunkfrom
jkmassel/deflake-three-unit-tests
Open

De-flake three unit tests (async ordering, sync timeout, off-main Core Data)#25978
jkmassel wants to merge 3 commits into
trunkfrom
jkmassel/deflake-three-unit-tests

Conversation

@jkmassel

@jkmassel jkmassel commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Three independent, test-only fixes for unit tests flagged flaky in Buildkite Test Engine. Each is a root-cause fix in a different test file; no production code changes.

Summary

  • CommentDetailViewModelTests.actionsWaitForReplyCountAfterDetailRenders (46% reliability) — the test synchronized on the wrong async signal, then asserted on unrelated state.
  • BlogJetpackTests.testSyncBlogsAndSignOut (84%) — a 1.0s expectation timeout, where the two sibling tests exercising the same sync pipeline wait 5.0s.
  • ReaderTopicServiceTest.testReaderSiteTopicUpdated (94%) — the suite's only async test, un-@MainActor'd, drove mainContext Core Data off the main queue.

Root Cause & Fix

1. actionsWaitForReplyCountAfterDetailRenders — wait on the state the assertion checks

vm.onAppear() kicks off two independent, unordered MainActor tasks: a comment fetch and a reply-count fetch. The test waited until the reply-count call was recorded, then asserted on the comment-detail state (vm.content == .loaded(detail)). Those are set by different continuations with no happens-before edge, and waitUntil checks its predicate before the first Task.yield() — so it returned as soon as the reply-count invocation was recorded, often before the fetch continuation ran applyLoaded(detail). ~half the runs observed content still .loading.

Fix: wait on the state the assertion actually checks, keeping the existing conjunct that the later resolveNumberOfReplies relies on.

-        await waitUntil { !service.numberOfRepliesInvocations.isEmpty }
+        await waitUntil { vm.content == .loaded(detail) && !service.numberOfRepliesInvocations.isEmpty }

2. testSyncBlogsAndSignOut — match the sibling timeout

The sync always completes, but under CI CPU contention it occasionally crosses 1s (a low-priority .global(qos: .background) notify hop in the capabilities fan-out adds jitter). The two sibling tests that drive the same syncBlogs pipeline wait 5.0s and don't flake.

-        wait(for: [syncExpectation], timeout: 1.0)
+        wait(for: [syncExpectation], timeout: 5.0)

3. testReaderSiteTopicUpdated — pin to the main thread

This is the only async test in ReaderTopicSwiftTest and it is not @MainActor, so its body runs on a background cooperative thread. Its seeding helpers create/relate ReaderPost/ReaderSiteTopic objects and save() on the main-queue mainContext. Off-queue Core Data access nondeterministically corrupts the store bookkeeping, surfacing ~6% of runs as NSCocoaErrorDomain 133010 "Cannot save objects with references outside of their own stores". The other 8 tests are synchronous (main-thread) and never flake. The awaits still suspend and free the main thread for the OHHTTPStubs response and the main-queue completion, so there's no deadlock.

-    func testReaderSiteTopicUpdated() async throws {
+    @MainActor func testReaderSiteTopicUpdated() async throws {

Test plan

  • Full suite green: xcodebuild -workspace WordPress.xcworkspace -scheme WordPress -testPlan WordPressUnitTests test (iOS 26.x sim, Xcode 26.6).
  • Stress-run the 46% test ~100 iterations (-test-iterations 100 -run-tests-until-failure) — 0 failures.
  • BlogJetpackTests and ReaderTopicSwiftTest green.
  • Repeated CI runs green (this PR is validated by re-running CI, not a single pass).

@jkmassel jkmassel added this to the 27.3 milestone Sep 2, 2026
@jkmassel jkmassel added the Testing Unit and UI Tests and Tooling label Sep 2, 2026
@jkmassel jkmassel self-assigned this Sep 2, 2026
@wpmobilebot

wpmobilebot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor
App Icon📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.
App NameWordPress
ConfigurationRelease-Alpha
Build Number34328
VersionPR #25978
Bundle IDorg.wordpress.alpha
Commitd912f9a
Installation URL2hh443v6g0dro
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot

wpmobilebot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor
App Icon📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.
App NameJetpack
ConfigurationRelease-Alpha
Build Number34328
VersionPR #25978
Bundle IDcom.jetpack.alpha
Commitd912f9a
Installation URL6ibrpfj44f6vg
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@jkmassel
jkmassel marked this pull request as ready for review September 2, 2026 14:42
@jkmassel
jkmassel requested a review from crazytonyli September 2, 2026 14:43
@jkmassel
jkmassel force-pushed the jkmassel/deflake-three-unit-tests branch from f9100cb to f143e2b Compare September 2, 2026 15:29
@dangermattic

Copy link
Copy Markdown
Collaborator
1 Warning
⚠️ This PR is larger than 500 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.

Generated by 🚫 Danger

@crazytonyli

Copy link
Copy Markdown
Contributor

@jkmassel There some already-merged changes in this PR.

@jkmassel

jkmassel commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

@jkmassel There some already-merged changes in this PR.

Yeah I goofed – there was a PR that kept failing because of the issue fixed by this PR and I wanted to stack them, but auto-merge was turned on, so when I changed the base it just merged instantly. Whoops!

@jkmassel
jkmassel enabled auto-merge September 2, 2026 21:36
@jkmassel
jkmassel disabled auto-merge September 2, 2026 21:41
Root-cause fixes for three unit tests flagged flaky in Buildkite Test
Engine. Test-only; no production changes.

- CommentDetailViewModelTests.actionsWaitForReplyCountAfterDetailRenders:
  wait on the state the assertion checks (vm.content == .loaded(detail))
  instead of the unrelated reply-count invocation, which is set by a
  different, unordered continuation.
- BlogJetpackTests.testSyncBlogsAndSignOut: bump the expectation timeout
  from 1.0s to 5.0s to match the two sibling tests that drive the same
  syncBlogs pipeline.
- ReaderTopicServiceTest.testReaderSiteTopicUpdated: annotate the suite's
  only async test @mainactor so its main-queue Core Data access stays on
  the main thread.
The wp-migration-notifications-explainer-{ltr,rtl} imagesets shipped as
Figma-exported PDFs that each embed a 1960x1408 raster, but the illustration
renders at its intrinsic ~204pt in a center-aligned, aspect-fit stack. Replace
the four oversized PDFs (4.95MB) with @2x/@3x PNGs rendered at the display size
(416KB total) — a ~4.5MB uncompressed reduction with no visible change.
The renditions in the prior commit were baked with a soft resampler and
read as blurry on-screen — worst on the dark variants (Laplacian-variance
sharpness ~28 vs ~200 for a clean render). Re-rendered all eight from the
source PDFs at the same sizes with high-quality interpolation.

Identical dimensions (408×270 @2x / 612×405 @3x) and no Contents.json
change. On-disk: ~405 KB → ~281 KB.
@jkmassel
jkmassel force-pushed the jkmassel/deflake-three-unit-tests branch from 04a2756 to d912f9a Compare September 2, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Testing Unit and UI Tests and Tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants