Re-record sample and library baselines on an API 37 emulator - #309
Conversation
AndroidTestifyBot
changed the title
fd0ace2 to
9b3158b
Compare
The baseline images checked into this repository for the Testify samples and library tests have been recorded on an API 29 (Android 10) emulator since 2021. Holding the emulator constant kept the signal clean. A failing screenshot test meant a change in Testify, not a change in the platform. But, the cost has outgrown the benefit: - API 29 no longer resembles the platform anyone builds against, so the samples demonstrate Testify against rendering behaviour that has since moved on. - An API 29 emulator cannot exercise 16 KB memory pages, which shipping devices now use and the current toolchain expects apps to support. - Several debugging and inspection features in recent Android Studio releases do not work against an API 29 device. This change re-records every in-repo baseline on an API 37 emulator, moves CI to match, and adds the device configuration task needed to keep captures stable on that platform.
9b3158b to
794e3ec
Compare
AndroidTestifyBot
left a comment
AndroidTestifyBot
left a comment
There was a problem hiding this comment.
Review
I went through the non-image changes line by line, verified the baseline set is complete (nothing dropped — every 29-* file has a 37-* counterpart, plus two new ones), ran Plugin:ktlintCheck Plugin:test locally (green), and then took the review note in the description seriously and diffed the re-recorded images against their API 29 predecessors.
The plugin change is clean and matches its siblings exactly. The Library README checks out against Library/build.gradle (artifact id, api project(":Ktx"), allWarningsAsErrors, JDK 25, maven-publish). The blog post's minSdk claim is correct.
Two things need to be resolved before this merges, plus a set of smaller items left inline.
🔴 The re-recorded Compose baselines bake in an ActionBar overlap
This is the regression the review note asks about, and it is not explained by the platform in the "fonts and ripples moved" sense — it is a real layout defect that the new baselines now lock in as expected.
Every ComposableScreenshotRule capture in Samples/Legacy now has a strip of the app's "Testify Sample" ActionBar drawn over the top of the Compose content, clipping it.
The clearest case is ComposableScreenshotTest_default.png (227x45). On API 29 it was the text Hello, Testify!. On API 37 the image is the indigo ActionBar with a clipped Testify in it — the composable under test is no longer visible at all. That baseline no longer tests anything.
Others confirmed by eye:
| Baseline | What the new image shows |
|---|---|
ComposableScreenshotTest_default |
content fully obscured by the ActionBar |
ComposableScreenshotTest_topAppBar |
ActionBar over the Compose TopAppBar, both titles clipped |
ComposableScreenshotTest_paddedBoxes |
ActionBar over the box's top border |
ComposableScreenshotTest_clientListItem |
ActionBar over the top of the card |
ComposeRuleInteropScreenshotTest_default |
ActionBar where the Default label was; label gone |
ComposeRuleInteropScreenshotTest_scrollingList |
ActionBar over the first row |
ja_JP/ComposableScreenshotTest_localeAndFontScale |
ActionBar over the text |
...and the six Scenario* twins of the above.
Cause. Ext/Compose/src/main/AndroidManifest.xml declares ComposableTestActivity with no android:theme, so it inherits the host app's theme. Samples/Legacy/src/main/AndroidManifest.xml:12 sets android:theme="@style/AppTheme", which is Theme.MaterialComponents.Light.DarkActionBar. With targetSdk = 36 running on API >= 35, edge-to-edge is enforced, the ActionBar is no longer inset out of the content frame, and it draws over the ComposeView that ComposableScreenshotRule captures.
Two things corroborate this:
ComposableScreenshotTest_dropdownMenuis not affected — it is a full-Activity capture, so the ActionBar is legitimately in frame.Samples/FlixandSamples/Flix/FlixLibraryCompose baselines are not affected —Theme.Flixisandroid:Theme.Material.Light.NoActionBar. Their captures are clean, which is the control case.
Why it matters beyond the samples. ComposableTestActivity ships in dev.testify:testify-compose. Any consumer whose application theme has an ActionBar gets the same overlap on an API 35+ device. This PR did not cause it, but it is the first thing to surface it, and recording the baselines is the moment it stops being visible.
Suggested fix: give ComposableTestActivity an explicit no-ActionBar theme in Ext/Compose/src/main/AndroidManifest.xml so it does not inherit the host's, then re-record the ~13 affected images. That also makes the Compose captures independent of the consuming app's theme, which seems desirable on its own.
🔴 Dropping _emulatorSetup from test_plugin silently skips most of ConfigurationCacheTest
Left inline on bitrise.yml. Short version: assumeDevice() uses Truth's assume(), so with no emulator the 11 device-requiring cases are skipped, not failed, and the workflow still goes green.
🟡 Smaller items
1. The emulator setup page still says API 29. docs/docs/get-started/2-configuring-an-emulator.md documents Q API level 29, x86, Android 10.0 for both Intel and M1. That page is the one the updated KDoc in AssertExpectedDeviceTest and ScenarioRuleAssertExpectedDeviceTest links to (https://testify.dev/docs/get-started/configuring-an-emulator), and the one the new blog post duplicates rather than links to. RECIPES.md:10 has the same stale block. A contributor who follows the doc rather than the PR description builds an API 29 AVD and hits the failure this PR warns about.
2. A docs image will 404 after merge. docs/docs/recipes/16-custom-bitmap.md:38 embeds:
https://github.com/ndtp/android-testify/blob/main/Samples/Legacy/src/androidTest/assets/screenshots/29-1080x2220@440dp-en_US/ScreenshotRuleExampleTests_captureMethodExample.png?raw=true
That path is deleted by this PR, so the image breaks on the live docs site the moment main moves. The three versioned_docs copies point at main too and will break identically — worth deciding whether to repoint them or pin them to a tag.
3. Paparazzi sample library baselines were not re-recorded. Samples/Paparazzi/PaparazziSampleLibrary/src/androidTest/assets/screenshots/29-1080x2220@440dp-en_US/ still holds three CastDetailScreenshotTest_* baselines, and CastDetailScreenshotTest is a real ComposableScreenshotRule test. CI does not catch this — test_paparazzi only runs verifyPaparazziDebug — but it contradicts "every in-repo baseline" and will fail for anyone running PaparazziSampleLibrary:screenshotTest on the new emulator. (Note these would show the same ActionBar overlap as above once re-recorded.)
4. Two stray artifacts landed as baselines. ScreenshotRuleExampleTests_exclusions.diff.png and ScreenshotScenarioRuleExampleTests_exclusions.diff.png are new — no API 29 predecessor. They are GenerateDiffs byproducts, not baselines. Same class as the pre-existing Library/.../testing.png, which is written by BitmapCompareTest under fileName = "testing" and jumped 480x606 -> 1080x2220 here. If screenshotRecord is sweeping the whole device output directory into the baseline directory, that is worth a separate issue.
5. Three different system images for one set of baselines. Per the description the images were recorded on the ARM64 16 KB image; bitrise.yml verifies on google_apis_ps16k / x86_64; .github/workflows/flix_sample.yml verifies on plain google_apis / x86_64. Flagged inline on both — mostly a question, since if CI is green all three agree, but it is worth being deliberate about rather than incidental.
6. disableStylusInput is missing from two places the other utility tasks appear in — ConfigurationCacheTest and the published copy of the plugin docs. Both inline.
7. Version claim. Blog line 16 says "Starting with Testify 7.0"; gradle.properties has testify_version=6.0.0. Inline.
Happy to re-review as soon as the Compose captures are sorted — everything else here is small.
046a402 to
45f38ab
Compare
daad221 to
c1c0cc4
Compare
AndroidTestifyBot
left a comment
AndroidTestifyBot
left a comment
There was a problem hiding this comment.
Re-review — c1c0cc4, 3b62014d
Thanks for the quick turnaround, and for the two design corrections — both are fair and I've dropped those threads.
Resolved
| Item | Status |
|---|---|
_emulatorSetup restored to test_plugin |
✅ bitrise.yml:298 |
disableStylusInput in ConfigurationCacheTest |
✅ verified locally — 15 cases, all green, [4] disableStylusInput, true PASSED |
disableStylusInput in the published plugin docs |
✅ 8-use-gradle-plugin.md:583 |
2-configuring-an-emulator.md + RECIPES.md on API 29 |
✅ both updated |
| GH Actions / Bitrise system image mismatch | ✅ both on google_apis_ps16k now, and the action is pinned to @2.0 instead of @main — nice |
| "Starting with Testify 7.0" | ✅ confirmed intentional |
Accepted as designed, no further comment: delta-E absorbing platform differences, and instrumented tests being runnable without the Gradle plugin. Both are right, and the mapper.ranchu.so comment in the workflow is exactly the kind of note that saves the next person a day.
I also checked the new AVD inputs against ndtp/android-avd-manager-action@2.0's action.yml — api-level, target, channel, arch, profile, ram-size, disk-size are all valid, the 2.0 tag exists, and 37.2-beta3 still reports SDK_INT == 37, so DeviceStringFormatter.androidVersion keeps producing the 37-1080x2220@440dp-en_US key. No baseline-key risk from the beta image.
🔴 Still blocking: the Compose baselines
This was the main item in the last review and it's the one thing that hasn't been touched or answered — no images changed in either new commit, and Ext/Compose/src/main/AndroidManifest.xml is unchanged. I suspect it got lost because it lived in the review body rather than an inline thread, so here it is again with harder evidence.
Pre-empting the delta-E answer: that argument is correct for the rest of this PR, and it's why I didn't flag any of the other 150-odd images. It doesn't apply here. This isn't anti-aliasing, ripple timing or font hinting moving under a perceptual threshold — the captured content is categorically different. A comparison that tolerated this would tolerate anything.
ComposableScreenshotTest_default.png, at native size (226x45 → 227x45), upscaled 4x nearest-neighbour so it's legible:
- API 29 baseline: the text
Hello, Testify!— the composable under test. - API 37 baseline: solid indigo, with the descenders of the word
Testifyfrom the ActionBar titleTestify Sampleclipped across it. The composable is not in the frame at all.
That baseline no longer asserts anything about the composable. It asserts that an ActionBar is indigo.
Confirmed by eye, all 13:
| en_US | |
|---|---|
| 1 | ComposableScreenshotTest_default — content fully replaced by the ActionBar |
| 2 | ComposableScreenshotTest_clientListItem — ActionBar over the top of the card |
| 3 | ComposableScreenshotTest_paddedBoxes — ActionBar over the box's top border |
| 4 | ComposableScreenshotTest_topAppBar — ActionBar over the Compose TopAppBar, both titles clipped |
| 5 | ComposeRuleInteropScreenshotTest_default — ActionBar where the Default label was; label gone |
| 6 | ComposeRuleInteropScreenshotTest_scrollingList — ActionBar over the first row |
| 7-12 | the six Scenario* twins of 1-6, all showing the same Testify Sample strip |
| ja_JP | |
|---|---|
| 13 | ComposableScreenshotTest_localeAndFontScale — ActionBar over the text |
Cause — now confirmed from the merged manifest, not inferred. Samples/Legacy/build/intermediates/merged_manifest/debugAndroidTest/mergeDebugAndroidTestManifest/AndroidManifest.xml:
<application
android:theme="@style/Theme.MaterialComponents.Light.DarkActionBar" >
...
<activity
android:name="dev.testify.ComposableTestActivity"
android:exported="false" />ComposableTestActivity carries no android:theme of its own, so it inherits the host application's ActionBar theme. With targetSdk = 36 on API >= 35 edge-to-edge is enforced, the ActionBar is no longer inset out of the content frame, and it draws over the ComposeView that ComposableScreenshotRule captures.
Two controls hold this up:
ComposableScreenshotTest_dropdownMenuis unaffected — it's a full-Activity capture, so the ActionBar is legitimately in frame. Only the view-scoped captures are wrong.Samples/FlixandSamples/Flix/FlixLibraryare unaffected —Theme.Flixisandroid:Theme.Material.Light.NoActionBar. I diffed their Compose baselines old-vs-new and they're clean. Same rule, same emulator, same PR, different host theme.
Scope beyond the samples. ComposableTestActivity ships in dev.testify:testify-compose. Any consumer whose application theme has an ActionBar gets this on an API 35+ device. This PR didn't cause it — it's the first thing to surface it, and recording the baselines is the moment it stops being visible.
Suggested fix: set an explicit no-ActionBar theme on the activity in Ext/Compose/src/main/AndroidManifest.xml so it stops inheriting the host's, then re-record those 13. Worth doing on its own merits — it makes Compose captures independent of the consuming app's theme, and it's the same thing androidx.test does for its own helper activities (InstrumentationActivityInvoker$BootstrapActivity pins @style/WhiteBackgroundTheme rather than inheriting).
If you'd rather ship the API 37 move now and treat this as its own bug, that works for me too — but then these 13 shouldn't be re-recorded in this PR, because a baseline recorded against a known-broken layout is harder to notice later than a missing one.
🟡 Two items from last time, still open
1. docs/docs/recipes/16-custom-bitmap.md:38 will 404 on merge. Unchanged, and it embeds a raw-GitHub image from a path this PR deletes:
.../blob/main/Samples/Legacy/src/androidTest/assets/screenshots/29-1080x2220@440dp-en_US/ScreenshotRuleExampleTests_captureMethodExample.png?raw=true
One-character fix (29- → 37-). The three versioned_docs copies point at main too and break identically — those are frozen snapshots so they arguably want pinning to a tag rather than editing, but either way they go broken.
2. Samples/Paparazzi/PaparazziSampleLibrary/.../29-1080x2220@440dp-en_US/ still holds three CastDetailScreenshotTest_* Testify baselines. CI won't catch it (test_paparazzi only runs verifyPaparazziDebug), so this is purely about PaparazziSampleLibrary:screenshotTest failing for a contributor on the new emulator. Fine to defer if it's deliberate — worth a word in the description either way, since it's the one exception to "every in-repo baseline". (It uses ComposableScreenshotRule, so it'd need the fix above before it could be re-recorded cleanly.)
🟢 Nits on the new commits
Three small things inline — a mangled system-image name, a prompt-style inconsistency, and the screenshotTest sample output blocks. None of them blocking.
| - Q API level 29, x86, Android 10.0 (Google APIs) | ||
| - RAM: 1536 MB | ||
| - Phone: Pixel 3a (Obsolete) (1080x2220 440dpi) | ||
| - 16 KB Page Size Google APIs **x86_64** v8a System Image, API 37.0 (CinnamonBun; Android 17.0) |
There was a problem hiding this comment.
Small naming slip: x86_64 v8a isn't a thing — v8a is the ARM suffix (arm64-v8a), and it got carried across when the M1 block was copied. In the AVD manager the Intel image is listed as 16 KB Page Size Google APIs Intel x86_64 Atom System Image.
| - 16 KB Page Size Google APIs **x86_64** v8a System Image, API 37.0 (CinnamonBun; Android 17.0) | |
| - 16 KB Page Size Google APIs **x86_64** System Image, API 37.0 (CinnamonBun; Android 17.0) |
Two related notes on this page and its copies:
RECIPES.md:10and the blog post (2026-08-20-samples-update.md:51) both give the ARM 64 image as the single unconditional recommendation, with no Intel variant. This page is the only one that splits by host architecture.RECIPES.mdhad one config before this PR so that's not a regression, but it now points Intel contributors at an ARM image.- The
.github/workflows/flix_sample.ymlcomment added inc1c0cc4says API 37.0 and 37.1 crash-loop SurfaceFlinger. All three of these docs prescribe 37.0. See the note on that workflow.
| # aborts on `!hasReadColorBufferDma`, taking system_server with it. | ||
| # 37.2-beta3 is the first image where that is fixed -- it carries a | ||
| # DEV-branch build, so the fix has not reached a release image yet. | ||
| api-level: '37.2-beta3' |
There was a problem hiding this comment.
Pinning to @2.0, moving to google_apis_ps16k and adding the settle step all look right — and I checked the inputs against ndtp/android-avd-manager-action@2.0's action.yml: api-level, target, channel, arch, profile, ram-size and disk-size are all valid, and the 2.0 tag exists. 37.2-beta3 still reports SDK_INT == 37, so DeviceStringFormatter.androidVersion keeps producing the 37-… key and the baselines still resolve. No concern there.
The question is the divergence this creates. The comment states plainly that:
37.0 and 37.1 both crash-loop SurfaceFlinger: mapper.ranchu.so aborts on
!hasReadColorBufferDma, taking system_server with it.
Meanwhile API 37.0 is what everything else in the repo prescribes — bitrise.yml:124 (api_level: '37.0'), docs/docs/get-started/2-configuring-an-emulator.md, RECIPES.md:10, and the blog post's setup instructions.
So either:
- the crash is specific to the GH Actions environment (nested virt +
swiftshader_indirect), in which case 37.0 is fine locally and on Bitrise and it's worth saying so in the comment — otherwise the next person reads this and assumes the documented local setup is broken; or - it's a property of the 37.0 system image, in which case Bitrise's
_emulatorSetupis running on a known-crashy image and the contributor docs are sending people to one.
Worth resolving either way — right now the repo simultaneously documents 37.0 as the target and 37.0 as broken.
Minor: enable-hw-keyboard defaults to false in this action and cores defaults to 2, while the emulator docs specify keyboard input enabled and 4 cores. Pre-existing and almost certainly irrelevant for Flix (no text input), so only worth touching if you're already in here.
| Disables stylus input on the device. The stylus input method can disrupt screenshots of text input fields, so it's good to turn it off. | ||
|
|
||
| ```console | ||
| ~/: ./gradlew FlixSample:disableStylusInput |
There was a problem hiding this comment.
Content is right. Two cosmetic things from the copy across from Plugins/Gradle/README.md:
Prompt style. This file uses $ in its command blocks — disableSoftKeyboard at line 573 and hidePasswords at line 603 are both $ ./gradlew …. The block here uses the README's ~/: style:
| ~/: ./gradlew FlixSample:disableStylusInput | |
| $ ./gradlew FlixSample:disableStylusInput |
The screenshotTest sample output still doesn't list the task. This was the second half of my earlier note and I don't think it got picked up. Now that disableStylusInput is in ScreenshotTestTask.dependsOn, a real screenshotTest / screenshotRecord run prints it — but the transcripts in both files still go straight from deviceTimeZone to disableSoftKeyboard to hidePasswords:
docs/docs/get-started/8-use-gradle-plugin.md:167and:248Plugins/Gradle/README.md:101and:182
It belongs between deviceTimeZone and disableSoftKeyboard in all four:
> Task :FlixSample:disableStylusInput
------------------------------------------------------------
Disables stylus input on the device
------------------------------------------------------------
Success
(Those lines are outside this PR's diff hunks so I couldn't anchor there directly.)
…es to the original baseline images
3b62014 to
02e875d
Compare
AndroidTestifyBot
left a comment
AndroidTestifyBot
left a comment
There was a problem hiding this comment.
Re-review — ad292d5, 02e875d
The blocker is fixed. I verified it by running the suites against a real API 37 emulator rather than reading the images, and everything passes.
✅ Compose baselines — resolved and verified
You went a different way than I suggested, and I think the route you picked is the better one. Pinning AppTheme.NoActionBar on ComposableTestActivity (ad292d5) would have fixed the overlap but left every other Legacy capture rendering edge-to-edge — status bar blanked, nav bar gone, content reflowed. Opting the sample out of enforcement instead brings the whole set back into line with the API 29 originals, which is what makes a 165-image diff reviewable at all.
MainActivityScreenshotTest_default, the three states side by side, tells the story:
| Status bar | Nav bar | |
|---|---|---|
API 29 (main) |
dark blue colorPrimaryDark |
black |
API 37 before the opt-out (ad292d5) |
blank white | blank white |
API 37 now (02e875d) |
dark blue | black |
And the captures that were broken are whole again — ComposableScreenshotTest_default is Hello, Testify! instead of a slab of ActionBar, ComposeRuleInteropScreenshotTest_default has its Default label back, topAppBar is a clean Compose Sample bar. All 13 confirmed against their API 29 predecessors.
Dimensions corroborate it: of the changed baselines, only 15 still differ in size from their API 29 counterparts, and every Legacy one is now explainable — 1080x1868 → 1080x1934 is +66px, exactly a 24dp status bar at 2.75x, and 226x45 → 227x45 is a one-pixel text metric. Before this commit those same images were 1080x2066.
✅ Verified by execution
Against emulator-5554 — sdk_gphone16k_arm64, SDK 37, 1080x2220, 440dpi, en-US, animation scales off:
| Suite | Result |
|---|---|
LegacySample:screenshotTest |
OK (88 tests) — 130s |
FlixSample:screenshotTest |
OK (20 tests) — 24s |
FlixLibrary:screenshotTest |
OK (3 tests) — 6s |
Library:connectedDebugAndroidTest |
53 tests, 0 skipped, 0 failed — 132s |
Plugin:ktlintCheck Plugin:test |
green, 15 config-cache cases |
164 instrumented tests across every module this PR touches, all green against the re-recorded baselines.
Working tree still clean afterwards, so nothing was silently re-recorded.
Also fixed without my needing to chase it: the two stray *_exclusions.diff.png byproducts are deleted.
Approving
Everything below is small and none of it blocks the API 37 move. One request before you merge, then some things for later.
Please land before merging
docs/docs/recipes/16-custom-bitmap.md:38 still points at a path this PR deletes:
.../blob/main/Samples/Legacy/.../screenshots/29-1080x2220@440dp-en_US/ScreenshotRuleExampleTests_captureMethodExample.png?raw=true
29- → 37- and it's done. It's the only item here with a live consequence — the image on testify.dev goes broken the moment main moves, and nothing in CI will tell you.
Worth a line in the description
The edge-to-edge opt-out isn't mentioned anywhere outside the commit subject — not in the PR description, not in the blog post. That's out of character for a description that documents the AVD spec down to the VM heap, and it matters more than the things that are documented: it's a behavioural change to the sample app, and it's the mechanism holding the new baselines comparable to the old ones. The next person bisecting a sample rendering change will want to find it.
The blog post is worth a sentence too. It currently tells readers the samples moved to API 37 to stop "demonstrating Testify on rendering behaviour that has since moved on" — while the Legacy sample now opts out of the single largest rendering change of that era. Both things can be true, but a reader who diffs the images will notice, and one sentence saying "we opt Legacy out deliberately, to keep the baseline diff reviewable" closes it.
Follow-ups, not for this PR
1. The library-side issue is still live for consumers. Ext/Compose/src/main/AndroidManifest.xml declares ComposableTestActivity with no android:theme, so it still inherits the host application's. Any consumer on an ActionBar theme targeting SDK 35+ gets the overlap this PR just spent two commits removing from the samples — and now that Legacy opts out of enforcement, the repo no longer has a test that would catch it. The canary went quiet. Worth an issue for pinning a theme on the activity in Ext/Compose itself, which fixes it for everyone and doesn't depend on the host.
2. Legacy opts out, Flix does not. Flix's captures grew to the full 1080x2220 and MoviePoster to 1080x1110, so it's rendering edge-to-edge. Each module is internally consistent and both suites pass, so this is only a question: deliberate — Legacy stays legacy, Flix shows the modern platform — or just where the fix happened to land? If deliberate it's a nice split and worth saying so; the two samples then demonstrate both sides of the migration.
3. 22 orphaned baselines. Nothing in the repo references BackwardsCompatibility — no test class, no source file, anywhere. The 22 BackwardsCompatibility*.png files (plus BackwardsCompatibilityests_*.diff.png, with the typo) are dead, and this PR carries them into the new directory as renames. Pre-existing on main, not something you introduced, but a re-record pass is the natural moment to drop them. That also explains the 88-tests-to-112-baselines gap I saw.
4. Library/.../testing.png is still the BitmapCompareTest byproduct rather than a baseline — createBitmapFromActivity(fileName = "testing", …) writes it, no test reads it, and it went 480x606 → 1080x2220 here. Same class as the .diff.png files you just removed.
5. Paparazzi sample library still has its three CastDetailScreenshotTest_* baselines under 29-1080x2220@440dp-en_US. CI doesn't run them, so this is purely PaparazziSampleLibrary:screenshotTest failing for a contributor on the new emulator.
Cosmetic, from last round
The three threads from last round are still open on the same lines and I haven't duplicated them — a mangled x86_64 v8a system-image name (2-configuring-an-emulator.md:8), a ~/: prompt in a $-prompt file (8-use-gradle-plugin.md:588), and the screenshotTest transcripts in four places that don't list disableStylusInput now that it's a dependsOn. Take or leave them.
Nice work on the emulator archaeology in the workflow comment, and on picking the fix that keeps the baseline diff meaningful rather than the one that was merely closest to hand. 🚀
| <item name="colorPrimary">@color/colorPrimary</item> | ||
| <item name="colorPrimaryDark">@color/colorPrimaryDark</item> | ||
| <item name="colorAccent">@color/colorAccent</item> | ||
| <item name="android:windowOptOutEdgeToEdgeEnforcement" tools:targetApi="35">true</item> |
There was a problem hiding this comment.
This is the line that resolves the Compose blocker, and I think it's the right call — it fixes the whole Legacy set at once rather than just the Compose captures, and it's what keeps the new baselines comparable to the API 29 originals. Verified working: LegacySample:screenshotTest is OK (88 tests) on a real API 37 emulator, and MainActivityScreenshotTest_default has its dark-blue status bar and black nav bar back.
Two things I'd attach to it.
It deserves a mention in the PR description. Right now the only place this appears is a commit subject. The description documents the AVD down to the VM heap but says nothing about the sample opting out of a platform behaviour — and this is the mechanism the whole baseline diff rests on. Same for the blog post, which tells readers the samples left API 29 so they'd stop "demonstrating Testify on rendering behaviour that has since moved on", while Legacy now opts out of the largest such change. One sentence — we opt Legacy out deliberately, to keep the baseline diff reviewable — reconciles the two and saves the next person a bisect.
It's an escape hatch with a shelf life. windowOptOutEdgeToEdgeEnforcement was introduced as a migration aid, and the tools:targetApi="35" suppression you needed is the tell. I'm not going to assert a removal timeline I can't verify from here, and it demonstrably works on API 37 today — but the day it stops being honoured, this exact set of baselines breaks again, in the same way, and that's the scenario this PR exists to handle. Worth a comment on the line pointing at whatever issue tracks the eventual migration:
| <item name="android:windowOptOutEdgeToEdgeEnforcement" tools:targetApi="35">true</item> | |
| <!-- Opt out of edge-to-edge enforcement so the sample renders comparably to the | |
| pre-API-35 baselines. This attribute is a migration aid and will eventually | |
| stop being honoured; when it does, these baselines need re-recording. --> | |
| <item name="android:windowOptOutEdgeToEdgeEnforcement" tools:targetApi="35">true</item> |
Related, and the reason I'd not let this sit forever: Ext/Compose/src/main/AndroidManifest.xml still declares ComposableTestActivity with no android:theme, so consumers on an ActionBar theme targeting SDK 35+ still get the overlap. With Legacy opted out, the repo no longer has a test that would surface it. Follow-up issue rather than anything for this PR.
2 participants
What does this change accomplish?
The baseline images checked into this repository for the Testify samples and library tests have been recorded on an API 29 (Android 10) emulator since 2021. Holding the emulator constant kept the signal clean. A failing screenshot test meant a change in Testify, not a change in the platform.
But, the cost has outgrown the benefit:
This change re-records every in-repo baseline on an API 37 emulator, moves CI to match, and adds the device configuration task needed to keep captures stable on that platform.
How have you achieved it?
Emulator and baselines
bitrise.yml: the shared_emulatorSetupAVD moves fromapi_level: '29'to'37'.All baseline images were re-recorded on API 37. Testify keys baselines by device configuration, so they move from
29-1080x2220@440dp-en_USto37-1080x2220@440dp-en_US(and the matching landscape,ja_JP,fr_FRanden_CAkeys):Samples/LegacyLibrarySamples/FlixSamples/Flix/FlixLibrary36 of the 165 images are byte-identical to their API 29 predecessors and appear in the diff as pure directory moves; the rest changed as a result of the platform.
The two
.a11y.jsonreports underSamples/Legacymove to the new device key as well.AssertExpectedDeviceTestandScenarioRuleAssertExpectedDeviceTestassert on the device key string, so their expected values and the KDoc describing the recording device were updated to API 37.New
disableStylusInputGradle taskThe stylus input method can disrupt screenshots of text input fields, so it is turned off before capturing.
DisableStylusTaskissuesadb shell settings put secure stylus_handwriting_enabled 0. It is registered inTestifyPluginand added to thedependsOnlist ofScreenshotTestTaskalongsidehidePasswords,disableSoftKeyboard,localeandtimeZone, so it runs automatically as part ofscreenshotTest. Documented in the Gradle plugin README next to the other utility tasks.Documentation
Library/README.md— the core library module had no README. Adds an overview of what the library does, its core use cases, and the build/test commands for the module.docs/blog/2026-08-20-samples-update.md— explains the emulator change for users: why API 29 was held for so long, why it is being retired, why API 37, and the emulator configuration needed to run the tests. Notes that nothing changes for consumers' own baselines.Version bumps
jacoco0.8.10 → 0.8.15 anddokka2.1.0 → 2.2.0.Scope of Impact and Testing instructions
Not a breaking change for consumers. There is no public API change,
minSdkis unchanged, and Testify still does not require a particular API level. The baselines here are the ones checked into this repository for our own samples and library tests; consumers' baselines and the emulator they record on remain their choice.Breaking for contributors. The API 29 baseline directories are gone. Running the sample or library tests against an existing API 29 AVD will fail looking for a baseline directory that no longer exists. Configure a new emulator:
pixel_3askin, keyboard input enableden_US); window animation, transition animation and animator duration scales all set tooffTo verify, on that emulator:
The full Bitrise pipeline exercises the same set on the API 37 AVD.
Review note. Re-recorded baselines are not self-verifying — a real regression is easy to miss in a diff of several hundred images. When reviewing, the question to ask of each changed image is whether the difference is explained by the platform (system UI, fonts, ripples, insets) rather than by a change in Testify's capture or comparison behaviour.
Notice
Warning
This change must keep
mainin a shippable state; it may be shipped without further notice.