Skip to content

test(android): Add Jetpack Macrobenchmark for SDK init performance (JAVA-604)#5680

Merged
runningcode merged 1 commit into
mainfrom
no/java-604-add-jetpack-macrobenchmark
Jul 2, 2026
Merged

test(android): Add Jetpack Macrobenchmark for SDK init performance (JAVA-604)#5680
runningcode merged 1 commit into
mainfrom
no/java-604-add-jetpack-macrobenchmark

Conversation

@runningcode

Copy link
Copy Markdown
Contributor

📜 Description

Adds sentry-uitest-android-macrobenchmark, a Jetpack Macrobenchmark module that measures cold start of sentry-samples-android on a connected device. It reports timeToInitialDisplay (StartupTimingMetric) per iteration, taken from framework trace events — no trace markers needed in the SDK or the app. Each iteration's perfetto trace is saved for deeper analysis.

Details:

  • CompilationMode.Full() pins ART AOT compilation so dexopt state can't drift between runs; StartupMode.COLD force-stops (no pm clear) so app data persists.
  • Iterations are capped at 12 — back-to-back cold starts thermally throttle an unlocked-clock device after ~14 iterations.
  • The sample app gains a <profileable android:shell="true"/> manifest flag (required to profile its non-debuggable release build) — its only change.
  • The module's debug variant is disabled; the benchmark build type pairs the debuggable test APK with the app's release build via matchingFallbacks.
  • Requires androidx.benchmark 1.4.1: 1.3.4 fails with "Unable to confirm activity launch completion" on Android 16 (API 36).
  • Not run in CI; see the module README for device hygiene and interleaved A/B methodology.

💡 Motivation and Context

We currently rely on ad-hoc btrace/perfetto captures to measure Android SDK init cost. A Macrobenchmark gives a stable, reproducible, on-device measurement to validate init-time changes, and could be wired into CI later.

Closes #5679 (JAVA-604)

💚 How did you test it?

Ran connectedBenchmarkAndroidTest on a Galaxy A55 (Android 16): TTID median 514.5 ms (min 496.2, max 585.4, stdev 24.3, n=12), with no thermal degradation across iterations.

📝 Checklist

  • I added GH Issue ID & Linear ID
  • I added tests to verify the changes.
  • No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
  • I updated the docs if needed.
  • I updated the wizard if needed.
  • Review from the native team if needed.
  • No breaking change or entry added to the changelog.
  • No breaking change for hybrid SDKs or communicated to hybrid SDKs.

🔮 Next steps

  • Use this to A/B upcoming SDK init-time changes (interleaved rounds, see README).
  • Potentially wire into CI with a device farm later.

#skip-changelog

🤖 Generated with Claude Code

@linear-code

linear-code Bot commented Jul 2, 2026

Copy link
Copy Markdown

JAVA-604

@sentry

sentry Bot commented Jul 2, 2026

Copy link
Copy Markdown

📲 Install Builds

Android

🔗 App Name App ID Version Configuration
SDK Size io.sentry.tests.size 8.47.0 (1) release

⚙️ sentry-android Build Distribution Settings

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Performance metrics 🚀

  Plain With Sentry Diff
Startup time 346.28 ms 427.00 ms 80.72 ms
Size 0 B 0 B 0 B

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
abfcc92 304.04 ms 370.33 ms 66.29 ms
4e3e79d 365.83 ms 477.62 ms 111.79 ms
d8912da 329.94 ms 389.68 ms 59.74 ms
d15471f 294.13 ms 399.49 ms 105.36 ms
abf451a 332.82 ms 403.67 ms 70.85 ms
604a261 380.65 ms 451.27 ms 70.62 ms
806307f 357.85 ms 424.64 ms 66.79 ms
22f4345 307.87 ms 354.51 ms 46.64 ms
d217708 375.27 ms 415.68 ms 40.41 ms
c3ee041 310.64 ms 361.90 ms 51.26 ms

App size

Revision Plain With Sentry Diff
abfcc92 1.58 MiB 2.13 MiB 557.31 KiB
4e3e79d 0 B 0 B 0 B
d8912da 0 B 0 B 0 B
d15471f 1.58 MiB 2.13 MiB 559.54 KiB
abf451a 1.58 MiB 2.20 MiB 635.29 KiB
604a261 1.58 MiB 2.10 MiB 533.42 KiB
806307f 1.58 MiB 2.10 MiB 533.42 KiB
22f4345 1.58 MiB 2.29 MiB 719.83 KiB
d217708 1.58 MiB 2.10 MiB 532.97 KiB
c3ee041 0 B 0 B 0 B

Previous results on branch: no/java-604-add-jetpack-macrobenchmark

Startup times

Revision Plain With Sentry Diff
ccccf4a 311.09 ms 358.70 ms 47.61 ms

App size

Revision Plain With Sentry Diff
ccccf4a 0 B 0 B 0 B

@runningcode runningcode force-pushed the no/java-604-add-jetpack-macrobenchmark branch from e6f2ae9 to 9f8ff3d Compare July 2, 2026 11:24
@runningcode runningcode marked this pull request as ready for review July 2, 2026 14:40

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9f8ff3d. Configure here.

Comment thread settings.gradle.kts
…AVA-604)

Add sentry-uitest-android-macrobenchmark, a cold-start Macrobenchmark
targeting sentry-samples-android. It reports timeToInitialDisplay from
framework trace events, requiring no trace markers in the SDK or app.
Per-iteration perfetto traces are saved for deeper analysis.

CompilationMode.Full pins ART AOT; iterations are capped at 12 because
back-to-back cold starts thermally throttle after ~14 iterations. To
A/B an SDK change, interleave per-variant rounds so thermal drift does
not systematically penalize the second variant (see README).

Verified on a Galaxy A55 (Android 16): TTID median 514.5ms over 12
iterations. Requires androidx.benchmark 1.4.1; 1.3.4 fails to confirm
activity launches on API 36.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@runningcode runningcode force-pushed the no/java-604-add-jetpack-macrobenchmark branch from 9f8ff3d to 22177e4 Compare July 2, 2026 16:02
@runningcode runningcode enabled auto-merge (squash) July 2, 2026 17:05
@runningcode runningcode merged commit 5043164 into main Jul 2, 2026
70 checks passed
@runningcode runningcode deleted the no/java-604-add-jetpack-macrobenchmark branch July 2, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Jetpack Macrobenchmark for SDK init performance

3 participants