CI: emulator 'package' service unavailable during install (bundletool 'Can't find service: package')

[simonrozsival]

Summary

The Android emulator on CI intermittently enters a system_server restart loop under heavy resource pressure. While system_server is down/restarting, its package binder service is deregistered, so Google bundletool fails at APK/AAB install time while computing the device spec:

[BT : 1.18.3] error : Unexpected output of 'pm list features' command: 'cmd: Can't find service: package'
error XABAS0000: com.android.tools.build.bundletool.model.exceptions.AdbOutputParseException:
  Unexpected output of 'pm list features' command: 'cmd: Can't find service: package'
    at com.android.tools.build.bundletool.device.DeviceFeaturesParser.parse(DeviceFeaturesParser.java:42)
    at com.android.tools.build.bundletool.device.DdmlibDevice.getDeviceFeatures(DdmlibDevice.java:142)
    at com.android.tools.build.bundletool.device.DeviceAnalyzer.getDeviceSpec(DeviceAnalyzer.java:75)
    ...

Surfaces at Xamarin.Android.Common.targets(2885,3) during -t:Install. Same underlying disease as #11065 (which caught it in the test-run phase); this is the install phase.

Failing build (repro)

• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1509131
• Pipeline: dotnet-android (dnceng-public), PR-validation
• Stage / Job: Package Tests → macOS > Tests > APKs 2
• Failed step: build Mono.Android.NET_Tests-CoreCLR (dotnet build -t:Install)

Root cause — CONFIRMED from this build's logcat

The failing lane's logcat is in the published "Test Results - APKs .NET Release - macOS 2" artifact at net11.0-CoreCLR/logcat-Mono.Android.NET_Tests-CoreCLR.txt (CoreCLR builds under the Release configuration). It captures the whole failure window and shows a system_server restart loop:

system_server PID	"Entered the Android system server"	package service died
1925	(initial)	00:18:52.070
2657	00:18:56.242	00:21:39.442
3493	00:22:01.054	00:25:54.690
4712	00:25:58.942	—

system_server died and restarted ≥3 times in ~7 minutes. bundletool ran pm list features at 00:25:59.4 — ~0.5 s after the 3rd restart (pid 4712) began booting — so PackageManagerService had not yet re-registered the package binder → cmd: Can't find service: package.

Proximate trigger: the Android Watchdog (not a direct lmkd kill)

The fatal restart that broke our install is logged explicitly at 00:25:51.850:

W Watchdog: *** WATCHDOG KILLING SYSTEM PROCESS: Blocked in handler on ui thread (android.ui)
W Watchdog: *** GOODBYE!
E AndroidRuntime: DeadSystemException: The system died
I Zygote  : Process 3493 exited due to signal 9 (Killed)

The Watchdog killed system_server because its android.ui handler thread was blocked past the timeout.

Underlying condition: severe memory/CPU pressure (lmkd active throughout)

Throughout the window the emulator is thrashing — consistent with the OOM diagnosis in #11065:

• repeated lowmemorykiller: lmkd data connection dropped (00:18:51, 00:21:38, 00:25:53)
• continuous Zygote: Process … exited due to signal 9 (Killed)
• ActivityManager: SLOW OOM ADJ …, libprocessgroup: Successfully killed process cgroup …
• SystemServerTiming: SystemUserUnlock took to complete: 32940ms

So the accurate chain is: resource-starved emulator → system_server handler thread stalls → Watchdog kills & restarts system_server (repeatedly) → package binder transiently absent → bundletool's pm list features fails.

Relationship to #11065: same root disease (emulator resource starvation → system_server death), but a different proximate mechanism. Here: Watchdog kill (blocked android.ui handler). In #11065's build 1362050: direct lmkd kill. Two signatures, one cause.

Why the readiness gate didn't catch it

Start emulator (WaitForAndroidEmulator) passed at 00:18:08 because system_server pid 1925 was briefly up and serving package at that instant. It was Watchdog/pressure-killed 44 s later (00:18:52), after the gate had already returned. The gate is a point-in-time check with no re-verification before install.

Mechanism references (AOSP)

• cmd/pm print cmd: Can't find service: package when servicemanager has no package binder (non-blocking checkService() returns null). frameworks/native/cmds/cmd/cmd.cpp
• sys.boot_completed stays 1 across the restart because it is held by init's property service, separate from system_server. Android memory-management docs
• adbd is independent, so adb get-state=device and transport_id is unchanged across a restart. ADB reference
• init restarts the zygote (class main) service, which re-forks system_server, which re-registers all binders. init.zygote64.rc
• bundletool has no retry on this path: DeviceFeaturesParser.java
• lmkd/PSI killing under critical memory pressure: AOSP lmkd, lmkd README

Two independent secondary problems this exposes

1. Same-agent retry is ineffective and destructive. retryCountOnTaskFailure: 3 on the build+install task (a) can't help — same broken emulator/agent, and (b) re-running the full dotnet build -t:Install over a dirty obj/ produced spurious CS0436/CS0108/CS0114 duplicate-generated-type errors on retries 2–4, burying the real "device not ready" cause behind compiler errors (retries 2–4 never even reached the device). Incremental-build corruption class similar to BuildArchive incremental build bug: Java resources alternate between present and missing in APK on consecutive publishes #11543.
2. The failure is not distinguishable from a real product/test failure — XABAS0000 + CS0436 look like code bugs, not infrastructure.

Proposed direction

1. Detect early — before -t:Install, gate on package-service readiness, not just sys.boot_completed. Layered probe (weakest→strongest): getprop sys.boot_completed=1 → getprop init.svc.bootanim=stopped → service check package=found → pm path android=package:…. Treat Can't find service as "device not ready". (Aligns with CI: Emulator OOM kills system_server, causing Mono.Android.NET_Tests-NoAot to fail with no results #11065 Option 2.)
2. Fail distinguishably — classify the signatures (Can't find service: package, AdbOutputParseException, and in logcat: WATCHDOG KILLING SYSTEM PROCESS, lowmemorykiller/lmkd, multiple ServiceManager: service '…' died) as an infrastructure failure with a unique marker, not a build/test failure. (Overlaps with Add crash diagnostics to device test pipeline step #11298.)
3. Retry on a fresh device/agent — the fault is the emulator/agent, so a same-agent task retry is wrong; the retry must re-provision the emulator (or re-queue on a different agent) and must not re-run the full build over a dirty obj/.
4. Reduce frequency (address the starvation) — emulator stability flags for CI (Android emulator docs / android-emulator-runner): explicit RAM (-memory 4096 / hw.ramSize=4096), cold boot (-no-snapshot), -gpu swiftshader (swiftshader_indirect deprecated since emulator 36.4.9), -no-boot-anim, -noaudio, disable animations, and cap concurrent emulators per host (~3–4 on a ~14 GB macOS runner). The system_server restart loop indicates the emulator is under-resourced for the number running per host.

Related issues

• CI: Emulator OOM kills system_server, causing Mono.Android.NET_Tests-NoAot to fail with no results #11065 — CI: Emulator OOM kills system_server, causing Mono.Android.NET_Tests-NoAot to fail with no results (same root disease; test-run-phase manifestation; direct-lmkd-kill variant, build 1362050)
• Add crash diagnostics to device test pipeline step #11298 — Add crash diagnostics to device test pipeline step (detection/classification of these exact logcat signatures)
• Emulator version compatibility check for system images #12077 — Emulator version compatibility check for system images (same error string, different cause; ruled out here — boot_completed reaches 1 and recovers)
• BuildArchive incremental build bug: Java resources alternate between present and missing in APK on consecutive publishes #11543 — BuildArchive incremental build bug (related class of retry-over-dirty-obj corruption)