[PR #8216/ad596660 backport][3.118] fix(content): exit on database failures

[patchback]

This is a backport of PR #8216 as merged into main (ad59666).

Problem

Content workers can lose database connectivity during PostgreSQL outage. content endpoint can continue returning HTTP 200 while Pulp removes worker's AppStatus heartbeat entry. operator sees online_content_apps: [] Hub readiness never completes.

Root Cause

heartbeat runs in detached asyncio task. previous failure path called:

exit(Arbiter.WORKER_BOOT_ERROR)

In deployed aiohttp/Gunicorn worker, did not terminate worker process. HTTP server continued serving /pulp/content/, so Pod remained Ready Kubernetes did not restart it.

Changes

• Use os._exit(Arbiter.WORKER_BOOT_ERROR) when initial content-app registration fails, including duplicate registration and other database failures.
• Use os._exit(Arbiter.WORKER_BOOT_ERROR) when a heartbeat write fails after the existing connection reset and second write attempt.
• Remove retry loop CONTENT_APP_MAX_DB_RETRIES setting.
• Add focused tests for registration and heartbeat database failures.
• Add a subprocess regression test that runs the detached heartbeat task with real os._exit, injects cleanup failure, and verifies the worker exits with the Gunicorn boot-error code.

worker now fails closed. Kubernetes restarts container. If PostgreSQL still unavailable, replacement process exits again; once PostgreSQL recovers, next replacement registers heartbeat normally. No registration retry loop required.

Validation

change tested against original unpatched Galaxy image using pre-PR #302 operator:

• PostgreSQL unavailable 45 seconds.
• Content workers terminated HaltServer('Worker failed boot.', 3).
• Each content Pod restarted three times while PostgreSQL unavailable.
• After PostgreSQL recovered, next replacement workers registered successfully.
• All four online_content_apps entries returned.

Focused heartbeat tests cover both registration error types, heartbeat write failures, and process-boundary termination. Ruff, Black, Python compilation, and diff checks also pass.

PR #302 remains possible operator-side mitigation already-running images contain Pulpcore defect, but change fixes failure content-worker level.