fix: Underlay Push scale updates

[isTravis]

Problem

Large communities could not push to Underlay. The push died after 33 seconds with JS heap out of memory, and showed as "Running" in the UI indefinitely — the worker recorded the error on the WorkerTask, but nothing ever finalized the UnderlayPushLog, so the failure was invisible.

Root cause: pushToUnderlay loaded every pub with four hasMany associations eagerly joined in one query. Sibling hasMany includes produce a cartesian product, each repeating the pub's full payload (description, htmlDescription, metadata). One pub alone expands to 4,368 rows.

Changes

Query / memory

• separate: true on all four hasMany includes — makes association cost additive, not
  multiplicative.
• Pub hydration runs in chunks of 250, so Sequelize model instances are never all resident.
• File bytes (release HTML, PDF/EPUB exports, images) stream to Underlay as each pub is
  mapped, then drop. Peak file memory is one pub's worth, not the collection's.

Throughput

• Uploads run through a bounded 12-way concurrency pool. Each PUT is ~390 ms of pure
  latency.
• Commit now uses Underlay's async mode (?async=true) and polls the session. Previously a
  multi-minute commit blew the 60s client timeout and was retried four times, re-running
  the work server-side each attempt. Falls back cleanly to a synchronous 201, so this does
  not require Underlay and PubPub to deploy in a particular order.

Resumability / observability

• Cache entries are checkpointed every 200 pubs (after their uploads land), so a push that
  times out or crashes resumes instead of restarting from zero.
• UnderlayPushLog is finalized when the worker process dies, not just when the task throws.
• An adopted running row is re-pointed at the task actually running it — otherwise it was
  invisible to the crash-finalizer and stuck at running forever.
• Progress logging (Checkpointed N pub(s); M file(s) uploaded).
• Stored warnings cap at 100 with the true total kept in the message.
• getBaseVersion / ensureCollection now send credentials. Unauthenticated, they 404 on a
  private collection, which was read as "no versions yet" and caused every push after the
  first to fail with a misleading 409 conflict.

Verification

Run against a local copy of the real database:

• Old vs. new query compared across 200 pubs (60 worst-case + 140 by creation order),
  fingerprinting every association incl. nested user / externalPublication:
  identical — 4,043 attributions, 1,324 exports, 161 edges, 224 releases.
• Chunked vs. whole-set facet resolution: identical across 500 pubs (all 500 signatures
  distinct, so the cascade genuinely varies).
• Memory flat at 2.03–2.09 GiB from 2,200 → 4,000 pubs, returning to 139 MiB idle.
• Upload throughput 1.85 → 22.5 files/sec.
• 1,400 checkpointed pubs survived a mid-push crash and were skipped on resume.
• Full push → commit end to end on a small community (autogeddon, 18 pubs): committed
  v1.0.0, 218 records / 419 files, integrity confirmed on the Underlay side.
• 84 unit tests; typecheck and lint clean.

Deploy notes

• infra/stack.yml's worker service has no memory limit. Consider setting WORKER_MAX_OLD_SPACE_MB (added here) and/or a container limit, so a runaway push fails one task rather than the host.

Follow-ups (pre-existing, not addressed)

• Legacy .epub exports 403 even after the presigned-URL retry; every one was skipped
  locally. Worth confirming whether the fallback works against prod S3
• Underlay has no orphan-file GC, and resumable pushes now quietly depend on that. If file
  cleanup is ever added, exclude recently-uploaded blobs.

[Copilot]

Pull request overview

Improves the scalability and reliability of “push to Underlay” for large communities by reducing memory blowups during hydration/mapping, streaming file uploads with bounded concurrency, adding checkpointed resumability, and ensuring push logs are finalized even when worker processes die.

Changes:

• Chunked pub hydration + separate: true hasMany includes to avoid cartesian explosion and reduce heap usage.
• Stream files during mapping with bounded upload concurrency, checkpoint cache entries during the push, and negotiate/commit with async commit polling.
• Improve push-log correctness/observability: adoption of stale running logs, crash-finalization, and warning truncation caps.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
workers/tasks/pushToUnderlay.ts	Chunked pub hydration, early ensureCollection, streaming uploads + periodic cache checkpoints, capped warning logging
workers/queue.ts	Adds worker-thread heap limit support and crash-finalization for underlay push logs
server/underlayPushLog/queries.ts	Caps stored warnings, fixes running-log adoption, adds “fail log by workerTaskId” finalizer
server/underlayPushEntry/queries.ts	Extracts cache-entry upsert so it can be called mid-push (checkpointing)
server/underlay/mapping.ts	Adds fileHashes to support streaming pushes without retaining bytes
server/underlay/incremental.ts	Streaming upload pipeline, upload pool concurrency, checkpoint hooks, avoids memoizing file bytes
server/underlay/client.ts	Authenticated reads for private collections; async commit + polling; missing-file recovery helpers
server/underlay/tests/pushLogWarnings.test.ts	Tests warning truncation/message behavior
server/underlay/tests/pushLogAdoption.test.ts	Tests adoption behavior repointing workerTaskId
server/underlay/tests/incremental.test.ts	Adds streaming/concurrency/checkpointing tests; updates memoization expectations
server/underlay/tests/client.test.ts	Tests async commit handshake + authenticated reads
server/envSchema.ts	Adds WORKER_MAX_OLD_SPACE_MB env var schema

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.