Skip to content

Alpha gate: assert and document the blast radius of a single backend crash #217

Description

@jdatcmd

Alpha gate 4 of 4, from the pre-alpha review note. @ChronicallyJD for review.

What is true today

The note asks that a backend crash be confirmed and documented to drop
connections and reinitialize without corrupting committed columnar data, and
says explicitly that this should be an asserted guarantee rather than an
inference from repros. It is currently an inference.

test/recovery.sh covers a neighbouring scenario, not this one. It SIGKILLs the
entire postmaster process group, using a negative-PID kill so the backends die
with it, then restarts into crash recovery and asserts two things against a heap
mirror: committed writes replay, and an in-flight uncommitted write leaves
nothing visible. It also confirms recovery actually ran by finding a redo entry
in the log.

That is whole-cluster crash and restart. The scenario the note is about is one
backend dying while the postmaster lives, which is the #210 shape: a SIGSEGV in
a parser takes down a single backend, the postmaster reinitializes shared
memory, and every other session is dropped. Whether committed columnar data is
intact on the other side of that is not asserted anywhere.

Nothing in the documentation states the guarantee either. grep -niE 'crash|blast radius|never corrupt|committed data' over docs/limitations.md
and docs/ARCHITECTURE.md returns nothing.

Why the guarantee is arguable rather than hopeful

The header of test/recovery.sh states the mechanism: columnar data lives in
the relation main fork through the buffer manager and WAL, and metadata lives in
WAL-logged heap catalogs. If that holds with no exception, a single backend
death is no different from any other backend death in PostgreSQL, and the
guarantee follows from core rather than from this extension. The work is to
confirm there is no exception, not to build a new mechanism.

The thing to look for is any write path that bypasses the buffer manager or
leaves shared state that a dead backend does not clean up. Two candidates worth
checking specifically rather than assuming: an in-progress write state flushed
partially at the point of death, and any cache or shared allocation whose
invariant a dead backend leaves broken for the next one.

Plan

Add the single-backend-death scenario to the crash suite, then write the
guarantee down with the conditions under which it holds.

Todo

  • Add a scenario to test/recovery.sh, or a new suite if it does not fit,
    that kills one backend mid-write with SIGSEGV or SIGKILL while the
    postmaster lives.
  • Assert in that scenario: the postmaster survives and reinitializes; other
    sessions are dropped and the cluster accepts new connections; data
    committed before the kill matches the heap mirror exactly; the partial
    write leaves no visible rows and no partial row group.
  • Repeat the kill at several points in the write path, including mid-stripe
    and between a group flush and the commit, rather than at one point.
  • Audit for exceptions to the buffer manager and WAL statement above, and
    name any found in the issue. An exception is a defect, not a caveat to the
    guarantee.
  • Run the same scenario against a table under concurrent readers, since the
    dropped-connection half of the claim is about other sessions.
  • Write the guarantee into docs/limitations.md or docs/ARCHITECTURE.md,
    stating what survives a backend crash, what does not, and that it rests on
    columnar data going through the buffer manager and WAL.
  • Add the suite to the SUITES array in test/run_all_versions.sh, or it
    is never gated.

What closes this

A suite that fails if the guarantee is broken, and a documented statement of the
guarantee that cites the mechanism it rests on. The inference from #210 repros
is not enough on its own.

The set

Alpha gates from the pre-alpha review note, ranked cheapest-highest-value first:
#214 fuzz the parsers, #215 limitations and release status, #216 untrusted-input
boundary, #217 backend-crash blast radius.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions