Skip to content

An INSERT in unique_conc spun at 100% CPU for three days, and survived its postmaster #212

Description

@jdatcmd

Found while clearing the box before a benchmark: a backend from the unique_conc suite, started Sat Jul 25 16:38:33, still running three days later at 98.9% CPU.

PID      PPID  STARTED                   ELAPSED       TIME        %CPU  COMMAND
1017577  1     Sat Jul 25 16:38:33 2026  3-02:37:08    3-01:50:19  98.9  postgres: postgres uconc [local] INSERT

It is a userspace spin, not a wait

State:  R (running)
utime = 26577363 ticks     (~3 days of pure userspace CPU)
stime = 4586 ticks         (essentially nothing in the kernel)
voluntary_ctxt_switches:      33
nonvoluntary_ctxt_switches:   19881103

33 voluntary context switches in three days. It never yields, never blocks, never sleeps. This is a CPU-bound loop, not a lock wait and not a slow query.

It outlived its postmaster

PPID is 1 and the cluster is a dead test directory (/tmp/pgcolumnar-uconc.ZCXwt0/data) whose postmaster is long gone. A backend normally notices postmaster death while waiting on a latch; a loop that never waits and never reaches a CHECK_FOR_INTERRUPTS never finds out. So whatever the loop is, it contains no interrupt check — which is why this was still burning a core three days later instead of dying with its cluster.

That also makes it a gate-integrity problem: it silently contaminated every measurement taken on this box since Jul 25. At least one "loaded box" retraction this week was measured alongside it.

Where it is looping

Sampled six times with gdb over ~20 seconds (non-assert -O2 build, so deeper frames are unreliable; the top frames and the pgcolumnar symbols are the signal). The stack is consistently an INSERT with index maintenance:

#0  LWLockReleaseInternal / LWLockAttemptLock / hash_search_with_hash_value  (varies)
...
#3  _bt_doinsert
#4  btinsert
#5  ExecInsertIndexTuples
#6  ExecInsert
#7  ExecModifyTable
#8  standard_ExecutorRun

and, in other samples, pgcolumnar frames reached from inside that same loop:

ColumnarReadOptions ()            from pgcolumnar.so   (via systable_endscan / index_close)
ColumnarReadZoneMapList ()        from pgcolumnar.so   (via systable_beginscan / index_open)
ColumnarComputeFullyDeletedGroups () from pgcolumnar.so
delete_vector_flush_buffer ()     from pgcolumnar.so

ColumnarReadOptions and ColumnarReadZoneMapList are catalog scans, and they are being re-entered from within the row-insert loop. Per-row catalog scans in a write path is the same shape as the quadratic fetch-by-row problem, and would explain a statement that makes progress so slowly it looks infinite — but I have not proven which of the two this is:

  • a genuinely infinite loop, or
  • a finite loop whose per-row cost is so bad it had not finished in three days.

Distinguishing them is the first job. The evidence does not settle it and I am not going to guess.

Reproduction status

Not yet reproduced. The suite is unique_conc (concurrent inserts against a unique constraint); the cluster used FIFOs s1.in/s2.in for two psql sessions, so a lost session may be part of the setup. Its server log shows only an unrelated startup error:

ERROR:  extension "citext" is not available

The process has been killed and the container rebooted, so this instance is gone. Full evidence — process state, cluster layout, and all six backtrace samples — was captured before it was destroyed.

Why this is worth its own issue

Whatever the loop turns out to be, two properties are defects on their own:

  1. No interrupt check. A loop in a write path that cannot be cancelled, and does not notice postmaster death, is a hang that a DBA cannot get out of without kill -9. CHECK_FOR_INTERRUPTS in the loop is the floor.
  2. Per-row catalog reads in the insert path. ColumnarReadOptions / ColumnarReadZoneMapList should be read once per statement, not per row, regardless of what caused this particular spin.

Related: #143 (fetch-by-row quadratic), #155 (bulk-load throughput).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions