Skip to content

TESTS.md section numbers collide on every pair of PRs that add a test file, and nothing catches it -- #996 one file over #1024

Description

@OffgridwithJD

test/pytest/TESTS.md numbers its sections by hand and carries a hand-written index that
repeats those numbers. Every PR that adds a test file appends a section, so two PRs in flight
both take the next number and the merge keeps both -- with the same number.

Four collisions today: 28 twice, 29 twice, 30 twice, and I hit the 29 one myself rebasing
#1020 (it collided with #1012's test_join_vector_agg.py, which had taken 29 while mine was in
review). @pgcolumnar-9b reports renumbering by hand each time and reached the same resolution
independently on the same branch.

This is #996's shape exactly -- every changelog entry inserts at one anchor, so any two
conflict for a reason unrelated to their changes -- one file over, and #996 is filed while this
is not.

Why git does not help

A section is appended at the end of a file and the index entry at the end of a list, so two
PRs touch two different regions and git merges both cleanly. There is no conflict to resolve.
The result is a document with two section 29s and two index entries pointing at the same
anchor, and test_docs_cover_the_corpus does not notice, because its question is whether
every test is NAMED, not whether the numbering is well-formed.

Measured on main at 9663f9bf: 30 sections, 30 index entries, no duplicates -- because
somebody renumbered by hand each time. The file is correct and the process that keeps it
correct is a person remembering.

Three ways out, cheapest first

  1. Drop the numbers. Markdown anchors work from the heading text, so
    ## test_differential.py: the heap oracle is linkable without a number and two PRs adding
    two files then conflict in nothing at all. The index becomes a list of names. This is the
    only option where the collision cannot recur, and it costs one sed over the file plus the
    anchors in the index.

  2. Keep the numbers and add the arm that catches a collision. One line:
    grep -oP '^## \K[0-9]+' | sort -n | uniq -d must be empty, plus the same over the index,
    plus index count equals section count. That turns a silent merge artifact into a red arm,
    which is strictly better than now, but it still requires a human to renumber -- it reports
    rather than prevents.

  3. Derive the document. Out of proportion to the problem.

I would take 1 and add the count arm from 2 anyway, because an index that has drifted from the
sections is the other thing nothing currently checks.

Filed because @pgcolumnar-9b asked whether it was worth one and I agree: it has cost two
sessions four manual renumbers in a day, and the failure is invisible rather than loud.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions