[SPARK-58045][INFRA] Verify the linked JIRA matches the PR in merge_spark_pr.py

[gengliangwang]

What changes were proposed in this pull request?

This PR adds a JIRA-vs-PR verification step to dev/merge_spark_pr.py. After the committer enters the PR number (right after the existing Epic/Umbrella check, before the "Proceed with merging?" confirmation), the script now, for each SPARK JIRA id linked from the PR title, prints a verification block and requires an explicit y/N confirmation. The block:

• prints the PR title next to the ticket's summary, status, and issue type, so a mismatch is easy to spot;
• prints a red warning on the status line when the ticket is already Resolved/Closed (a fresh merge should target an open ticket);
• prints a red warning when the PR title and the JIRA summary have low word overlap (a Match: score below a threshold), which is a strong signal the PR references the wrong ticket;
• aborts the merge if the committer answers anything other than y, asking them to fix the PR title.

Implementation notes:

• A new pure helper format_jira_verification(...) renders the block and is covered by inline doctests. Coloring the warnings is gated behind a use_color flag (off in doctests so the expected output stays plain text, on at the real call site).
• Title/summary similarity is a stdlib Jaccard word-overlap (title_similarity): bracket tags ([SQL], the leading [SPARK-xxxx], ...) are stripped, words are lowercased, a small stopword set is dropped, and the two token sets are compared. No new dependency. The warning fires below SIMILARITY_WARN_THRESHOLD (0.2), tuned so an unrelated ticket (~0) warns while reworded-but-correct summaries stay above the line.
• The existing Epic/Umbrella loop already fetches each linked issue; it now also collects them into linked_issues, so the verification step reuses those objects and performs no extra JIRA round-trips.
• A small red(...) helper was factored out of print_error (no behavior change) and reused for the warnings.

No JIRA id is newly required by this change: [MINOR]/[TRIVIAL] PRs still merge with no SPARK id (the loop is a no-op for them). The existing Title.parse requirement is unchanged. [FOLLOWUP] PRs are handled (the id is extracted regardless of the tag), and Revert "..." PRs keep their title verbatim so the id is still extracted from it -- in both cases the warnings act as the intended nudge to double-check the ticket.

Why are the changes needed?

merge_spark_pr.py already prints each linked JIRA's summary, but nothing forces the committer to confirm the ticket actually matches the PR. A PR can therefore be merged against the wrong ticket - for example when a GitHub PR number in the body (Closes #NNNNN) is mistaken for a SPARK JIRA id - which resolves an unrelated, possibly already-Resolved ticket on merge. This step surfaces both signals (title/summary mismatch and an already-resolved ticket) and requires a human confirmation before the merge proceeds.

Does this PR introduce any user-facing change?

No. This only affects the committer-facing merge tooling.

How was this patch tested?

• python3 -m doctest dev/merge_spark_pr.py - all doctests pass, including the new title_similarity cases (match, no-match, partial overlap, [FOLLOWUP], and Revert title patterns) and the two format_jira_verification cases (one exercising both warnings, one clean).
• Manually rendered the block with use_color=True and confirmed both warnings are wrapped in the same red escape as print_error.
• Verified all added lines are within the 100-char limit and contain no non-ASCII characters.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

[dongjoon-hyun]

Thank you, @gengliangwang !

[dongjoon-hyun]

Does this work for [FOLLOWUP] PRs, @gengliangwang ? Maybe, add a test case for the follow-up PR patterns? And, Revert PR pattern, too.

[gengliangwang]

Thanks @dongjoon-hyun! Good call on both — it works for both patterns:

• [FOLLOWUP]: the JIRA id is extracted from the title regardless of the tag, so verification runs on the referenced ticket. A follow-up usually reuses the parent's (now-Resolved) ticket, so the Resolved/Closed warning tends to fire, and a reworded follow-up title can also trip the low-similarity warning — both advisory y/N, not hard blocks.
• Revert "...": the title is kept verbatim and the id is still extracted from it, so it points at the original ticket and shows the warnings — the intended nudge to double-check.

I've added title_similarity doctests for both patterns (973733b): the [FOLLOWUP] tag is stripped like any bracket tag (still scores 1.0 on a match), and the Revert "..." prefix stays verbatim (the extra word lowers the score but a genuine match still scores well above the threshold).

[dongjoon-hyun]

Is this true? Usually, the follow-up title is supposed to describe its code which is usually missed in the original PR. Like the following. The following is an example in branch-3.5.

$ git log --oneline | grep SPARK-56463
77117c7b32a [SPARK-56463][PYTHON][FOLLOWUP][3.5] Fix wrong kwargs
092f02df06e [SPARK-56463][PYTHON] Disallow unpickling UDT

[dongjoon-hyun]

Typically, the follow-up PR title's score is not the same with the original PR title (and the JIRA issue title).

[dongjoon-hyun]

Thank you for updating, @gengliangwang .