[SPARK-56923][SDP] Implement SCD1 Batch Processor; Merge Microbatch onto Auxiliary

[AnishMahto]

Approved AutoCDC SPIP: https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7

---

This is a stacked PR. Review incremental diff here: AnishMahto/spark@SPARK-56249-merge-tombstones-onto-microbatch...SPARK-56923-SCD1-merge-microbatch-onto-auxiliary

Link to previous PR: #55993

---

Preamble:

The SCD type 1 flow is a foreachBatch streaming query on an input change-data-feed, and is responsible for reconciling the incoming change data onto some target table that follows SCD1 replication semantics.

SCD1 flows also maintain an "auxiliary" table to keep track of early-arriving out-of-order received events state. Each microbatch will need to reconcile against this auxiliary table as well, and update the auxiliary table's state appropriately for future microbatches.

Merge Microbatch onto Auxiliary:

The auxiliary table should be updated such that;

1. Stale tombstones (delete events now succeeded by upserts/deletes in the incoming microbatch) should be removed from the auxiliary table. They will never be meaningful for SCD1 again.
2. New tombstones introduced in the incoming microbatch should be persisted in the auxiliary table, for future microbatches to reconcile agianst.

Implement this update operation as a MERGE.

[szehon-ho]

Reviewed the incremental diff on top of SPARK-56249-merge-tombstones-onto-microbatch (stacked on #55993). The mergeMicrobatchOntoAuxiliaryTable implementation and Scd1BatchProcessorMergeSuite coverage look solid overall — merge clauses match the stated aux-table semantics, projection to keys + _cdc_metadata is correct, and tie-breaking/idempotency cases are well documented in tests.

Suggestions

1. Add merge tests for composite and dotted keys — deduplicateMicrobatch and applyTombstonesToMicrobatch already have coverage for composite keys (region, customer_id) and literal-dot names (`user.id`), but Scd1BatchProcessorMergeSuite only uses a single id key. Since mergeMicrobatchOntoAuxiliaryTable uses the same k.quoted / doKeysMatch plumbing, a lightweight test (e.g. advance tombstone or insert for composite keys, and one dotted-key case) would lock in MERGE/catalog behavior without duplicating the full merge matrix.

2. PR description typo — "reconcile agianst" → "reconcile against".

Thanks for the stacked compare link — much easier to review just this slice.