[JingsongLi]

Purpose

Add stored generated columns whose SQL expressions are part of the column schema. Both partition and non-partition columns can be filled before partition routing when omitted from an insert, for example dt DATE GENERATED ALWAYS AS (CAST(event_time AS DATE)) or label STRING GENERATED ALWAYS AS (UPPER(name)).

This is automatic filling of omitted columns, not a strict computed-column constraint. A supplied column, including an explicit NULL, is preserved without evaluating or checking its expression. Normal type, NOT NULL, primary-key, and partition constraints still apply. Updating a source does not recompute a generated column already supplied in the write row. Readers and optimizers cannot assume stored values satisfy the expression or infer partition filters from it.

• Persist optional generationExpression metadata on DataField and preserve it through serialization, copying, schema merging, catalog conversion, and cloning.
• Compile expressions for omitted columns into existing Paimon Transform operations once at writer initialization. Support CAST, date extraction and operations, string operations, literals in calls, and nested calls. References are limited to ordinary top-level columns; nested access, generated dependencies, nondeterministic functions, unsupported SQL, and TIMESTAMP WITH LOCAL TIME ZONE inputs or cast targets are rejected. The result type must match the declared field type.
• Reuse withWriteType to declare the input layout and materialize the full table row. When ordinary columns are also omitted, delegate to the underlying partial-column writer without generating unselected columns. Tables without generated columns retain the original routing and bundle fast paths.
• Support Spark 3.5 and Spark 4 GENERATED ALWAYS AS DDL and omitted-column inserts through SparkCatalog, including case-insensitive and quoted source names. Spark and the core writer use the same DEFAULT normalization and Transform evaluator for omitted columns, including Paimon's CAST formatting and invalid-input behavior. Normalize string literals to SQL-standard quoting independently of Spark's SQL printer, preserving quotes, backslashes, and newlines across Spark 3 and 4. Adapt SQL TRIM argument order. Automatic generation of BLOB or TIMESTAMP results above precision 6 is rejected where Spark cannot represent the result losslessly; explicitly supplied columns follow normal Spark writes.
• Preserve extra columns and generation metadata during schema merging, and row-tracking metadata during complete-row writes. Preserve the metadata through the Flink catalog without changing virtual computed-column semantics.
• Reject changes to generated definitions or names/types of referenced source columns. Document the protocol semantics, supported expressions, SQL syntax, and Java writer API.

Support all primary-key merge engines and Data Evolution partial-column updates in Java, Spark, and Flink. Generated values are filled before primary-key merging; normal merge rules apply without recomputing expressions after merging. DE partial writes preserve unselected generated columns and accept selected values without requiring expression dependencies. Generated columns cannot be used as rowkind.field.

Compatibility limitation: no writer-protocol feature gate is introduced. Older writers may ignore the attribute, so use clients that support it. Definitions and dependencies are immutable after table creation.

Validation

Focused tests passed: API/common 18, core 104, Flink 6, Spark 3.5/Scala 2.12 18, and Spark 4.1.2/Scala 2.13 18 (164 test executions).

• GeneratedDataFieldTest and GeneratedColumnRowTest: schema/JSON round trips, omitted-field generation, supplied values and NULL preservation, mixed supplied/omitted generated fields, no evaluation of supplied expressions even for invalid source input, binary/complex results, supported transforms, literals, and rejected expressions.
• GeneratedPartitionColumnTest and TableWriteTest: generation, routing, commits and reads across append, primary-key and Data Evolution tables; supplied partition values surviving reopen; unchanged NOT NULL checks, reordered inputs, RowKind/default handling, and the original routing path without generated columns. Adds real merge/read tests for partial-update, aggregation, and first-row engines, plus native DE partial updates and switching back to a complete input layout.
• Spark 3.5 and Spark 4.1.2 GeneratedPartitionColumnTest and DefaultValueTest: SQL creation and insertion, shared Transform/default evaluation for omitted columns, supplied values including NULL and mixed omissions, schema merging, complete row-tracking updates retaining supplied or existing values, DE updates retaining untouched generated columns or writing explicit results/NULL, quote/backslash/newline literal round trips, and generated-result representability checks applying only to omitted columns. Fixes the Spark 4 CI failure caused by backslash-escaped SQL literals.
• Core SchemaValidationTest and Flink catalog/property/virtual-computed-column regressions, including a synchronous DE merge procedure updating source and generated columns independently.
• Final Maven checks run without fast-build, including Checkstyle, Spotless, Enforcer, and RAT. git diff --check passes.