Skip to content

[FLINK-39758] Fix DECIMAL OOB in SchemaMergingUtils#4419

Open
haruki-830 wants to merge 4 commits into
apache:masterfrom
haruki-830:FLINK-39758
Open

[FLINK-39758] Fix DECIMAL OOB in SchemaMergingUtils#4419
haruki-830 wants to merge 4 commits into
apache:masterfrom
haruki-830:FLINK-39758

Conversation

@haruki-830
Copy link
Copy Markdown
Contributor

Summary

This commit fixes the DECIMAL precision overflow (Out-Of-Bounds) issue in SchemaMergingUtils and SchemaUtils, which previously caused pipeline failures when merging DECIMAL types from multiple tables with significantly different precisions and scales.

Key Changes

  1. New createDecimalBounded Utility Method
  • Introduced createDecimalBounded method in SchemaMergingUtils to handle DECIMAL precision overflow gracefully
  • When computed precision exceeds DecimalType.MAX_PRECISION (38), the method truncates scale digits instead of throwing an exception
  • Truncation formula: lossDigits = precision - MAX_PRECISION, then resultPrecision = precision - lossDigits, resultScale = scale - lossDigits
  • Follows the same approach as Flink SQL for handling DECIMAL overflow
  1. SchemaMergingUtils Fix
  • Replaced Preconditions.checkArgument with createDecimalBounded call in mergeDecimalTypes()
  • Removed the hard failure that threw IllegalArgumentException when merged precision exceeded 38
  • SchemaUtils Fix
  • Applied the same createDecimalBounded fix in inferWiderType() for DECIMAL-to-DECIMAL merging
  • Applied the same fix in mergeExactNumericsIntoDecimal() for DECIMAL-to-integer merging
  1. Comprehensive Testing
  • Updated SchemaUtilsTest to verify that overflow cases now return bounded DECIMAL types instead of throwing exceptions
  • Added testMergingDecimalWithOutOfBoundPrecisions with 6 parameterized test cases covering various overflow scenarios
  • Refactored FlinkPipelineComposerITCase to support generic merging test infrastructure via runGenericMergingTest

Before fix: Merging DECIMAL(5, 5) and DECIMAL(38, 0) throws IllegalArgumentException (requires precision 43, max 38)
After fix: Merging DECIMAL(5, 5) and DECIMAL(38, 0) returns DECIMAL(38, 0)

JIRA Reference

https://issues.apache.org/jira/browse/FLINK-39758

@haruki-830
Copy link
Copy Markdown
Contributor Author

@lvyanquan could you please help review this PR

@lvyanquan
Copy link
Copy Markdown
Contributor

There are compilation failures.

春栖 added 3 commits May 31, 2026 21:33
- Remove InitializeTablesEvent usage
- Remove testcontainers ImmutableMap import
- Simplified test cases from 6 to 3 scenarios
The AlterColumnTypeEvent serialization now includes comments field.
Updated 3 test cases in decimalOOB() to match the expected output.
- Fix mergeExactNumericsIntoDecimal to use createDecimalBounded for overflow handling
- Fix test expectations: CreateTableEvent uses first table's DECIMAL precision
- Fix test replace logic: only replace tableId={} to avoid corrupting comments={}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants