Skip to content

Issue #8143 : Add Database value validation transform - #8148

Open
mattcasters wants to merge 3 commits into
apache:mainfrom
mattcasters:issue-8143
Open

Issue #8143 : Add Database value validation transform#8148
mattcasters wants to merge 3 commits into
apache:mainfrom
mattcasters:issue-8143

Conversation

@mattcasters

Copy link
Copy Markdown
Contributor

What

Add a Database value validation transform that checks incoming rows against a target table before insert, so a PostgreSQL (or other RDBMS) failure is no longer "one column of one row among millions" with no useful field/value in the message.

Fixes #8143

Why

Table Output, Insert/Update and bulk COPY typically stop at the first database error. The message rarely names the stream field and the offending value. Data Validator can report all errors, but every rule is hand-authored and drifts from the live table.

How

  • At start, load table column types, lengths, nullability, defaults and encoding, then disconnect. Row processing is CPU-only.
  • Every mapped field on every row is checked. All failures on a row are concatenated onto one error-handling hop (DBV001DBV008).
  • Valid rows pass through unchanged.
  • Empty mapping matches stream fields to table columns by name.

Checks (generic JDBC + PostgreSQL enricher):

Code Meaning
DBV001 Null in a NOT NULL column
DBV002 String too long (Unicode code points for varchar)
DBV003 Encoding / NUL (U+0000)
DBV004 numeric(p,s) overflow after rounding to scale
DBV005 Type conversion
DBV006 Integer out of range (int2/int4/int8)
DBV007 Invalid UUID
DBV008 Invalid JSON/JSONB

Unique keys, foreign keys and CHECK constraints are not pre-flighted.

Tests

  • Unit: ColumnValueValidatorTest (11), PostgreSqlColumnValueConstraintsTest (6), DatabaseValueValidationMetaTest (load/save)
  • Integration: main-0043-database-value-validation (PostgreSQL unit-test-db) — mixed good/bad rows, golden error hop, only the valid row is inserted

Docs

User-manual page plus links from Table Output, Insert/Update, PostgreSQL bulk loader, Data Validator and error handling.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Run mvn clean install apache-rat:check to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
  • If you have a group of commits related to the same change, please squash your commits into one and force push your branch using git rebase -i.
  • Mention the appropriate issue in your description (for example: addresses #123), if applicable.

To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

Load target table constraints at start and report every failing field
on a row (length, encoding, numeric overflow, nulls, UUID/JSON) before
insert. PostgreSQL dialect enricher, error hop, docs, and integration
test 0043.

Fixes apache#8143
@mattcasters
mattcasters marked this pull request as ready for review August 28, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Data value validation

2 participants