Skip to content

[runtime] CDC YAML supports inline Python UDF definition#4395

Draft
yuxiqian wants to merge 1 commit into
apache:masterfrom
yuxiqian:feat/python-udf
Draft

[runtime] CDC YAML supports inline Python UDF definition#4395
yuxiqian wants to merge 1 commit into
apache:masterfrom
yuxiqian:feat/python-udf

Conversation

@yuxiqian
Copy link
Copy Markdown
Member

@yuxiqian yuxiqian commented May 13, 2026

This PR provides the ability to write inline Python UDFs with YAML pipeline jobs. It could be used like this:

transform:
  - source-table: db.users
    projection: ID, py_normalize(EMAIL) AS EMAIL_NORM, py_accumulate(AGE) AS acc

pipeline:
  user-defined-function:
    - name: py_normalize
      classpath: org.apache.flink.cdc.python.PythonUdf
      options:
        python-executable: /usr/bin/python3
        source: |
          def eval(s: str) -> str:
            return s.strip().lower()
    - name: py_accumulate
      classpath: org.apache.flink.cdc.python.PythonUdf
      options:
        python-executable: /usr/bin/python3
        source: |
          total = 0
          def eval(x: int) -> int:
            global total
            total += x
            return total

The wrapper itself is implemented as a Java UDF as well. No changes are made in the existing framework except the following:

  • Runtime UDF binding names are slightly changed to allow defining multiple UDFs with the same class.
  • Added an overload function for UserDefinedFunction#getReturnType to pass extra context info.

@github-actions github-actions Bot added docs Improvements or additions to documentation common runtime build e2e-tests labels May 13, 2026
@yuxiqian yuxiqian force-pushed the feat/python-udf branch 2 times, most recently from 94f43dd to 73f04b2 Compare May 13, 2026 14:15
@leonardBang leonardBang self-requested a review May 14, 2026 12:39
@yuxiqian yuxiqian added this to the V3.7.0 milestone May 15, 2026
@yuxiqian yuxiqian marked this pull request as draft June 1, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build common docs Improvements or additions to documentation e2e-tests runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant