Skip to content

[FLINK-40513][runtime] Introduce multimodal AI functions for image understanding - #4520

Merged
yuxiqian merged 3 commits into
apache:masterfrom
haruki-830:FLINK-40513
Sep 2, 2026
Merged

[FLINK-40513][runtime] Introduce multimodal AI functions for image understanding#4520
yuxiqian merged 3 commits into
apache:masterfrom
haruki-830:FLINK-40513

Conversation

@haruki-830

Copy link
Copy Markdown
Contributor

This PR adds multimodal AI function support to Flink CDC Pipeline transforms.

AI_IMAGE_COMPLETE uses the standard OpenAI-compatible vision chat protocol. AI_IMAGE_EMBED and SupportsImageEmbedding are added as extension points, but image embedding is not yet supported by any built-in client. DashScope-specific embedding is intentionally out of scope.

Brief change log

  • Added SupportsImageTextGeneration and SupportsImageEmbedding.
  • Added AI_IMAGE_COMPLETE and AI_IMAGE_EMBED.
  • Added runtime parsing, type inference, capability validation, and UDF precedence handling.
  • Implemented OpenAI-compatible vision chat using MIME detection and Base64 data URLs.
  • Added tests for supported image formats, Base64 encoding, null input, and invalid input.
  • Updated the English and Chinese documentation.

Verifying this change

This change added tests and can be verified as follows:

  • Updated AiFunctionsTest and AiFunctionParserTest.
  • Updated OpenAiCompatibleModelClientTest and OpenAiCompatibleModelClientFactoryTest.
  • Updated FlinkPipelineAiFunctionITCase.
  • Verified the tests with both the default Flink profile and -Pflink2.
  • Verified the complete repository with mvn spotless:check.

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs and JavaDocs

Was generative AI tooling used to co-author this PR?
  • Yes (OpenAI Codex)

Generated-by: OpenAI Codex

@github-actions github-actions Bot added docs Improvements or additions to documentation composer common runtime add-ons labels Sep 1, 2026
@haruki-830
haruki-830 marked this pull request as ready for review September 2, 2026 01:48

@yuxiqian yuxiqian left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Haruki for the nice work, just left some minor comments.

Comment thread docs/content.zh/docs/core-concept/ai-model.md Outdated
@yuxiqian
yuxiqian merged commit fdb7652 into apache:master Sep 2, 2026
42 of 44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants