[Feature] Support native Flink 2.0 job lineage (FLIP-314) for Paimon tables #7306
Description
Search before asking
- I searched in the issues and found nothing similar.
Motivation
Flink 2.0 introduced native job lineage as a first-class feature (FLIP-314). When a job starts, Flink builds a LineageGraph of sources and sinks and fires it to any registered JobStatusChangedListener, enabling external lineage systems (for example open lineage) to consume accurate lineage without custom instrumentation.
The FLIP-314 proposal itself uses Paimon as one of its motivating example.
Solution
To support Flink’s native lineage tracking, implement the LineageVertexProvider interface in Paimon’s FlinkSource and FlinkSink classes. The implementation should return the table name in the name() method and the warehouse path in the namespace() method, enabling table-level lineage reporting to external systems. This is a small, self-contained change with no impact on Paimon’s internal data model or APIs.
Note: I see there were other similar issues reported related to PIP-5 ( https://cwiki.apache.org/confluence/display/PAIMON/PIP-5%3A+Paimon+Table+And+Data+Lineage+For+Flink), I ended up creating a separate issue because this change is much more contained with in Flink's effort to support native table-level lineage and should require lot less effort.
Anything else?
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!