diff --git a/CHANGES.md b/CHANGES.md
index fc34c155256c..da3a54204e69 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -76,6 +76,9 @@
* (Java/Python) `Watch` can bound its deduplication state by event time, retiring an output key once the greatest emitted timestamp has moved more than the allowed lateness past it. Java adds `Watch.growthOf(...).withTimestampCursor()`. Python adds `allowed_lateness` for the existing `timestamp_cursor` option ([#18459](https://github.com/apache/beam/issues/18459)).
* (Java) Spark Structured Streaming runner: stateful ParDo with state, timers, `@RequiresTimeSortedInput` and tagged outputs is now supported in batch mode ([#39779](https://github.com/apache/beam/issues/39779)).
* (Python) Added support for Vertex AI Model Monitoring V2 in RunInference ([#39738](https://github.com/apache/beam/issues/39738)).
+* [Flink Runner] Added opt-in static round-robin split assignment for small bounded sources via the new `sourceStaticSplitThresholdMb` pipeline option. The default of 0 keeps the existing lazy pull-based assignment ([#39873](https://github.com/apache/beam/issues/39873)).
+* Added automatic caching of bounded, single-pane side-input views for classic Java Flink DataStream execution ([#39866](https://github.com/apache/beam/issues/39866)).
+* (Python) Added `Sample.Any`, the Python equivalent of Java's `Sample.any`, which returns up to n arbitrary elements from a PCollection ([#18552](https://github.com/apache/beam/issues/18552)).
## Breaking Changes
@@ -123,8 +126,6 @@
## New Features / Improvements
-* [Flink Runner] Added opt-in static round-robin split assignment for small bounded sources via the new `sourceStaticSplitThresholdMb` pipeline option. The default of 0 keeps the existing lazy pull-based assignment ([#39873](https://github.com/apache/beam/issues/39873)).
-* Added automatic caching of bounded, single-pane side-input views for classic Java Flink DataStream execution ([#39866](https://github.com/apache/beam/issues/39866)).
* Added `GroupIntoBatches` transform and the standard
`beam:coder:sharded_key:v1` coder to the Go SDK, along with
`beam.Coder.IsDeterministic`, `beam.PCollection.WindowingStrategy`,
@@ -145,7 +146,6 @@
* (Python) Added `Watch`, a transform that polls a growing set of outputs for each input element, deduplicates outputs across poll rounds, and stops per a user-supplied termination condition
([#21521](https://github.com/apache/beam/issues/21521)).
* (Python) Added support to analyze core dumps created after python worker segmentation faults with `pystack` (or `gdb` if installed) using the `--profiler_agent=coredump` pipeline option. ([#39484](https://github.com/apache/beam/issues/39484)).
-* (Python) Added `Sample.Any`, the Python equivalent of Java's `Sample.any`, which returns up to n arbitrary elements from a PCollection ([#18552](https://github.com/apache/beam/issues/18552)).
## Breaking Changes
@@ -160,9 +160,6 @@
Use pipeline option `--updateCompatibilityVersion=2.75.0` (or any older version) to keep the old behavior ([#39344](https://github.com/apache/beam/issues/39344)).
* `DoFn.process` returning a `str`, `bytes`, or `dict` (instead of an iterable wrapping one) now raises a `TypeError` rather than silently iterating per-character/byte/key (Python) ([#18712](https://github.com/apache/beam/issues/18712)).
* (Java) Added `DRAINING` and `DRAINED` states to `PipelineResult`, including runner state mappings and Dataflow update handling ([#39020](https://github.com/apache/beam/issues/39020)).
-* (Python) Typehints of dataclass fields are honored during type inferences. To restore the behavior of fallback-to-any,
- use pipeline option `--exclude_infer_dataclass_field_type` ([#38797](https://github.com/apache/beam/issues/38797)).
- However fixing forward is recommended.
* (Java) IcebergIO and projects that use it must now be built with Java 17 or later as a result of Iceberg 1.11.0 upgrade ([#38925](https://github.com/apache/beam/issues/38925)).
## Bugfixes
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index 75afa5d7c968..339d87c76cbe 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -107,16 +107,24 @@ tasks.register("prepareExpansionServicesForYamlDocs") {
dependsOn ":sdks:java:extensions:schemaio-expansion-service:shadowJar"
dependsOn ":sdks:java:extensions:sql:expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:amazon-web-services2:expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:debezium:expansion-service:shadowJar"
dependsOn ":sdks:java:io:expansion-service:shadowJar"
dependsOn ":sdks:java:io:google-cloud-platform:expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:messaging-expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:snowflake:expansion-service:shadowJar"
doLast {
// Copy expansion service jar into cache path (.apache_beam/cache/jars/).
copy {
from project.tasks.findByPath(":sdks:java:extensions:schemaio-expansion-service:shadowJar")
from project.tasks.findByPath(":sdks:java:extensions:sql:expansion-service:shadowJar")
+ from project.tasks.findByPath(":sdks:java:io:amazon-web-services2:expansion-service:shadowJar")
+ from project.tasks.findByPath(":sdks:java:io:debezium:expansion-service:shadowJar")
from project.tasks.findByPath(":sdks:java:io:expansion-service:shadowJar")
from project.tasks.findByPath(":sdks:java:io:google-cloud-platform:expansion-service:shadowJar")
+ from project.tasks.findByPath(":sdks:java:io:messaging-expansion-service:shadowJar")
+ from project.tasks.findByPath(":sdks:java:io:snowflake:expansion-service:shadowJar")
into "${System.getProperty('user.home')}/.apache_beam/cache/jars/"
}
}
diff --git a/website/www/site/content/en/blog/beam-2.76.0.md b/website/www/site/content/en/blog/beam-2.76.0.md
index cea653eaade3..240c961cbd99 100644
--- a/website/www/site/content/en/blog/beam-2.76.0.md
+++ b/website/www/site/content/en/blog/beam-2.76.0.md
@@ -63,7 +63,6 @@ For more information on changes in 2.76.0, check out the [detailed release notes
* (Python) Added `Watch`, a transform that polls a growing set of outputs for each input element, deduplicates outputs across poll rounds, and stops per a user-supplied termination condition
([#21521](https://github.com/apache/beam/issues/21521)).
* (Python) Added support to analyze core dumps created after python worker segmentation faults with `pystack` (or `gdb` if installed) using the `--profiler_agent=coredump` pipeline option. ([#39484](https://github.com/apache/beam/issues/39484)).
-* (Python) Added `Sample.Any`, the Python equivalent of Java's `Sample.any`, which returns up to n arbitrary elements from a PCollection ([#18552](https://github.com/apache/beam/issues/18552)).
### Breaking Changes
@@ -78,9 +77,6 @@ For more information on changes in 2.76.0, check out the [detailed release notes
Use pipeline option `--updateCompatibilityVersion=2.75.0` (or any older version) to keep the old behavior ([#39344](https://github.com/apache/beam/issues/39344)).
* `DoFn.process` returning a `str`, `bytes`, or `dict` (instead of an iterable wrapping one) now raises a `TypeError` rather than silently iterating per-character/byte/key (Python) ([#18712](https://github.com/apache/beam/issues/18712)).
* (Java) Added `DRAINING` and `DRAINED` states to `PipelineResult`, including runner state mappings and Dataflow update handling ([#39020](https://github.com/apache/beam/issues/39020)).
-* (Python) Typehints of dataclass fields are honored during type inferences. To restore the behavior of fallback-to-any,
- use pipeline option `--exclude_infer_dataclass_field_type` ([#38797](https://github.com/apache/beam/issues/38797)).
- However fixing forward is recommended.
* (Java) IcebergIO and projects that use it must now be built with Java 17 or later as a result of Iceberg 1.11.0 upgrade ([#38925](https://github.com/apache/beam/issues/38925)).
### Bugfixes
diff --git a/website/www/site/content/en/documentation/io/connectors.md b/website/www/site/content/en/documentation/io/connectors.md
index 679b6bf1e0e3..f3078c6b378a 100644
--- a/website/www/site/content/en/documentation/io/connectors.md
+++ b/website/www/site/content/en/documentation/io/connectors.md
@@ -429,7 +429,10 @@ This table provides a consolidated, at-a-glance overview of the available built-
✔
native
- | Not available |
+
+ ✔
+ via X-language
+ |
Not available |
Not available |
Not available |
@@ -790,7 +793,11 @@ This table provides a consolidated, at-a-glance overview of the available built-
via X-language
Not available |
- Not available |
+
+ ✔
+
+ read
+ |
✔ |
✔ |
✘ |
@@ -1322,18 +1329,25 @@ This table provides a consolidated, at-a-glance overview of the available built-
| Delta Lake |
✔ |
- ✘ |
+ ✘ |
✔
native
|
+
+ ✔
+ via X-language
+ |
Not available |
Not available |
- Not available |
- Not available |
+
+ ✔
+
+ read
+ |
✔ |
✘ |
- ✔ |
+ ✔ |
@@ -1346,205 +1360,130 @@ This table provides a consolidated, at-a-glance overview of the available built-
Connector Name |
Source Supported |
Sink Supported |
- Java |
- Python |
- Go |
- Typescript |
- Yaml |
+ Language |
Batch Supported |
Streaming Supported |
|
- SAP Hana to Google BigQuery
+ Cloud Bigtable (HBase based)
|
✔ |
✔ |
-
- ✔
- native
- |
- Not available |
- Not available |
- Not available |
- Not available |
+ Java |
✔ |
✘ |
|
- MySQL
+ Cognite Data Fusion
|
✔ |
✔ |
- Not available |
-
- ✔
- native
- |
- Not available |
- Not available |
-
- ✔
-
- read / write
- |
+ Java |
+ ✔ |
✔ |
- ✘ |
|
- TrepWsIO
+ KineticaDB
|
✔ |
- ✘ |
-
- ✔
- native
- |
- Not available |
- Not available |
- Not available |
- Not available |
✔ |
+ Java |
✔ |
+ ✘ |
|
- KineticaDB
+ SAP Hana to Google BigQuery
|
✔ |
✔ |
-
- ✔
- native
- |
- Not available |
- Not available |
- Not available |
- Not available |
+ Java |
✔ |
✘ |
|
- Cognite Data Fusion
+ TrepWsIO
|
✔ |
- ✔ |
-
- ✔
- native
- |
- Not available |
- Not available |
- Not available |
- Not available |
+ ✘ |
+ Java |
✔ |
✔ |
|
- Pyodbc
+ Beam PyIO (Collection of Python IO connectors)
|
✔ |
✔ |
- Not available |
-
- ✔
- native
- |
- Not available |
- Not available |
- Not available |
+ Python |
+ ✔ |
✔ |
- ✘ |
|
- Go Connect
+ Cloud SQL
|
✔ |
- ✔ |
✘ |
- ✘ |
-
- ✔
- native
- |
- Not available |
- Not available |
- ✔ |
+ Python |
✔ |
+ ✘ |
|
- Tinybird
+ MySQL
|
- ✘ |
✔ |
- Not available |
-
- ✔
- native
- |
- Not available |
- Not available |
- Not available |
✔ |
+ Python |
✔ |
+ ✘ |
|
- Cloud SQL
+ Pyodbc
|
✔ |
- ✘ |
- Not available |
-
- ✔
- native
- |
- Not available |
- Not available |
- Not available |
+ ✔ |
+ Python |
✔ |
✘ |
|
- Cloud Bigtable (HBase based)
+ Tinybird
|
+ ✘ |
✔ |
+ Python |
✔ |
-
- ✔
- native
- |
- Not available |
- Not available |
- Not available |
- Not available |
✔ |
- ✘ |
|
- Beam PyIO (Collection of Python IO connectors)
+ Go Connect
|
✔ |
✔ |
- Not available |
-
- ✔
- native
+ | Go |
+ ✔ |
+ ✔ |
+
+
+ |
+ MySQL, PostgreSQL, Oracle, SQL Server
|
- Not available |
- Not available |
- Not available |
✔ |
✔ |
+ Yaml |
+ ✔ |
+ ✘ |
diff --git a/website/www/site/data/capability_matrix.yaml b/website/www/site/data/capability_matrix.yaml
index e4db68c0cbeb..3f3bf39594c5 100644
--- a/website/www/site/data/capability_matrix.yaml
+++ b/website/www/site/data/capability_matrix.yaml
@@ -1736,7 +1736,6 @@ capability-matrix:
l1: "Unverified"
l2:
l3:
-
- class: kafka-streams
l1: "No"
l2: not implemented