SnowflakeIO yaml by Amar3tto · Pull Request #39742 · apache/beam

Amar3tto · 2026-08-13T11:25:33Z

Fixes #28691

Adds Beam YAML support for Snowflake IO.

Adds ReadFromSnowflake for batch reads from a table or query.
Adds WriteToSnowflake for batch writes and streaming writes via Snowpipe.
Supports password, OAuth, and private key authentication.
Adds Beam Row / Snowflake schema conversion.
Registers both transforms in standard_io.yaml.
Adds unit tests for configuration, authentication, schema mapping, and batch/streaming behavior.

https://github.com/apache/beam/actions/runs/31692701363
https://github.com/apache/beam/actions/runs/31691059139

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Abacn · 2026-08-13T16:49:55Z

+import org.apache.beam.sdk.values.Row;
+
+/** A {@link SchemaTransformProvider} for reading rows from Snowflake. */
+@SuppressWarnings({


Prefer not introducing SuppressWarnings for new code

Abacn · 2026-08-13T16:51:29Z

+})
+public class SnowflakeSchemaTransformUtils {
+
+  public static SnowflakeIO.DataSourceConfiguration createDataSourceConfiguration(


Why not just make SnowflakeReadSchemaTransformProvider.Configuration as createDataSourceConfiguration's parameter?

It is shared by both the read and write providers, which have different configuration classes. Passing the individual fields keeps this utility independent of either provider-specific configuration.

Abacn · 2026-08-13T16:53:40Z

+
+    String debugMode = getDebugMode();
+    if (debugMode != null) {
+      parseStreamingLogLevel(debugMode);


return value is unused. I understand it's for validation. Consider add a comment noting it.

Please check, I don't see a code change?

Added one comment above (line 142) for the whole part:

Parse configured enum values to validate that they are supported.

github-actions · 2026-08-13T17:22:46Z

Assigning reviewers:

R: @claudevdm for label python.
R: @kennknowles for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

derrickaw · 2026-08-18T12:03:48Z

Hi @Amar3tto, can you add a yaml pipeline test under yaml/tests or yaml/extended_tests with any improvements necessary in integration_tests.py? Thanks.

Amar3tto · 2026-08-20T13:46:28Z

Hi @Amar3tto, can you add a yaml pipeline test under yaml/tests or yaml/extended_tests with any improvements necessary in integration_tests.py? Thanks.

Added a Snowflake YAML pipeline test under yaml/extended_tests/databases together with the required fixture in integration_tests.py.
Currently the existing Snowflake integration test is not run by CI and the YAML PostCommit does not provide Snowflake credentials/configuration either, so the test will be skipped until those are configured.

Abacn · 2026-08-20T14:16:35Z

  dependsOn ":sdks:java:io:expansion-service:build"
  dependsOn ":sdks:java:io:google-cloud-platform:expansion-service:build"
  dependsOn ":sdks:java:io:debezium:expansion-service:shadowJar"
+  dependsOn ":sdks:java:io:snowflake:expansion-service:shadowJar"


Consider move this to postCommitYamlIntegrationTests. The test only need to be exercised on postsubmit time. The PreCommit becomes heavier as we are adding more expansion services (and test container). Same apply to kenesis / JmsIO / debezium

Abacn · 2026-08-25T14:30:20Z

   */
  @FunctionalInterface
  public interface UserDataMapper<T> extends Serializable {
+    @org.checkerframework.checker.nullness.qual.Nullable


nit: Import the name

I kept javax.annotation.Nullable as the existing import to avoid conflicting Nullable imports. The Checker Framework annotation is only needed for this specific type-use case, so I used its fully qualified name there

We should replace all with org.checkerframework.checker.nullness.qual.Nullable, which is Beam repo's convention. But code completion tool keeps adding javax and sometimes slipped from review

Abacn · 2026-08-25T14:32:16Z

+
+    String debugMode = getDebugMode();
+    if (debugMode != null) {
+      parseStreamingLogLevel(debugMode);


Please check, I don't see a code change?

Abacn · 2026-08-25T14:33:43Z

  implementation project(path: ":sdks:java:extensions:google-cloud-platform-core")
  permitUnusedDeclared project(path: ":sdks:java:extensions:google-cloud-platform-core")
  implementation library.java.slf4j_api
+  implementation library.java.everit_json_schema


Why do we need it now? Is it now a runtime dependency, or needed by the expansion service now?

It is a runtime dependency of the new read SchemaTransform. JsonUtils.beamSchemaFromJsonSchema(...) uses Everit JSON Schema to parse the configured output schema, and the expansion service picks it up transitively from the Snowflake IO module.

It's transient dependency shouldn't need to be added here. In fact it's a dependency of Java core:

beam/sdks/java/core/build.gradle

Line 115 in 13875fc

implementation library.java.everit_json_schema

In general new codes should not need permitUnusedDeclared ...

I tried removing the dependency, but :sdks:java:io:snowflake:compileJava then fails with:

class file for org.everit.json.schema.ObjectSchema not found

I see, thanks, then consider declare it as "provided", in align with

beam/sdks/java/io/kafka/build.gradle

Line 95 in 13875fc

provided library.java.everit_json_schema

beam/sdks/java/io/mongodb/build.gradle

Line 35 in 13875fc

provided library.java.everit_json_schema

Changed to provided, thanks

Abacn

Thanks!

Amar3tto requested a review from Abacn August 13, 2026 11:25

github-actions Bot added python java io yaml Next Action: Reviewers labels Aug 13, 2026

Abacn reviewed Aug 13, 2026

View reviewed changes

Amar3tto requested a review from Abacn August 17, 2026 20:50

Abacn reviewed Aug 20, 2026

View reviewed changes

Amar3tto added 8 commits August 25, 2026 13:46

Add Snowflake YAML write transform

69f7fd4

Add Snowflake Read and Streaming Write

85bad22

Fix bytes

90c93d7

Fix imports

d131e0e

Refactoring

f4e62fd

Add extended test

15efa5d

Add license

b275c84

Move depends

27c4e63

Amar3tto force-pushed the snowflakeio-yaml branch from 25a4801 to 27c4e63 Compare August 25, 2026 09:49

Fix jms yaml

a93fdcd

Amar3tto requested a review from Abacn August 25, 2026 12:54

Abacn reviewed Aug 25, 2026

View reviewed changes

Amar3tto added 2 commits August 25, 2026 23:36

Fix import Nullable

da201e6

Change to provided

497870a

Amar3tto requested a review from Abacn August 26, 2026 14:52

Abacn approved these changes Aug 26, 2026

View reviewed changes

Amar3tto merged commit 8cfbd8e into master Aug 26, 2026
116 of 117 checks passed

Uh oh!

Conversation

Amar3tto commented Aug 13, 2026 • edited by derrickaw Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

derrickaw commented Aug 18, 2026

Uh oh!

Amar3tto commented Aug 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Amar3tto Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Abacn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Amar3tto commented Aug 13, 2026 •

edited by derrickaw

Loading

Amar3tto Aug 25, 2026 •

edited

Loading