feat(gcp_cloud_storage sink): support Parquet batch encoding

[dshmatov]

Summary

Adds Apache Parquet columnar batch encoding to the gcp_cloud_storage sink, matching the capability the aws_s3 sink already has. Enable it with batch_encoding.codec = "parquet".

When batch_encoding is set, events are encoded together as a batch into a single Parquet file per batch (instead of the standard per-event, framing-based encoding). Parquet handles its own compression internally, so:

• the top-level compression setting is bypassed (a warning is logged if it was set),
• the object Content-Type defaults to application/vnd.apache.parquet,
• the filename extension defaults to parquet.

This reuses the shared codec/batch-encoder infrastructure introduced for aws_s3 (the BatchEncoder / BatchSerializerConfig / EncoderKind machinery in lib/codecs and src/sinks/util). The change is purely the sink-side wiring plus a GcsBatchEncoding config enum — no new encoding logic.

Vector configuration

sources:
  demo:
    type: demo_logs
    format: json

sinks:
  gcs_out:
    type: gcp_cloud_storage
    inputs: [demo]
    bucket: my-bucket
    encoding:
      codec: text
    batch_encoding:
      codec: parquet
      schema_mode: auto_infer
      compression:
        algorithm: snappy

Schema-file mode is also supported:

    batch_encoding:
      codec: parquet
      schema_mode: strict
      schema_file: /etc/vector/schema.parquet
      compression:
        algorithm: zstd
        level: 10

How did you test this PR?

• cargo test --features "sinks-gcp,codecs-parquet" --lib gcp::cloud_storage — 9 new Parquet tests + the 15 existing GCS sink tests pass.
  • Config-level: TOML/YAML shape, schema_mode parsing/defaults, content-type auto-detection + user override, .parquet extension default + override, top-level-compression bypass, rejection of non-parquet codecs.
  • End-to-end encoding: parquet_encodes_valid_file runs a real batch through the sink's request builder and asserts the output is a valid Parquet file (PAR1 magic bytes, correct row count, inferred message/host columns) using the parquet reader.
• Verified the crate compiles with codecs-parquet both enabled and disabled (the feature-gated paths).
• make check-clippy, make fmt, and make check-generated-docs (regenerated gcp_cloud_storage.cue) pass.

Note: this repo currently has no GCS-cloud-storage integration-test harness (no fake-gcs-server service under scripts/integration/), so Parquet output is validated at the unit level through the real encode path rather than against a live backend. Happy to add an integration harness if maintainers would prefer one.

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Builds on the shared Parquet codec infrastructure from fix(releasing): enable codecs-parquet in all release feature sets #25321.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[dshmatov]

I have read the CLA Document and I hereby sign the CLA

[drichards-87]

Left a couple of suggestions from Docs and approved the PR.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a4a585b76

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[akukhar]

Hi, I wanted to follow up on this PR. It would help fix the limitation I'm facing, and I'd really like to use it upstream.