Skip to content

feat(aws_s3 sink): add key option for full object key control#25575

Open
aravind-n wants to merge 10 commits into
vectordotdev:masterfrom
aravind-n:feat/aws-s3-sink-templated-key
Open

feat(aws_s3 sink): add key option for full object key control#25575
aravind-n wants to merge 10 commits into
vectordotdev:masterfrom
aravind-n:feat/aws-s3-sink-templated-key

Conversation

@aravind-n

@aravind-n aravind-n commented Jun 4, 2026

Copy link
Copy Markdown

Summary

Adds an optional templated key field to the aws_s3 sink. When set, the rendered template is used as the full S3 object key — key_prefix, filename_time_format, filename_append_uuid, and filename_extension are ignored (with a warn! if they are non-default). Closes the parity gap with Fluentd's s3_object_key_format.

Vector configuration

sinks:
  s3:
    type: aws_s3
    inputs: [in]
    bucket: my-bucket
    key: "logs/{{ host }}/%F-{{ message_id }}.json"
    encoding:
      codec: json

How did you test this PR?

  • New unit tests in sinks::aws_s3::{config,sink} and sinks::s3_common::partitioner (6 total, all passing).
  • cargo fmt --check, cargo clippy --features sinks-aws_s3 --lib --tests, make check-generated-docs, ./scripts/check_changelog_fragments.sh — all clean.

Change Type

  • Bug fix
  • New feature
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

@aravind-n aravind-n requested review from a team as code owners June 4, 2026 22:09
@github-actions github-actions Bot added domain: sinks Anything related to the Vector's sinks domain: external docs Anything related to Vector's external, public documentation docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. labels Jun 4, 2026
@github-actions

github-actions Bot commented Jun 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@aravind-n

aravind-n commented Jun 4, 2026

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jun 4, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e043c9276b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sinks/aws_s3/config.rs
@aravind-n

aravind-n commented Jun 4, 2026

Copy link
Copy Markdown
Author

The failing Check Spelling job is flagging deser at website/cue/reference/components/sources/internal_metrics.cue:1188 ("deser_failed": — a buffer error code). That line was introduced on master by #25559 (commit c0806d3) and is not part of this PR's diff (git diff master..HEAD | grep deser returns 0). Happy to add deser to .github/actions/spelling/allow.txt here if a maintainer would prefer that path; otherwise this seems like a master-side fix.

@aravind-n aravind-n force-pushed the feat/aws-s3-sink-templated-key branch from e043c92 to b01a4f0 Compare June 11, 2026 00:29
@aravind-n

aravind-n commented Jun 11, 2026

Copy link
Copy Markdown
Author

@rtrieu It says I need a code owner to review this to merge. Would you be able to do this please? If not can you point me in the right direction please?

aravind-n added 10 commits June 11, 2026 13:35
@aravind-n
feat(aws_s3 sink): add optional key field to S3SinkConfig
8dff27e 
Introduces an optional templated `key` field. When set, it will be used
as the full S3 object key, bypassing the implicit prefix/timestamp/UUID/
extension concatenation. Wiring through the partitioner and request
builder happens in following commits.
@aravind-n
feat(aws_s3 sink): plumb full-key template through S3KeyPartitioner
0956654 
`S3PartitionKey` now carries an `is_full_key` flag and `S3KeyPartitioner`
accepts an optional key template via `with_key_template`. When the key
template is set, the partitioner renders it and marks the resulting
partition key as a full key so downstream consumers know to skip the
implicit timestamp/UUID/extension suffix.
@aravind-n
feat(aws_s3 sink): skip filename build when partition key is a full key
d5bb699 
`S3RequestOptions::build_request` now uses the rendered partition key
verbatim whenever `is_full_key` is set, bypassing the timestamp / UUID /
extension suffix. Existing behavior is preserved when `is_full_key` is
`false`.
@aravind-n
feat(aws_s3 sink): build key template in build_processor and warn on …
a492bb2 
…overrides

When `key` is set, the sink builds a Template from it (with timezone
offset), passes it to the partitioner via `with_key_template`, and logs
a single warning if `key_prefix`, `filename_time_format`,
`filename_append_uuid`, or `filename_extension` are non-default — those
fields are silently ignored in this mode.
@aravind-n
test(aws_s3 sink): add unit tests for templated full key
2e0a731 
Extracts the suffix-building logic into `S3RequestOptions::finalize_s3_key`
and adds unit tests covering:
- full-key partition keys are used verbatim
- non-full-key partition keys still get timestamp/extension appended
- TOML config parses the new `key` field, defaulting to `None`
@aravind-n
docs(aws_s3 sink): document key option for full object key control
3e2ee9d 
Regenerates the component cue from the schema (adds the `key` field plus
the "Ignored when `key` is set" notes on the existing filename fields)
and adds a hand-written "Specifying the full object key" section that
calls out the Fluentd-migration use case and the uniqueness caveat.
@aravind-n
chore(aws_s3 sink): add changelog fragment for templated key feature
287f554
@aravind-n
chore(aws_s3 sink): add authors line to changelog fragment
c96f1db
@aravind-n
style(aws_s3 sink): apply rustfmt to test setup helper
49ce65a
@aravind-n
docs(aws_s3 sink): use a unique-per-event example for key
dfa96e9 
`logs/{{ host }}/%F.log` rendered to day granularity, so two events
from the same host on the same day would overwrite each other in S3
— at odds with the uniqueness warning in the same docstring. Swapped
for `logs/{{ host }}/%F-{{ message_id }}.json`, which mirrors the
example in the hand-written cue and produces a unique key per event.
@aravind-n aravind-n force-pushed the feat/aws-s3-sink-templated-key branch from b01a4f0 to dfa96e9 Compare June 11, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@rtrieu rtrieu rtrieu approved these changes

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Allow specifying the object key in the aws_s3 sink

2 participants

@aravind-n @rtrieu