feat(sink): add configurable payload limit to datadog_logs sink

[dd-sebastien-lb]

Summary

Allow max_payload_bytes to be configurable for datadog_logs sink.

References

Vector configuration

sources:
  http_in:
    type: http_server
    address: 0.0.0.0:8081
    encoding: json

sinks:
  dd_logs:
    type: datadog_logs
    inputs: [http_in]
    endpoint: http://127.0.0.1:8080
    default_api_key: test_key
    compression: none
    max_payload_bytes: 10000000  # 10 MB — above the default 5 MB Datadog limit
    batch:
      max_bytes: 9000000         # batch goal: 9 MB (headroom = 1 MB > BATCH_HEADROOM_BYTES)
      timeout_secs: 5

How did you test this PR?

• Units tests
• Above configuration successfully sending events to compatible sink

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

Contributor Guidelines

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Before pushing, follow our pre-push guidance.
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 35e041d056

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[gwenaskell]

just a suggestion on the docs

[gwenaskell]

IMO these details could be skipped or simplified. I think that what is worth highlighting is that the sink will split the batch if it exceeds the max_payload_size, and if a single event exceeds it it will be tossed.

[dd-sebastien-lb]

I find it interesting to provide as theses details as it can help making tradeoff if for some reason you want to lower the max_payload_bytes since the safety headroom is not configurable

[gwenaskell]

technically, it does not need to be at least 5MB, but there's not really a point in setting a lower value. I think we can keep it this way

[thomasqueirozb]

Agreed, I think that the wording here can be changed, Something like:

"it is recommended to be set to at least 5,000,000 (5 MB, the standard Datadog API limit)"

or similar

[dd-sebastien-lb]

I actually made a validation to enforce a minimum limit of 5Mb since all events below that limit are going to be accepted. I think it can cause confusion if we allow users to configure a lower limit and they start losing events.
Setting a lower value would be similar to have a filtering on large event. I don't know if this is something we want to offer in the destination?

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bdf1add038

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".