out_file: support timestamp placeholder - #12245
Conversation
📝 WalkthroughWalkthroughThe file output plugin now supports UTC ChangesTimestamp-based file destinations
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant EventDecoder
participant OutFilePlugin
participant OutputFile
EventDecoder->>OutFilePlugin: decoded record with timestamp
OutFilePlugin->>OutFilePlugin: format UTC strftime placeholders
OutFilePlugin->>OutputFile: write to resolved path and filename
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/out_file/file.c`:
- Around line 421-428: Update the timestamp expansion logic in
plugins/out_file/file.c at lines 421-428 and 431-463 so strftime processes only
configured path/file placeholders, preserving record-accessor values literally,
including values containing %Y. Add or update the mixed accessor/timestamp
coverage in tests/runtime/out_file.c lines 618-662 to verify only configured
placeholders expand and accessor output remains unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7277a0d0-45a7-4062-80c7-0ab20c32757e
📒 Files selected for processing (5)
plugins/out_file/file.ctests/runtime/CMakeLists.txttests/runtime/flb_tests_runtime.h.intests/runtime/out_file.ctests/runtime/out_file_rotation.c
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6174e0eee9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (ctx->out_file != NULL && strchr(ctx->out_file, '%') != NULL) { | ||
| ctx->time_file = FLB_TRUE; | ||
| ctx->dynamic_destination = FLB_TRUE; |
There was a problem hiding this comment.
When an existing file value contains a literal % (which is valid in filenames), this condition now classifies it as a dynamic destination, and the check below rejects the configuration unless fallback_file is also supplied. Configurations such as file metrics%used.log therefore stop starting after this change; when a fallback is supplied, strftime can additionally reinterpret %u and change the destination. Avoid treating every percent sign as an implicit opt-in, or provide a backward-compatible way to distinguish literal percentages from timestamp placeholders.
Useful? React with 👍 / 👎.
| if (ctx->time_path == FLB_TRUE) { | ||
| timestamp_path = format_event_timestamp(path, timestamp); |
There was a problem hiding this comment.
When a timestamp format is combined with a record accessor, such as /logs/%Y/$TAG, this call runs strftime after accessor translation. If the tag or referenced field itself contains a sequence such as %m, that user-provided text is interpreted as another timestamp directive, mutating the destination and potentially merging distinct tag values into the same file. Expand timestamp directives in the configured template before inserting accessor values, or escape percent signs introduced by accessor translation.
Useful? React with 👍 / 👎.
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io> wwip
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
6174e0e to
87d2f4b
Compare
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
87d2f4b to
f21930c
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tests/runtime/out_file.c (1)
643-645: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy liftMake the UTC assertions timezone-sensitive.
A local-time implementation can pass these tests when the runner timezone is UTC. Run the scenarios with a controlled non-UTC local timezone and use a timestamp that crosses a local-date boundary. Restore the process timezone after each test.
tests/runtime/out_file.c#L643-L645: verify the date-expanded filename uses the UTC event date.tests/runtime/out_file.c#L673-L719: make the standalone timestamp test distinguish UTC from local time.tests/runtime/out_file.c#L721-L778: retain the percent-value coverage while verifying the UTC year and date components.As per coding guidelines, runtime tests must cover plugin-level semantics.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/runtime/out_file.c` around lines 643 - 645, Make the UTC assertions in tests/runtime/out_file.c:643-645, tests/runtime/out_file.c:673-719, and tests/runtime/out_file.c:721-778 timezone-sensitive by setting a controlled non-UTC process timezone for each scenario, using timestamps that cross a local-date boundary, and restoring the original timezone afterward; verify UTC date expansion, distinguish standalone UTC timestamps from local time, and preserve percent-value coverage while asserting UTC year and date components.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/runtime/out_file.c`:
- Around line 569-603: Update flb_test_file_literal_percent to construct its
output and fallback filenames under TEST_LOGPATH instead of the working
directory, including configuring both paths accordingly. Replace direct remove
calls and existence cleanup with flb_test_rmdir() for the test-specific
directory, while preserving the assertions that the output is created and the
fallback is absent.
---
Nitpick comments:
In `@tests/runtime/out_file.c`:
- Around line 643-645: Make the UTC assertions in
tests/runtime/out_file.c:643-645, tests/runtime/out_file.c:673-719, and
tests/runtime/out_file.c:721-778 timezone-sensitive by setting a controlled
non-UTC process timezone for each scenario, using timestamps that cross a
local-date boundary, and restoring the original timezone afterward; verify UTC
date expansion, distinguish standalone UTC timestamps from local time, and
preserve percent-value coverage while asserting UTC year and date components.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: fe95adb7-b3b6-4fea-bedc-b01656d0ce3f
📒 Files selected for processing (2)
plugins/out_file/file.ctests/runtime/out_file.c
🚧 Files skipped from review as they are similar to previous changes (1)
- plugins/out_file/file.c
| remove(file); | ||
| remove(fallback); | ||
|
|
||
| ctx = flb_create(); | ||
| flb_service_set(ctx, "Flush", "1", "Grace", "1", "Log_Level", "error", NULL); | ||
|
|
||
| in_ffd = flb_input(ctx, (char *) "lib", NULL); | ||
| TEST_CHECK(in_ffd >= 0); | ||
| flb_input_set(ctx, in_ffd, "tag", "test", NULL); | ||
|
|
||
| out_ffd = flb_output(ctx, (char *) "file", NULL); | ||
| TEST_CHECK(out_ffd >= 0); | ||
| flb_output_set(ctx, out_ffd, "match", "test", NULL); | ||
| flb_output_set(ctx, out_ffd, "file", file, NULL); | ||
| flb_output_set(ctx, out_ffd, "fallback_file", fallback, NULL); | ||
|
|
||
| ret = flb_start(ctx); | ||
| TEST_CHECK(ret == 0); | ||
|
|
||
| bytes = flb_lib_push(ctx, in_ffd, p, strlen(p)); | ||
| TEST_CHECK(bytes == strlen(p)); | ||
| ret = wait_for_file(file, 1, TEST_TIMEOUT); | ||
| TEST_CHECK(ret == 0); | ||
|
|
||
| flb_stop(ctx); | ||
| flb_destroy(ctx); | ||
|
|
||
| fp = fopen(file, "r"); | ||
| TEST_CHECK(fp != NULL); | ||
| if (fp != NULL) { | ||
| fclose(fp); | ||
| } | ||
| TEST_CHECK(access(fallback, F_OK) != 0); | ||
| remove(file); | ||
| remove(fallback); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Use a test-specific directory for flb_test_file_literal_percent.
Lines 569-570 delete metrics%used.log and fallback.log from the process working directory. The output configuration has no path. Running this test from a directory that contains either file can delete non-test data.
Use TEST_LOGPATH for the output and fallback paths. Clean that directory with flb_test_rmdir().
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/runtime/out_file.c` around lines 569 - 603, Update
flb_test_file_literal_percent to construct its output and fallback filenames
under TEST_LOGPATH instead of the working directory, including configuring both
paths accordingly. Replace direct remove calls and existence cleanup with
flb_test_rmdir() for the test-specific directory, while preserving the
assertions that the output is created and the fallback is absent.
1 participant
This should be a parity feature of Fluentd's placeholder support on Fluent Bit's out_file plugin.
Implemented and verified on Windows.
strftimeplaceholders in file.c.%Y%m%dcoverage, includingpath/$TAG/$proxy_name/file.%Y%m%d, in out_file.c.The reported failures occurred because duplicate instances of the same test binary ran concurrently and shared their fixed fixture directories. Isolated sequential execution passes consistently:
Targeted MSVC rebuild passed,
git diff --checkpassed, and the repository commit-prefix checker passed. The branch is clean at6174e0eee.No focused
tests/integrationscenario exists forout_file. Platform memory checking was not run because this is Windows; Valgrind and macOS Leaks are unavailable.Closes #12233.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
strftimeplaceholders for dynamic output file paths and filenames when enabled.Bug Fixes