Skip to content

out_azure_logs_ingestion: increase timestamp precision to 100ns#11534

Open
hashtagchris wants to merge 1 commit intofluent:masterfrom
hashtagchris:azure-log-ingestion-10ns-precision
Open

out_azure_logs_ingestion: increase timestamp precision to 100ns#11534
hashtagchris wants to merge 1 commit intofluent:masterfrom
hashtagchris:azure-log-ingestion-10ns-precision

Conversation

@hashtagchris
Copy link

@hashtagchris hashtagchris commented Mar 11, 2026

Summary

Increase the ISO 8601 timestamp precision in the azure_log_ingestion output plugin from milliseconds (3 decimal digits) to 100 nanoseconds (7 decimal digits).

Before: 2026-03-11T00:10:36.123Z
After: 2026-03-11T00:10:36.1234567Z

Details

The flb_time struct already stores nanosecond-precision timestamps via tv_nsec. This change preserves that precision in the formatted output instead of truncating to milliseconds.

Changes

  • Format specifier: %03%07 (7 zero-padded digits)
  • Divisor: / 1000000/ 100 (100-nanosecond granularity)

Also updates the time_generated config description to accurately reflect that it controls the timestamp format (ISO 8601 vs floating-point), not whether a timestamp is appended. Doc PR: fluent/fluent-bit-docs#2411

The time_formatted buffer (32 bytes) has sufficient capacity for the longer output (29 bytes max including null terminator).

Reference

"Time values are measured in 100-nanosecond units called ticks..." - https://learn.microsoft.com/en-us/kusto/query/scalar-data-types/datetime

Summary by CodeRabbit

  • Bug Fixes

    • Increased timestamp precision in Azure Logs ingestion to seven fractional digits, improving time resolution and accuracy for ingested events.
  • Configuration / Behavior

    • Clarified time_generated behavior: when enabled, the configured key holds an ISO 8601 timestamp; when disabled, the key contains a floating-point number representing seconds since the Unix epoch.

@coderabbitai
Copy link

coderabbitai bot commented Mar 11, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d7cc1f2b-7c6e-477d-aaaf-e6fd5022f7f0

📥 Commits

Reviewing files that changed from the base of the PR and between 43c760b and 27f03bf.

📒 Files selected for processing (1)
  • plugins/out_azure_logs_ingestion/azure_logs_ingestion.c
🚧 Files skipped from review as they are similar to previous changes (1)
  • plugins/out_azure_logs_ingestion/azure_logs_ingestion.c

📝 Walkthrough

Walkthrough

ISO 8601 fractional-second precision in az_li_format increased to seven digits (from nanoseconds / 100). The time_generated configuration description was clarified: enabled yields an ISO 8601 string under time_key; disabled yields a floating-point seconds-since-epoch value.

Changes

Cohort / File(s) Summary
Azure Logs Ingestion Source
plugins/out_azure_logs_ingestion/azure_logs_ingestion.c
In az_li_format, changed fractional-seconds formatting to seven digits (.%07), computing the fraction from tm.tm.tv_nsec / 100. Updated time_generated configuration description to state enabled produces an ISO 8601 string at time_key; disabled produces a floating-point number of seconds.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

docs-required

Suggested reviewers

  • edsiper

Poem

🐇 I nibble nsecs beneath a digital sky,
Seven tiny digits now sparkle by,
Timestamps hum in ISO's tune,
Floats or strings beneath the moon,
I hop, I patch, I watch time fly.

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: increasing timestamp precision from milliseconds to 100-nanosecond units in the azure_logs_ingestion plugin.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@hashtagchris hashtagchris force-pushed the azure-log-ingestion-10ns-precision branch 2 times, most recently from 989e857 to a01d4e9 Compare March 11, 2026 01:23
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@plugins/out_azure_logs_ingestion/azure_logs_ingestion.c`:
- Around line 422-426: The config help text for time_generated is inconsistent
with the actual emitted value: verify what unit flb_time_to_double(&tm) returns,
and then either (A) restore the doc string to say "seconds" (reword the disabled
branch text in the help string for struct flb_az_li/time_generated to "a
floating-point number of seconds") if flb_time_to_double returns seconds, or (B)
change the serialization path that uses flb_time_to_double(&tm) to emit
milliseconds (multiply the returned value by 1000 or use an appropriate helper)
and keep the help text saying "milliseconds"; update references to
time_key/time_generated accordingly so the doc string and emitted value match.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9fdf6c91-66e9-4a90-9c1e-d2f8cfb4f326

📥 Commits

Reviewing files that changed from the base of the PR and between 989e857 and a01d4e9.

📒 Files selected for processing (1)
  • plugins/out_azure_logs_ingestion/azure_logs_ingestion.c

@hashtagchris hashtagchris force-pushed the azure-log-ingestion-10ns-precision branch from a01d4e9 to 43c760b Compare March 11, 2026 01:48
@hashtagchris hashtagchris changed the title out_azure_logs_ingestion: increase timestamp precision to 10ns out_azure_logs_ingestion: increase timestamp precision to 100ns Mar 11, 2026
@hashtagchris hashtagchris force-pushed the azure-log-ingestion-10ns-precision branch from 43c760b to 7c81398 Compare March 11, 2026 02:01
@hashtagchris
Copy link
Author

hashtagchris commented Mar 11, 2026

End-to-end Testing

Sent test logs to an Azure Monitor Log Analytics workspace.

Two log messages. Stock Fluent Bit has 3 digits of precision. My build has 7 digits of precision

Config

service:
  flush: 5
  log_level: info

pipeline:
  inputs:
    - name: tail
      # test Fluent Bit's extraction and standardization of the timestamp from the log record
      path: ./time-key-test.log
      parser: logfmt-time-key
      read_from_head: true
      exit_on_eof: true

  outputs:
    - name: azure_logs_ingestion
      match: '*'
      table_name: MyFluentBitLogs
      dce_url: ${DCE_URL}
      dcr_id: ${DCR_ID}
      tenant_id: ${TENANT_ID}
      client_id: ${CLIENT_ID}
      client_secret: ${CLIENT_SECRET}
      time_key: TimeGenerated
      time_generated: true

parsers:
  - name: logfmt-time-key
    format: logfmt
    time_key: time
    time_format: '%Y-%m-%dT%H:%M:%S.%LZ'
    time_keep: off

time-key-test.log.tmpl

time=2026-03-11T02:02:00.123456712Z EventTime=$EVENT_TIME SeverityLevel=info SyslogMessage="$SYSLOG_MESSAGE"

Increase the ISO 8601 timestamp precision from milliseconds
(3 decimal digits) to 100 nanoseconds (7 decimal digits).

The flb_time struct already stores nanosecond-precision
timestamps via tv_nsec. This change preserves that precision
in the formatted output instead of truncating to milliseconds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Chris Sidi <hashtagchris@github.com>
@hashtagchris hashtagchris force-pushed the azure-log-ingestion-10ns-precision branch from 7c81398 to 27f03bf Compare March 11, 2026 02:53
@hashtagchris
Copy link
Author

@coderabbitai resume

@coderabbitai
Copy link

coderabbitai bot commented Mar 11, 2026

✅ Actions performed

Reviews resumed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant