Bug Report
Describe the bug
When DB.compare_filename is enabled and the tail input monitors symbolic links (e.g., glog-style log files), Fluent Bit fails to match the file against its database entry on restart. The file is treated as a new file, and with Read_from_Head True, the entire file is re-read from the beginning, causing duplicate log ingestion.
The root cause is in flb_tail_target_file_name_cmp() (plugins/in_tail/tail_file.h). When inserting into the database, the symlink path is stored as name (file->name). However, during the filename comparison, the DB-stored name is compared against file->real_name (the resolved target path obtained via readlink(/proc/pid/fd/N)) rather than file->name. For symlinks, these basenames differ, so the comparison always fails.
For example:
- DB stores:
name = "trading_system_main.INFO" (symlink), inode = 12345
- On restart, comparison:
basename("trading_system_main.INFO") vs basename("trading_system_main.INFO.20250303-120000.12345") → mismatch
- Result: existing DB entry deleted as stale, new entry inserted with offset 0 → full re-read
To Reproduce
- Steps to reproduce the problem:
- Create a glog-style symbolic link:
trading_system_main.INFO -> trading_system_main.INFO.20250303-120000.12345
- Configure tail input with
DB.compare_filename true and Read_from_Head True to monitor the symlink via a glob pattern (e.g., *.INFO)
- Let Fluent Bit run and process some logs — the DB records the inode and symlink path
- Restart Fluent Bit
- Observe that the file is re-read from the beginning (duplicate logs emitted)
Expected behavior
On restart, Fluent Bit should recognize the file via its DB entry and resume reading from the previously recorded offset, without re-reading already-processed data.
Your Environment
- Version used: 3.2.7 (also confirmed on latest master — the code in
tail_file.h is identical)
- Configuration:
ini
[INPUT]
Name tail
Path /path/to/logs/.INFO
Db /var/log/flb-position.db
DB.compare_filename true
Read_from_Head True
Your Environment
- Version used: 3.2.7 (also confirmed on latest master — the code in
tail_file.h is identical)
- Configuration:
Bug Report
Describe the bug
When
DB.compare_filenameis enabled and the tail input monitors symbolic links (e.g., glog-style log files), Fluent Bit fails to match the file against its database entry on restart. The file is treated as a new file, and withRead_from_Head True, the entire file is re-read from the beginning, causing duplicate log ingestion.The root cause is in
flb_tail_target_file_name_cmp()(plugins/in_tail/tail_file.h). When inserting into the database, the symlink path is stored asname(file->name). However, during the filename comparison, the DB-stored name is compared againstfile->real_name(the resolved target path obtained viareadlink(/proc/pid/fd/N)) rather thanfile->name. For symlinks, these basenames differ, so the comparison always fails.For example:
name = "trading_system_main.INFO"(symlink),inode = 12345basename("trading_system_main.INFO")vsbasename("trading_system_main.INFO.20250303-120000.12345")→ mismatchTo Reproduce
trading_system_main.INFO -> trading_system_main.INFO.20250303-120000.12345DB.compare_filename trueandRead_from_Head Trueto monitor the symlink via a glob pattern (e.g.,*.INFO)Expected behavior
On restart, Fluent Bit should recognize the file via its DB entry and resume reading from the previously recorded offset, without re-reading already-processed data.
Your Environment
tail_file.his identical)Your Environment
tail_file.his identical)