Fluent Bit File rotation not detected

What is Fluent Bit File rotation not detected

Understanding Fluent Bit

Fluent Bit is a lightweight and high-performance log processor and forwarder that allows you to collect data and logs from different sources, unify them, and send them to multiple destinations. It is designed to handle high throughput and is often used in environments where resource efficiency is critical, such as in containerized applications.

Symptom: File Rotation Not Detected

One common issue encountered by Fluent Bit users is the failure to detect log file rotation. This problem manifests when Fluent Bit continues to read from an old log file and does not switch to the new file after rotation, leading to incomplete log data being processed.

What You Might Observe

When Fluent Bit does not detect file rotation, you might notice that new log entries are not being forwarded to the configured output, or there is a significant delay in log processing. This can lead to missing logs in your centralized logging system.

Details About the Issue

The root cause of this issue often lies in the configuration of the input plugin used by Fluent Bit. Fluent Bit needs to be correctly configured to handle file rotations, especially when dealing with log files that are rotated by external processes or tools.

Common Misconfigurations

Some common misconfigurations include incorrect path settings, not enabling file rotation detection features, or using an input plugin that does not support file rotation. For example, using the tail input plugin without enabling the Rotate_Wait option can lead to this issue.

Steps to Fix the Issue

To resolve the issue of Fluent Bit not detecting file rotation, follow these steps:

1. Verify Input Plugin Configuration

Ensure that you are using the tail input plugin, which is designed to handle file rotation. Check your Fluent Bit configuration file to confirm that the tail plugin is correctly set up:

[INPUT] Name tail Path /var/log/myapp/*.log Refresh_Interval 5 Rotate_Wait 30 DB /var/log/flb_kv.db

Make sure the Rotate_Wait parameter is set to a reasonable value (in seconds) to allow Fluent Bit to detect the rotation.

2. Check File Path Configuration

Ensure that the Path parameter in your configuration matches the location and naming pattern of your log files. Incorrect paths can prevent Fluent Bit from accessing the correct files.

3. Enable Inotify

In some cases, enabling inotify can help Fluent Bit detect changes in the file system more efficiently. Add the following to your configuration:

Inotify_Watcher true

4. Review Log Rotation Settings

Check the log rotation settings of your application or system to ensure they are compatible with Fluent Bit's configuration. For example, if logs are rotated too frequently, increase the Rotate_Wait value.

Additional Resources

For more information on configuring Fluent Bit for file rotation, refer to the official Fluent Bit Tail Input Plugin Documentation. Additionally, the Fluent Bit Documentation provides comprehensive guidance on setting up and troubleshooting Fluent Bit.