Read_from_Head only catches up after file is updated again

[erpel]

Before raising a bug, I'd like to try and confirm my observations.
TL;DR: A file with existing content will only be read and shipped from the beginning, if it is updated after being discovered.
A file that just appears and never gets updated does not get log lines processed.

Situation

Running cr.fluentbit.io/fluent/fluent-bit:4.0.7 on EKS as daemonset on ec2 nodes.
Pod is created that causes a new node to be created via autoscaling (so completely fresh instance of fluent-bit.
Pod is scheduled once fluent bit is started.

Config:

----                                                                                                                                                                             
[SERVICE]                                                                                                                                                                        
    Daemon Off                                                                                                                                                                   
    Flush 1                                                                                                                                                                      
    Log_Level debug                                                                                                                                                              
    Parsers_File /fluent-bit/etc/parsers.conf                                                                                                                                    
    Parsers_File /fluent-bit/etc/conf/custom_parsers.conf                                                                                                                        
    HTTP_Server On                                                                                                                                                               
    HTTP_Listen 0.0.0.0                                                                                                                                                          
    HTTP_Port 2020                                                                                                                                                               
    Health_Check On                                                                                                                                                              
                                                                                                                                                                                 
[INPUT]                                                                                                                                                                          
    Name tail                                                                                                                                                                    
    Path /var/log/containers/*.log                                                                                                                                               
    multiline.parser docker, cri                                                                                                                                                 
    Tag kube.*                                                                                                                                                                   
    # Make sure to read log lines that were emitted before fluentbit was up                                                                                                      
    Read_from_Head True                                                                                                                                                          
    # Keep track of file status across restart to prevent Read_from_Head duplicating lines on restart                                                                            
    DB /var/log/flb_kube_1.db                                                                                                                                                    
    Mem_Buf_Limit 5MB                                                                                                                                                            
    Skip_Long_Lines On      
[…]

Observation

What I see is this: A service writes a log line and then does nothing. The file gets discovered by fluent-bit but since it's not updated, the existing content is not processed.

If the service is modified to repeat the line after 60s, fluent-bit receives an update notification for the file and only then processes starting from the beginning as configured with Read_from_Head.

Is this known behavior? Could this be found in the docs? Is there a purpose behind it?
I'd really appreciate any feedback on this, don't want to waste developers time if I'm completely missing the mark.

fluent-bit logs

2025-08-21T14:15:44.168905046Z [2025/08/21 14:15:44] [debug] [input:tail:tail.0] file will be read in POSIX_FADV_DONTNEED mode /var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log
2025-08-21T14:15:44.168909036Z [2025/08/21 14:15:44] [debug] [input:tail:tail.0] inode=335787805 with offset=0 appended as /var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log
2025-08-21T14:15:44.168914626Z [2025/08/21 14:15:44] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log, inode 335787805
2025-08-21T14:15:44.168918846Z [2025/08/21 14:15:44] [debug] [input:tail:tail.0] 3 new files found on path '/var/log/containers/*.log'

2025-08-21T14:15:44.206856987Z [2025/08/21 14:15:44] [debug] [filter:kubernetes:kubernetes.1] Request (ns=karpenter-autoscaling, pod=log-generator-675bb55f8-wx6l4) http_do=0, HTTP Status: 200

2025-08-21T14:15:44.208443723Z [2025/08/21 14:15:44] [debug] [input:tail:tail.0] inode=335787805 file=/var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log promote to TAIL_EVENT
2025-08-21T14:15:44.208458463Z [2025/08/21 14:15:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=335787805 watch_fd=13 name=/var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log
2025-08-21T14:15:44.208467644Z [2025/08/21 14:15:44] [debug] [input:tail:tail.0] [static files] processed 10.4K

2025-08-21T14:15:50.323485223Z [2025/08/21 14:15:50] [debug] [input:tail:tail.0] inode=335787805, /var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log, events: IN_MODIFY 

2025-08-21T14:16:44.168403708Z [2025/08/21 14:16:44] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log, inode 335787805

2025-08-21T14:17:44.168681088Z [2025/08/21 14:17:44] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log, inode 335787805

2025-08-21T14:17:57.422598485Z [2025/08/21 14:17:57] [debug] [input:tail:tail.0] [static files] processed 32.0K
2025-08-21T14:17:57.422630056Z [2025/08/21 14:17:57] [debug] [input:tail:tail.0] inode=335787805, /var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log, events: IN_ATTRIB 
2025-08-21T14:17:57.423804853Z [2025/08/21 14:17:57] [debug] [input:tail:tail.0] inode=335787805 file has been deleted: /var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log
2025-08-21T14:17:57.423812583Z [2025/08/21 14:17:57] [debug] [input:tail:tail.0] db: file deleted from database: /var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log
2025-08-21T14:17:57.423816744Z [2025/08/21 14:17:57] [debug] [input:tail:tail.0] inode=335787805 removing file name /var/log/containers/log-generator-675bb55f8-wx6l4_karpenter-autoscaling_log-generator-fed1a387e739649d68ccff152e3a3188c44e246aa7ffaeff3780a10b81c2919f.log

Logs added to the file that is monitored

    "@timestamp": "2025-08-21T14:14:50.232Z",     "message": "eks-cluster-logshipping-marker-karpenter-autoscaling-cpt-5284-1755785635426032000",
    "@timestamp": "2025-08-21T14:15:50.233Z",     "message": "eks-cluster-logshipping-marker-karpenter-autoscaling-cpt-5284-1755785635426032000",

[patrick-stephens]

Yes, I think so.

Read from head only indicates what to do for that file when it is new or updated, it affects the behaviour of what happens on an update notification. Essentially tail functions as tail -f by default so this only gets "new" data, read from head says to read the whole file when this happens. You also have the db option in there which confuses things as this is intended to track where it last read in the file.

Combining read-from-head and the db option does not seem right: the database is responsible for tracking the offset in the file it has read up to, if the file is new then it will read from the start (i.e. offset=0) but if not it will read from where it last stopped. There is no need to tell it to read from the start.

[erpel]

Hi, thanks for your thoughts.

I take the initial "Yes" from your reply to mean that this is not considered wrong but rather expected behavior that just happens to be inconvenient in the edge case of "log file does not get updated after fluent-bit is started.

Is this too much of an edge case to put into the manual? I could raise an MR in in the fluent-bit-docs repo to spare others the headscratching I did to arrive at this conclusion.

About the combination of options: I could not verify this now, but to me the documentation does not look like db makes read_from_head unnecessary.

For new discovered files on start (without a database offset/position), read the content from the head of the file, not tail.

If a database file is present, the plugin restores the last known position (offset) from the database. If no previous position exists and read_from_head is false, it starts monitoring from the end of the file.

This reads to me like "If you use db and the db does not have a position then read from head".
If db implicitly contains read_from_head then this could be made more clear.

[patrick-stephens]

Yeah it's confusing so definitely send a docs update :)