send logs to multiple destinations dynamically

[ffatghub]

Hello team,

I am in a containerized environments and have a specific requirement. I am looking a way to collect logs directly from the standard output (stdout) of containers and send this data to 2 different output plugins: the Standard output plugin and the HTTP output plugin. Depending on the value of an attribute in a JSON configuration file, say “logDestination” I want to switch at runtime between the former, the latter or both (duplicating input). Suppose logDestination=stdout, logDestination=http or logDestination=both.

Is there an existing feature in fluent-bit that supports this kind of dynamic data routing?
If not, could this be implemented with a plugin (input, filter, output)?
What would be the recommended approach to implement this functionality if it doesn’t currently exist?

Thanks a lot for your help

[patrick-stephens]

Rewrite_tag lets you create a tag based on another key.

You can always also send to both then use grep processor on each output to drop logs you do not want. That's probably simpler and it's not Fluentd so logs are not duplicated per output.

Something like this:

  outputs:
    - name: stdout
      match: '*'
      processors:
        logs:
          - name: grep
            exclude: $logDestination http
    - name: http
      match: '*'
      processors:
        logs:
          - name: grep
            exclude: $logDestination stdout