cmake: backport #10778 and #10786 to 4.0.

[pwhelan]

Summary

Allow disabling simdutf support.

Description

Backport of the patches #10778 and #10786 to make the use of simdutf optional.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • Added Prometheus textfile input plugin.
  • Expanded Node Exporter metrics: sockstat, hwmon, path.rootfs support.
  • Windows Exporter: new cache and TCP metrics.
  • Elasticsearch: API key authentication.
  • Azure Kusto: supports service principal, managed identity, and workload identity auth.

• Improvements

  • NGINX exporter: configurable scrape_interval.
  • Output latency histogram metrics.
  • Syslog input batches and flushes encoded output.
  • Kinesis Firehose/Streams: robust port handling.

• Bug Fixes

  • OpenTelemetry logs: stricter ID validation and safer value handling.
  • cmetrics histogram allocation/off-by-one fixes.

• Chores

  • Version bump to 4.0.9.
  • Debian base images switched to archive mirrors.

[coderabbitai]

Caution

Review failed

Failed to post review comments.

Walkthrough

Adds new Prometheus textfile input; expands node/windows exporter metrics; introduces Windows TLS CertStore support; updates Azure Kusto auth (adds workload identity) and Elasticsearch API key; improves OTLP handling; adds output latency metrics; fixes JSON/CMetrics bugs; adds proxy-env ignore; multiple version bumps; Docker/CI updates; extensive Windows/Dockerfile changes; broad tests added.

Changes

Cohort / File(s)	Summary
Workflows .github/workflows/staging-build.yaml, .github/workflows/staging-release.yaml	Staging build frees disk space; staging release explicitly pulls image before retag/push.
Version bumps CMakeLists.txt, lib/cmetrics/CMakeLists.txt, dockerfiles/Dockerfile, fluent-bit-4.0.9.bb, snap/snapcraft.yaml	Bump Fluent Bit to 4.0.9; CMetrics patch to 5; propagate version in Dockerfile, Yocto recipe, and snap.
Build config / gating CMakeLists.txt, cmake/msgpack.cmake, cmake/plugins_options.cmake, plugins/CMakeLists.txt	Gate UNICODE encoder on SIMDUTF; define MSGPACK_EMBED_STACK_SIZE=64; add Prometheus textfile option; register new plugin.
Dockerfiles dockerfiles/Dockerfile.windows	Update to ltsc2025/MSVS 17; add vcpkg deps, cleanup, PATH fixes; copy build artifacts to runtime.
Compose/demo configs docker_compose/node-exporter-dashboard/docker-compose.yml, conf/fluent-bit-metrics.conf	Switch node-exporter paths to path.rootfs; add commented hints in metrics conf.
Core headers/APIs include/fluent-bit/flb_input_chunk.h, include/fluent-bit/flb_macros.h, include/fluent-bit/flb_metrics.h, include/fluent-bit/flb_network.h, include/fluent-bit/flb_opentelemetry.h, include/fluent-bit/flb_output.h, include/fluent-bit/tls/flb_tls.h	Add input chunk create_time; switch FLB_TRUE/FALSE to stdbool; add flb_metrics_is_empty(); add proxy_env_ignore; convert OTEL error codes to enum and add resource error; add Windows TLS fields and output latency histogram pointer.
Libraries (Windows path/dir handling) lib/chunkio/src/CMakeLists.txt, lib/chunkio/src/cio_os.c, lib/chunkio/src/win32/dirent.c	Remove Shlwapi.lib; replace SHCreateDirectoryExA with custom recursive mkdir; add PathIsDirectory replacement.
CMetrics fixes lib/cmetrics/src/cmt_cat.c, lib/cmetrics/src/cmt_histogram.c	Off-by-one fix in histogram buckets iteration; fix allocation for upper_bounds (count+1).
Core runtime/engine src/flb_engine.c, src/flb_input.c, src/flb_input_chunk.c, src/flb_input_metric.c, src/flb_metrics.c, src/flb_network.c, src/flb_output.c, src/flb_upstream.c, src/flb_pack.c	Add output latency histogram and use input chunk create_time; adjust timer fd handling; initialize chunk create_time; skip empty metrics on append; add flb_metrics_is_empty(); add proxy_env_ignore config/field; add Windows TLS CertStore support and latency buckets; add net.proxy_env_ignore config; JSON pack numeric/string/token count improvements.
OpenTelemetry input src/opentelemetry/flb_opentelemetry_logs.c, src/opentelemetry/flb_opentelemetry_utils.c	Enforce trace/span ID lengths; new resource invalid-attribute error path; stricter type checks; NIL/empty handling; adjusted error propagation.
AWS core src/aws/flb_aws_credentials_ec2.c	After upstream set, restore IMDS timeouts and disable keepalive.
New input: Prometheus textfile plugins/in_prometheus_textfile/*	Add textfile input plugin with globbing, scrape_interval, decoding via cmetrics, tests wiring.
Node exporter metrics plugins/in_node_exporter_metrics/*	Add sockstat and hwmon collectors (Linux impls + non-Linux stubs); add path.rootfs and hwmon filters; update defaults; compose paths from rootfs; minor stat error log.
NGINX exporter metrics plugins/in_nginx_exporter_metrics/nginx.{c,h}	Add configurable scrape_interval.
Kafka input plugins/in_kafka/in_kafka.c	Batch append result handling; commit after successful batch append when auto-commit disabled.
Syslog input plugins/in_syslog/syslog_prot.c	Reset encoder at start; defer append to end of processing; flush if output pending.
Windows exporter metrics – framework plugins/in_windows_exporter_metrics/CMakeLists.txt, we.c, we.h, we_metric.{c,h}	Add cache and TCP collectors; new perflib source handling (secondary value); add new struct fields and timers; link iphlpapi.
Windows exporter metrics – collectors we_cache.{c,h}, we_cpu.c, we_logical_disk.c, we_net.c, we_os.c, we_wmi_paging_file.c, we_wmi_memory.c, we_wmi_tcp.{c,h}	New cache collector; add CPU utility metrics; logical disk: add size/free bytes via WinAPI, queue length gauges; add net output queue length; use dynamic TZ; paging file gauges rename/labels; memory gauge rename; new TCP collector (WMI + state scan).
Azure Kusto output plugins/out_azure_kusto/*	Add auth_type (service principal/MSI/workload identity) and workload_identity_token_file; implement workload identity token exchange; random jitter helper; config validation changes.
Elasticsearch output plugins/out_es/es.{c,h}	Add http_api_key config and Authorization header handling.
CloudWatch logs output plugins/out_cloudwatch_logs/*	Add optional entity extraction/filtering; new entity data types; gated by kubernetes metadata detection and add_entity option; payload filtering helpers.
Chronicle output plugins/out_chronicle/*	Add test mode bypass for token retrieval; add test formatter; batch entries and refactor formatting; mutex for token.
Kinesis outputs plugins/out_kinesis_firehose/*, plugins/out_kinesis_streams/*	Firehose: dynamic port selection with uint16_t field; Streams: change port type/range logic; remove min/max macros.
Kafka output cleanup plugins/out_kafka/kafka_config.c	Stop freeing topic_key/message_key_field in destroy.
Windows EventLog input plugins/in_winevtlog/pack.c	Switch to Dynamic Time Zone APIs; _tzset(); DST inference tweaks.
Tests – runtime/internal tests/runtime/*, tests/internal/*	Add runtime tests for textfile and Chronicle; add OTEL binary size test; new packer tests (empty arrays, large uint64, token count overflow); test data updates.
Packaging – Debian packaging/distros/debian/Dockerfile	Switch debian/security mirrors to archive.debian.org for buster bases.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant User
  participant FB as Fluent Bit
  participant PT as in_prometheus_textfile
  participant FS as Filesystem
  participant CM as cmetrics Decoder

  User->>FB: Starts pipeline
  FB->>PT: cb_init (paths, scrape_interval)
  loop Every scrape_interval
    PT->>FS: Expand globs, read files
    alt file readable
      PT->>CM: decode(prometheus_text)
      alt decode ok
        CM-->>PT: cmt object
        PT-->>FB: flb_input_metrics_append(cmt)
      else decode error
        PT-->>PT: log error
      end
    else unreadable/empty
      PT-->>PT: log/debug
    end
  end

Loading

sequenceDiagram
  autonumber
  participant Out as out_azure_kusto
  participant OAuth as OAuth2 helper
  participant MSI as Azure MSI
  participant WI as Workload Identity

  Out->>Out: get_azure_kusto_token()
  alt token expired
    opt auth_type == workload_identity
      Out->>WI: read federated token file
      WI-->>Out: exchange for access token
    end
    opt auth_type == managed_identity (system/user)
      Out->>MSI: request token (system or user-assigned)
      MSI-->>Out: token
    end
    opt auth_type == service_principal
      Out->>OAuth: client_credentials grant
      OAuth-->>Out: token
    end
  else valid token
    Out-->>Out: reuse
  end

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120+ minutes

Possibly related PRs

• in_node_exporter_metrics: add Linux hwmon collector #10719 — Adds Linux hwmon collector to in_node_exporter_metrics (matches new hwmon files and registrations).
• pack: fix token count regression in JSON tokenizer and remove parser reset #10733 — Adjusts src/flb_pack.c tokenization/token counting (same logic updates here).
• out_cloudwatch: add entity support and remove unnecessary log content #10585 — CloudWatch Logs entity support (same files/functions adding entity parsing and payload changes).

Suggested labels

docs-required, backport to v4.0.x

Suggested reviewers

• niedbalski
• patrick-stephens
• cosmo0920

Poem

hop hop, I nibble code like clover leaves—
new metrics bloom, from textfile sheaves.
Windows certs? a burrowed stash!
Kusto keys and ES hash.
Sockets sing, hwmon hums,
OTLP tightens, latency drums.
ʕ•́ᴥ•̀ʔっ(=^‥^=) speedy buns!

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch pwhelan-backport-simdutf-optional-v4.0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbit in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbit in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbit gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbit read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbit help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbit ignore or @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbit summary or @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbit or @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.