enhancement(kubernetes_logs source): Allow disabling listwatching of namespaces

[imbstack]

Summary

On clusters with high numbers of namespaces, Vector's memory can grow rather large. Additionally the load that Vector puts on kube-apiserver can be rather heavy when rolling the Vector daemonset or otherwise have many Vector pods restarting. This change allows you to opt out of having namespace labels available for enrichment. I believe that if you try to use them with namespace listwatching disabled Vector will gracefully log an error rather than crash or something along those lines which feels like a good way to handle this scenario?

This is my first contribution to Vector, I think I've read through the guidelines and such but am happy to make any changes you think would be best, this is just my guess at a good way to solve this issue for us.

Vector configuration

There's quite significant amount of config on my test setup in our staging clusters. I can try to trim it down and get a test setup going if we'd be more comfortable with this change being tested that way first though.

How did you test this PR?

I backported these changes (technically just 6389f2f0c16a157579e2cfe4662d280e8688608a, the changes I made after that have not been deployed anywhere) to 0.37.1 and built/deployed images based on this into our staging environment. This cut the memory usage of Vector pods in half in a cluster with many tens of thousands of namespaces. I have not tested the version of this based on master. I'm happy to get a test setup going to try this out if we really want to, the clusters I have available to me now need to be updated too significantly for me to want to bother with it on them at the moment.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Implements: #22990

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

[imbstack]

Ah, this seems to have broken something. It runs but isn't emitting the logs I expect. I'll close this and re-open when I figure this out. Sorry for the noise!

This seems to work now!

[rtrieu]

Some suggestions to improve readability. Let me know if you have any questions about my feedback!

[imbstack]

Great feedback, thank you!