NETOBSERV-1324: Allow skipping per-query namespace inference via config

[jotak]

This is a way to open the current behaviour of Loki "meta-data" queries to other use cases where the list of namespaces cannot be inferred from queries.

NetObserv will use this flag because its queries are not namespaced. This allows to effectively use the fine-grained SAR feature with netobserv.

Here's a diagram summarizing the workflow related to namespaced SARs. Note that this PR doesn't change the diagram flows; it's only adding a use case to one of the branching, mentioned as "typical netobserv query"

 ┌───────────────────────────────────┐       ┌─────────────────────────────────────────────────────────┐
 │                                   │       │                                                         │
 │ Namespace(s) can be inferred      │       │ Namespace(s) can't be inferred                          │
 │ from query                        │       │ from query                                              │
 │ E.g: query/q=                     │       │ E.g:                                                    │
 │ {namespace="myns"} | pod:"my-pod" │       │ - label/key/values (metadata query)                     │
 │                                   │       │ - query/q={src_node="node-1"} (typical netobserv query) │
 └────────────────┬──────────────────┘       │                                                         │
                  │                          └─────────────────────────┬───────────────────────────────┘
                  │                                                    │
                  │                                                    │
       ┌──────────▼───────────┐                        ┌───────────────▼───────────────┐
       │                      │                        │                               │
       │ Extract namespace(s) │                        │ Get all accessible namespaces │
       │                      │                        │                               │
       └───────────┬──────────┘                        └───────────────┬───────────────┘
                   │                                                   │
                   │                                                   │
                   │                                                   │
                   │                                                   │
                   │             ┌────────────────────────┐            │
                   └─────────────►                        ◄────────────┘
                                 │ SAR for each namespace │
                                 │                        │
                                 └───────────┬────────────┘
                                             │
                                             │
                                             │
                            ┌────────────────▼───────────────────┐
                            │                                    │
                            │ Inject allowed namespaces in query │
                            │                                    │
                            └────────────────────────────────────┘

[jotak]

@periklis I'm testing and monitoring the impact on Kube API, what I can already tell is that the impact is largely contained by the QPS and Burst settings of the kube client, which by default lead to just 5 queries per second: that's very conservative and protective for the kube API and on the other hand it makes the SAR queries very long to answer when there are a lot of namespaces to check. For instance, 100 allowed namespaces lead to 20 seconds (minimum) for the query. Results are cached per gateway instance, which mitigates this slow response problem (next calls are much faster).

With this protective QPS, I didn't notice any negative impact on the kube API; for instance, check for reponse time metrics, they're very stable, looking not impacted by the many SAR queries.

[jotak]

Tested with 100 SARs and a QPS/Burst of 50 makes the queries much faster (10x faster) and still no noticeable impact on the kube API performance:

Expected increase in queries per second:

No visible impact on response time:

Note that the QPS / Burst client settings are shared across users (per gateway instance), so I don't think having multiple concurrent users would have much more impact ...

[xperimental]

@jotak Sorry, I did not get to this yet, but it is on my list. I want to first test this on a cluster myself, so that I can understand what you're trying to achieve. I hope I'll make it this week.

[jotak]

thanks @xperimental - I can help you to set up netobserv if you want: you would see how it enables fine-grained rbac for us

[jotak]

Hey @xperimental, @periklis : this has been waiting for a very long time, any chance you could look at it?

[jotak]

Reopening as we have asks to implement fine-grained rbac in netobserv. (I'll rebase it)

[jotak]

For testing, we'll need to reopen this as well: grafana/loki#13002