Unable to override Prometheus annotation

[arkodg]

@arkodg

Yes, so with this tested configuration:

kind: EnvoyProxy
spec:
  provider:
    kubernetes:
        pod:
          annotations:
            prometheus.io/scrape: "false"
  telemetry:
    metrics:
      prometheus:
        disable: false

still results in a pod with these annotations

│     prometheus.io/path: /stats/prometheus                                                                                                                                                                                                                                                                                                                                                                                       │
│     prometheus.io/port: "19001"                                                                                                                                                                                                                                                                                                                                                                                                 │
│     prometheus.io/scrape: "true"

Yes the intention is to to still have the container ports 19001 exposed and the prometheus metrics being generated , just wait to edit annotations

The reason for this is that it causes the datadog agent "openmetrics" check to kick in https://docs.datadoghq.com/containers/kubernetes/prometheus/?tab=kubernetesadv2#advanced-configuration and start scraping the metrics (costing us money). Instead we want the native envoy integration checks which are free https://docs.datadoghq.com/integrations/envoy/ but requires us to turn off the scrape annotation prometheus.io/scrape: "false"

I also tried

kind: Gateway
spec:
  gatewayClassName: envoy-gateway
  infrastructure:
    annotations:
      prometheus.io/scrape: "false"

but it did not regenerate a new replicaset

Originally posted by @crhuber in #6410 (comment)

[zirain]

let me take a look at it.

[arkodg]

thanks @zirain for proactively picking up this one

there are two approaches here

1. apply user annotations in the end so they can override system applied annotations
2. add a disableAnnotations field under the metrics.prometheus setting

wdyt @envoyproxy/gateway-maintainers, im +1 for 1, which maybe more useful in a generic way

@crhuber to unblock yourself, you can use the patch field to override annotations
https://gateway.envoyproxy.io/docs/tasks/operations/customize-envoyproxy/#customize-envoyproxy-with-patches

[crhuber]

@arkodg i can config the customization with patches does work

    provider:
      type: Kubernetes
      kubernetes:
        envoyDeployment:
          patch:
            type: StrategicMerge
            value:
              spec:
                template:
                  metadata:
                    annotations:
                      prometheus.io/scrape: "false"

[stevehipwell]

I'm interested in disabling the annotations from the context of using Prometheus Operator (e.g. ServiceMonitor & PodMonitor).

[zhaohuabing]

+1 on 1

This is more difficult to understand:

kind: EnvoyProxy
spec:
  telemetry:
    metrics:
      prometheus:
        disable: false
        disableAnnotations: true

[arkodg]

@guydc and I, plus +1d 1 in the community meeting this week

[Xunzhuo]

+1 for option 1, the UX is similar to how we customize envoyproxy service annotation.