From 046be247f28ce55f3729c2a2b6b92129b4709931 Mon Sep 17 00:00:00 2001 From: JacksonDavenport Date: Wed, 9 Jul 2025 13:45:11 -0700 Subject: [PATCH 1/4] Update AKS instructions for new certificate rotation format --- .../en/containers/kubernetes/distributions.md | 86 ++++++++++++++++--- 1 file changed, 74 insertions(+), 12 deletions(-) diff --git a/content/en/containers/kubernetes/distributions.md b/content/en/containers/kubernetes/distributions.md index 9aee8d749c073..5eaf64f697fb1 100644 --- a/content/en/containers/kubernetes/distributions.md +++ b/content/en/containers/kubernetes/distributions.md @@ -74,7 +74,18 @@ spec: ## Azure Kubernetes Service (AKS) {#AKS} -AKS requires a specific configuration for the `Kubelet` integration due to how AKS has set up the SSL Certificates. Additionally, the optional [Admission Controller][1] feature requires a specific configuration to prevent an error when reconciling the webhook. +Your AKS cluster may need a specific configuration to enable the Datadog Agent to connect to the Kubelet depending on its version. If your cluster has [kubelet serving certificate rotation][13] enabled, you do **not** need to provide any special configuration. This feature is enabled in Kubernetes clusters 1.27 and above on node pools updated after July 2025. + +Your nodes have this feature enabled if they have the label `kubernetes.azure.com/kubelet-serving-ca=cluster`. Verify if all of your nodes have this label by running: + +```shell +kubectl get nodes -L kubernetes.azure.com/kubelet-serving-ca +``` + +If all your nodes show `cluster` do not provide any specific `kubelet` configuration as the Agent will connect automatically. If your nodes do not have this feature enabled use the [Kubelet configurations without certificate rotation](#without-kubernetes-certificate-rotation). +**Note:** This configuration should be removed once certificate rotation is enabled in your cluster. + +Lastly, the optional [Admission Controller][1] feature requires a specific configuration to prevent an error when reconciling the webhook. {{< tabs >}} {{% tab "Datadog Operator" %}} @@ -93,12 +104,6 @@ spec: credentials: apiKey: appKey: - kubelet: - host: - valueFrom: - fieldRef: - fieldPath: spec.nodeName - hostCAPath: /etc/kubernetes/certs/kubeletserver.crt override: clusterAgent: containers: @@ -116,6 +121,66 @@ Replace `` with your [Datadog site][1]. Your site is {{< region-pa Custom `datadog-values.yaml`: +```yaml +datadog: + clusterName: + apiKey: + appKey: + +providers: + aks: + enabled: true +``` + +The `providers.aks.enabled` option sets the necessary environment variable `DD_ADMISSION_CONTROLLER_ADD_AKS_SELECTORS="true"` for you. + +{{% /tab %}} +{{< /tabs >}} + +### Without Kubernetes certificate rotation + +**Note:** When upgrading your AKS cluster you may see the [certificate rotation][13] feature enabled for you automatically which can negatively impact your Datadog Agent if you are using the below configuration to reference the certificate `/etc/kubernetes/certs/kubeletserver.crt`. This certificate file is removed once this feature enabled. Which can cause: + +- In Datadog Operator: The Agent container shuts down in `Error` as it cannot connect to the Kubelet and it logs `Error while getting hostname, exiting: unable to reliably determine the host name` +- In Helm: The Agent pod fails to start with the warning event `MountVolume.SetUp failed for volume "kubelet-ca" : hostPath type check failed: /etc/kubernetes/certs/kubeletserver.crt is not a file` + +In these cases remove the kubelet configurations and return to the defaults as seen above. Alternatively [connecting to the kubelet without TLS Verification](#without-tls-verification) is still supported on all AKS versions. + +{{< tabs >}} +{{% tab "Datadog Operator" %}} + +DatadogAgent Kubernetes Resource: + +```yaml +kind: DatadogAgent +apiVersion: datadoghq.com/v2alpha1 +metadata: + name: datadog +spec: + global: + clusterName: + site: + credentials: + apiKey: + appKey: + kubelet: + host: + fieldRef: + fieldPath: spec.nodeName + hostCAPath: /etc/kubernetes/certs/kubeletserver.crt + override: + clusterAgent: + containers: + cluster-agent: + env: + - name: DD_ADMISSION_CONTROLLER_ADD_AKS_SELECTORS + value: "true" +``` +{{% /tab %}} +{{% tab "Helm" %}} + +Custom `datadog-values.yaml`: + ```yaml datadog: clusterName: @@ -132,14 +197,10 @@ providers: aks: enabled: true ``` - -The `providers.aks.enabled` option sets the necessary environment variable `DD_ADMISSION_CONTROLLER_ADD_AKS_SELECTORS="true"` for you. - {{% /tab %}} - {{< /tabs >}} -The AKS Kubelet certificate requires changing the Kubelet host to the `spec.nodeName` and the `hostCAPath` location of the certificate, as seen in the previous snippets. This enables TLS verification. Without these changes, the Agent cannot connect to the Kubelet. +In these AKS node versions, the AKS Kubelet certificate requires changing the Kubelet host to the `spec.nodeName` and the `hostCAPath` location of the certificate, as seen in the previous snippets. This enables TLS verification. Without these changes, the Agent cannot connect to the Kubelet. ### Without TLS verification @@ -607,3 +668,4 @@ agents: [10]: https://cloud.google.com/kubernetes-engine/docs/how-to/autopilot-spot-pods [11]: https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-compute-classes [12]: https://cloud.google.com/kubernetes-engine/docs/how-to/disable-kubelet-readonly-port +[13]: https://learn.microsoft.com/en-us/azure/aks/certificate-rotation#kubelet-serving-certificate-rotation From 4be0284e4a6389ca67d0e83375020965a2013b2c Mon Sep 17 00:00:00 2001 From: JacksonDavenport Date: Wed, 9 Jul 2025 13:51:28 -0700 Subject: [PATCH 2/4] Minor changes for old setup --- content/en/containers/kubernetes/distributions.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/en/containers/kubernetes/distributions.md b/content/en/containers/kubernetes/distributions.md index 5eaf64f697fb1..dfd62a598f757 100644 --- a/content/en/containers/kubernetes/distributions.md +++ b/content/en/containers/kubernetes/distributions.md @@ -82,7 +82,7 @@ Your nodes have this feature enabled if they have the label `kubernetes.azure.co kubectl get nodes -L kubernetes.azure.com/kubelet-serving-ca ``` -If all your nodes show `cluster` do not provide any specific `kubelet` configuration as the Agent will connect automatically. If your nodes do not have this feature enabled use the [Kubelet configurations without certificate rotation](#without-kubernetes-certificate-rotation). +If all your nodes show `cluster` do not provide any specific `kubelet` configuration as the Agent successfully connects by default. If your nodes do not have this feature enabled use the [Kubelet configurations without certificate rotation](#without-kubelet-serving-certificate-rotation). **Note:** This configuration should be removed once certificate rotation is enabled in your cluster. Lastly, the optional [Admission Controller][1] feature requires a specific configuration to prevent an error when reconciling the webhook. @@ -137,7 +137,7 @@ The `providers.aks.enabled` option sets the necessary environment variable `DD_A {{% /tab %}} {{< /tabs >}} -### Without Kubernetes certificate rotation +### Without Kubelet serving certificate rotation **Note:** When upgrading your AKS cluster you may see the [certificate rotation][13] feature enabled for you automatically which can negatively impact your Datadog Agent if you are using the below configuration to reference the certificate `/etc/kubernetes/certs/kubeletserver.crt`. This certificate file is removed once this feature enabled. Which can cause: @@ -146,6 +146,8 @@ The `providers.aks.enabled` option sets the necessary environment variable `DD_A In these cases remove the kubelet configurations and return to the defaults as seen above. Alternatively [connecting to the kubelet without TLS Verification](#without-tls-verification) is still supported on all AKS versions. +When this feature is not enabled you can provide the Datadog Agent an updated kubelet configuration to allow it to properly connect. + {{< tabs >}} {{% tab "Datadog Operator" %}} From 83822cd9742fc5bd7c10784daf3724f9673de07c Mon Sep 17 00:00:00 2001 From: cecilia saixue watt Date: Tue, 15 Jul 2025 12:56:35 -0700 Subject: [PATCH 3/4] edits --- .../en/containers/kubernetes/distributions.md | 42 +++++++++++-------- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/content/en/containers/kubernetes/distributions.md b/content/en/containers/kubernetes/distributions.md index dfd62a598f757..2bbc2dd5a4987 100644 --- a/content/en/containers/kubernetes/distributions.md +++ b/content/en/containers/kubernetes/distributions.md @@ -74,18 +74,8 @@ spec: ## Azure Kubernetes Service (AKS) {#AKS} -Your AKS cluster may need a specific configuration to enable the Datadog Agent to connect to the Kubelet depending on its version. If your cluster has [kubelet serving certificate rotation][13] enabled, you do **not** need to provide any special configuration. This feature is enabled in Kubernetes clusters 1.27 and above on node pools updated after July 2025. - -Your nodes have this feature enabled if they have the label `kubernetes.azure.com/kubelet-serving-ca=cluster`. Verify if all of your nodes have this label by running: - -```shell -kubectl get nodes -L kubernetes.azure.com/kubelet-serving-ca -``` - -If all your nodes show `cluster` do not provide any specific `kubelet` configuration as the Agent successfully connects by default. If your nodes do not have this feature enabled use the [Kubelet configurations without certificate rotation](#without-kubelet-serving-certificate-rotation). -**Note:** This configuration should be removed once certificate rotation is enabled in your cluster. - -Lastly, the optional [Admission Controller][1] feature requires a specific configuration to prevent an error when reconciling the webhook. +### Admission Controller +The optional [Admission Controller][1] feature requires a specific configuration to prevent an error when reconciling the webhook. {{< tabs >}} {{% tab "Datadog Operator" %}} @@ -137,16 +127,21 @@ The `providers.aks.enabled` option sets the necessary environment variable `DD_A {{% /tab %}} {{< /tabs >}} -### Without Kubelet serving certificate rotation +### Kubelet serving certificate rotation +If your cluster, **does not** have [Kubelet serving certificate rotation][13] enabled, you must provide additional configuration to enable the Datadog Agent to connect to the Kubelet. Kubelet serving certificate rotation is enabled in Kubernetes clusters 1.27 and above on node pools updated after July 2025. -**Note:** When upgrading your AKS cluster you may see the [certificate rotation][13] feature enabled for you automatically which can negatively impact your Datadog Agent if you are using the below configuration to reference the certificate `/etc/kubernetes/certs/kubeletserver.crt`. This certificate file is removed once this feature enabled. Which can cause: +Your nodes have this feature enabled if they have the label `kubernetes.azure.com/kubelet-serving-ca=cluster`. Verify if all of your nodes have this label by running: -- In Datadog Operator: The Agent container shuts down in `Error` as it cannot connect to the Kubelet and it logs `Error while getting hostname, exiting: unable to reliably determine the host name` -- In Helm: The Agent pod fails to start with the warning event `MountVolume.SetUp failed for volume "kubelet-ca" : hostPath type check failed: /etc/kubernetes/certs/kubeletserver.crt is not a file` +```shell +kubectl get nodes -L kubernetes.azure.com/kubelet-serving-ca +``` -In these cases remove the kubelet configurations and return to the defaults as seen above. Alternatively [connecting to the kubelet without TLS Verification](#without-tls-verification) is still supported on all AKS versions. +Ensure that all your nodes show `cluster`. +**Note:** -When this feature is not enabled you can provide the Datadog Agent an updated kubelet configuration to allow it to properly connect. +#### Without Kubelet serving certificate rotation + +If Kubelet serving certificate rotation is not enabled, provide the following additional Kubelet configuration: {{< tabs >}} {{% tab "Datadog Operator" %}} @@ -204,6 +199,17 @@ providers: In these AKS node versions, the AKS Kubelet certificate requires changing the Kubelet host to the `spec.nodeName` and the `hostCAPath` location of the certificate, as seen in the previous snippets. This enables TLS verification. Without these changes, the Agent cannot connect to the Kubelet. +
After Kubelet serving certificate rotation is enabled in your cluster, remove this configuration.
+ +When you upgrade your AKS cluster, you may see the Kubelet serving certificate rotation feature enabled for you automatically, which can negatively impact your Datadog Agent if you are using the above special configuration to reference the certificate `/etc/kubernetes/certs/kubeletserver.crt`. When Kubelet serving certificate rotation is enabled, this certificate is removed, causing: + +- In Datadog Operator: The Agent container shuts down in `Error`, as it cannot connect to the Kubelet, and it logs `Error while getting hostname, exiting: unable to reliably determine the host name` +- In Helm: The Agent pod fails to start with the warning event `MountVolume.SetUp failed for volume "kubelet-ca" : hostPath type check failed: /etc/kubernetes/certs/kubeletserver.crt is not a file` + +In these cases, remove the additional Kubelet configurations. + +As an alternative, you can also [connect to the Kubelet without TLS verification](#without-tls-verification). + ### Without TLS verification In some clusters, DNS resolution for `spec.nodeName` inside Pods does not work in AKS. This affects: From c9d1ebf5f437f04df2e35447fab08b3c35cc8c17 Mon Sep 17 00:00:00 2001 From: JacksonDavenport Date: Tue, 15 Jul 2025 15:33:48 -0700 Subject: [PATCH 4/4] Fix typo --- content/en/containers/kubernetes/distributions.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/content/en/containers/kubernetes/distributions.md b/content/en/containers/kubernetes/distributions.md index 2bbc2dd5a4987..f002c44f70f19 100644 --- a/content/en/containers/kubernetes/distributions.md +++ b/content/en/containers/kubernetes/distributions.md @@ -136,8 +136,7 @@ Your nodes have this feature enabled if they have the label `kubernetes.azure.co kubectl get nodes -L kubernetes.azure.com/kubelet-serving-ca ``` -Ensure that all your nodes show `cluster`. -**Note:** +Ensure that all your nodes show `cluster`. #### Without Kubelet serving certificate rotation