[Bug]: `PodDisruptionBudget` not Created Per `StrimziPodSet` When Using `KafkaNodePool`

[colechiumento-coreweave]

Bug Description

When using KafkaNodePools to manage a broker NodePool and controller NodePool separately, it appears as though the PodDisruptionBudget is being created based on the Kind: Kafka resource still, rather than the KafkaNodePool resources used to manage the brokers and controllers. This does not seem to be expected behavior, according to the documentation for PodDisruptionBudgetTemplate:

Strimzi creates a PDB for every new StrimziPodSet or Deployment. By default, the PDB allows only one pod to be unavailable at any given time.

Additionally, it does not appear as though the PodDisruptionBudgetTemplate can be modified to include multiple PodDisruptionBudget resources, although that may be a skill issue/user oversight on my part.

A current workaround would be adding a bunk PodDisruptionBudgetTemplate (e.g. stripping out an references to Strimzi resources) and then creating our own separate from Strimzi, but ideally we would be able to use the KafkaNodePool resource and see a PodDisruptionBudget be created for each KafkaNodePool/child StrimziPodSet when strimzi.io/node-pools: enabled is included on the Kafka resource.

I've included my configuration below for reference/to reproduce the issue. Please let me know if there is any additional information I can offer. Also, thank you for maintaining such a well thought out and feature-rich operator. Strimzi has really been a joy to use.

Steps to reproduce

Create a Kafka cluster with two KafkaNodePool resources for managing brokers and controllers.

Inspect StrimziPodSet and PodDisruptionBudget resources using kubectl (aliased to k here):

$ k get poddisruptionbudget -n kafka
NAME                        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
kafka-kafka                 3               N/A               1                     96d
$ k get strimzipodsets -n kafka
NAME                        PODS   READY PODS   CURRENT PODS   AGE
kafka-broker                3      3            3              96d
kafka-controller            1      1            1              96d

Expected behavior

A PodDisruptionBudget is created per StrimziPodSet when using KafkaNodePool to manage brokers and controllers, so that there is a separate PodDisruptionBudget for brokers and controllers:

$ k get strimzipodsets -n kafka
NAME                        PODS   READY PODS   CURRENT PODS   AGE
kafka-broker                3      3            3              96d
kafka-controller            1      1            1              96d
$ k get poddisruptionbudget -n kafka
NAME                        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
kafka-nodepool-broker                 3               N/A               1                     96d
kafka-nodepool-controller                 3               N/A               1                     96d

The names for the PodDisruptionBudget do not matter, simply that there is one per StrimziPodSet.

Strimzi version

0.45.0, 0.46.0

Kubernetes version

Kubernetes v1.27.3, Kubernetes v1.29.3 (tested in multiple environments)

Installation method

Operator managed via Helm, Kafka Cluster managed via YAML

Infrastructure

Kind, Bare Metal (tested in multiple environments)

Configuration files and logs

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
    strimzi.io/cluster: kafka
spec:
  maintenanceTimeWindows:
    - "* * 16-18 ? * MON,TUE,WED,THU *"
  kafka:
    version: 3.9.0
    metadataVersion: 3.9-IV0
    authorization:
      type: simple
      superUsers:
        - CN=poc-internal
    jvmOptions: 
      -Xms: 1g
      -Xmx: 1g
    listeners: 
      - tls: false
        name: plain
        port: 9092
        type: cluster-ip
      - tls: false
        name: external
        port: 9093
        type: cluster-ip
        authentication:
          type: scram-sha-512
      - tls: true
        name: externaltls
        port: 9094
        type: nodeport
        authentication:
          type: tls
        configuration:
         # unrelated, used for networking in local kind clusters
          bootstrap:
            alternativeNames:
              - 172.17.0.4
            nodePort: 30956 
          brokers:
            - broker: 0
              nodePort: 30349 
              advertisedPort: 30349
              advertisedHost: 172.17.0.3 
            - broker: 1
              nodePort: 30420
              advertisedPort: 30420
              advertisedHost: 172.17.0.2 
            - broker: 2
              nodePort: 31030
              advertisedPort: 31030
              advertisedHost: 172.17.0.5 
    rack: 
      topologyKey: kubernetes.io/hostname
    config:  
      auto.create.topics.enable: false
      default.replication.factor: 3
      delete.topic.enable: true
      min.insync.replicas: 2
      offsets.topic.replication.factor: 3
  entityOperator:
    topicOperator: {}
    userOperator:
      secretPrefix: kafka-local-user-

kafka nodepool def:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  namespace: kafka
  labels:
    strimzi.io/cluster: kafka
spec: 
  replicas: 1
  resources:
    limits:
      cpu: 2
      memory: 2G
    requests:
      cpu: 1
      memory: 250m
  roles:
  - controller
  storage:
    class: standard
    deleteClaim: true
    size: 1Gi
    type: persistent-claim
  template:
    pod:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: strimzi.io/controller-role
                  operator: In
                  values:
                  - "true"
              topologyKey: kubernetes.io/hostname
---

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  namespace: kafka
  labels:
    strimzi.io/cluster: kafka
spec: 
  replicas: 3
  resources:
    limits:
      cpu: 2
      memory: 2G
    requests:
      cpu: 1
      memory: 250m
  roles:
  - broker
  storage:
    class: standard
    deleteClaim: true
    size: 1Gi
    type: persistent-claim
  template:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: strimzi.io/broker-role
                  operator: In
                  values:
                  - "true"
              topologyKey: kubernetes.io/hostname

Additional context

No response

[scholzj]

That is by design - the PDB has to cover the whole cluster. I assume the documentation you linked is from the old times when there was always only one PodSet for ZooKeeper and one for Kafka - each with its own PDB as Zoo and Kafka had their own PDBs. So it needs to be updated as now we have only one PDB per component (e.g. Kafka) which covers all its StriziPodsets.

So yes, this is a bug, but the bug is the documentation.

[colechiumento-coreweave]

Are there any plans to be able to set PDBs per KafkaNodePool using components in the Strimzi API spec, or should we plan on creating our own?

[scholzj]

I don't think there are any plans and I do not think there are any benefits it would bring. Any rolling anyway happens one node at a time, regardless of how many PDBs there are.

If you need it, you can disable the Strimzi managed PDBs and create your owns, but I do not think it would help with anything.

[colechiumento-coreweave]

It's useful in the context of some other controllers we have, but I recognize that's a bespoke problem that's not relevant for the community at large. We'll move forward with managing our own. Thanks for the quick response! I'm happy to close the issue or leave it open while the documentation triage is addressed.

[scholzj]

I would be quite interested to understand the context where you find it useful. In general, the evictions of the Kafka pods should be gated through the Strimzi Drain Cleaner and Strimzi Cluster Operator tool to ensure the availability of the environment. And they will ultimately serialize the eviction process anyway.

[colechiumento-coreweave]

To be honest, I believe we may have overlooked Strimzi Drain Cleaner when going about our initial implementation. I'll take a stab at implementing it for this use case and see if it solves our need - at first glance, it looks like it will. I'm happy to give more details if that also doesn't fit our use case, but I'm betting it's just a matter of me overlooking this useful tool!

[ppatierno]

@PaulRMellor can you take a look at the fix we would need in the documentation about this, please?

[anael-l]

I also encountered an issue with having only one PDB for the cluster using local storage with one KafkaNodePool for controllers and one for brokers.

If I have three of each (controllers and brokers), and one broker and one controller are located on the same node, the PDB will prevent the drain of one of two pods (due to ALLOWED DISRUPTIONS=1).

As I use node-local storage, the pod cannot be rescheduled to another node.

To resolve this issue, I need to disable the default PDB and create one PDB for each KafkaNodePool.

[scholzj]

I also encountered an issue with having only one PDB for the cluster using local storage with one KafkaNodePool for controllers and one for brokers.

If I have three of each (controllers and brokers), and one broker and one controller are located on the same node, the PDB will prevent the drain of one of two pods (due to ALLOWED DISRUPTIONS=1).

@anael-l Strimzi and Drain Cleaner would never roll both controller and broker at the same time anyway, regardless of the Pod Disruption Budget.