CI jobs occasionally hitting node not ready

[BenTheElder]

My suspicion is an issue with the approach to pod disruption budget on the GKE clusters. I'm less clear on the AWS clusters.

We've seen similar problems on both.

Filing this as an umbrella to note that we're aware this has happened and to collect examples.

/sig k8s-infra
/sig testing

[BenTheElder]

GKE can't perform maintenance because the Pod Disruption Budget allows for 0 Pod evictions. Update the Pod Disruption Budget.

"view details" shows:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy/v1","kind":"PodDisruptionBudget","metadata":{"annotations":{},"labels":{"infra.k8s.io/instance":"gke-prow-build"},"name":"prow-pods","namespace":"test-pods"},"spec":{"maxUnavailable":0,"selector":{"matchLabels":{"created-by-prow":"true"}}}}
  creationTimestamp: "2020-09-11T17:30:42Z"
  generation: 1
  labels:
    infra.k8s.io/instance: gke-prow-build
  name: prow-pods
  namespace: test-pods
  resourceVersion: "2676853668"
  uid: cc419911-f4a7-4ea6-b995-1a037b62fd6a
spec:
  maxUnavailable: 0
  selector:
    matchLabels:
      created-by-prow: "true"
status:
  conditions:
  - lastTransitionTime: "2024-05-21T11:28:52Z"
    message: ""
    observedGeneration: 1
    reason: InsufficientPods
    status: "False"
    type: DisruptionAllowed
  currentHealthy: 102
  desiredHealthy: 0
  disruptionsAllowed: 0
  expectedPods: 0
  observedGeneration: 1

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xmudrii]

This is still the issue occasionally AFAIK
/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale