PodTopologySpread DoNotSchedule-to-ScheduleAnyway fallback mode

[sanposhiho]

Enhancement Description

• One-line enhancement description (can be used as a release note): PodTopologySpread DoNotSchedule-to-ScheduleAnyway fallback mode based on new Pod conditions.
• Kubernetes Enhancement Proposal: KEP-3990: PodTopologySpread DoNotSchedule-to-ScheduleAnyway fallback mode #4150
• Discussion Link: PodTopologySpread DoNotSchedule-to-ScheduleAnyway fallback mode kubernetes#105977
• Primary contact (assignee): @sanposhiho
• Responsible SIGs: sig-scheduling sig-autoscaling
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): v1.30
  • Beta release target (x.y): v1.31
  • Stable release target (x.y): v1.33
• Alpha
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/sig scheduling
/assign

[sanposhiho]

/sig autoscaling

[knelasevero]

@sanposhiho do you have ETA for the KEP to be up? Any way we could help?

/cc @a7i

[sanposhiho]

I recently started a draft locally, but either way, as written at the top, we - sig-scheduling - don't plan to have this enhancement in v1.28 and gonna be in v1.29 at the earliest. You can help us improve the design once I've created the PR for KEP.

[sanposhiho]

@ahg-g @alculquicondor @Huang-Wei
Can we target this towards v1.29?
I'll write the KEP + take the implementation part, but looking for main reviewer(s) who can work on in the next release cycle.

[alculquicondor]

I'm in favor, but make sure you also have a reviewer from sig-autoscaling

[sanposhiho]

Thanks @alculquicondor.

@gjtempleton @mwielgus
Hello. Could either of you help us as a reviewer on this enhancement?
The cluster autoscaler is relevant to this enhancement, like CA tells the scheduler that it cannot bring any new Nodes in and the scheduler decides to fallback. (See kubernetes/kubernetes#105977 for more details)

[MaciekPytel]

Hi, I've been thinking about this for a long time and as a top-level owner of Cluster Autoscaler I'd be happy to get involved.

That being said - I'd like to start discussion with a possibly very controversial question: is scheduling the best layer to address zonal spreading?

---

Problem statement

Today I would recommend against anyone using PodTopologySpreading on a zonal topology in a cloud environment:

• When using whenUnsatisfiable: DoNotSchedule topology spreading introduces two major problems:
  • In the event of provider being out of capacity (stock-out) pods would not be able to schedule at all - they can't schedule in the zone with smallest number of replicas, because there is no capacity there. And they can't schedule in other zones, because it would violate the topology spreading constraints.
  • During zone failure, topology spreading will prevent pods from affected nodes from falling back to other zones unless every single node in an affected zone is removed. In my experience partial failures are more common than complete failure where every single node is gone.
• The alternative is to set whenUnsatisfiable: ScheduleAnyway, but in an autoscaled cluster this is unlikely to achieve much:
  • Autoscaling only looks at Filters and ignores Scores, so this will not affect in which zones new nodes are provided.
  • Autoscaling is only triggered by unschedulable pods, so in practice the scheduler will put the pods wherever there is free capacity and new nodes will not be provided at all. Unless cluster utilization is quite low, chances are that "wherever there is free capacity" doesn't result in pods being spread out between zones.

My understanding is that the goal of this proposal is to address the problems with whenUnsatisfiable: DoNotSchedule spreading that I described above. Am I correct?

---

Challenges

I'd love to see this solved, but I don't have any good ideas on how to do it. A timeout-based approach seems very fragile as node provisioning times vary widely between clouds and even within a single cloud they may be very different based on the type of hardware being used (ex. nodes with GPU often take more time to startup and initialize). And what happens when preferred instance types are unavailable (ex. stockout) and autoscaler needs to fallback to a different type - that would add anywhere between a few seconds to 15+ minutes of extra latency depending on the cloud and the exact reason for node creation failure.
We can say that it's up to the user to set the right timeout, but how many users will be able to set the right timeout in practice? I've been working on Cluster Autoscaler in OSS and GKE for >6 years and I have no idea what a good timeout value would be on GCP - much less in any other cloud.

An alternative would be some sort of communication mechanism between Autoscaler and Scheduler, but for that I think we should have an idea how to support this in Autoscaler: how do we make it aware that it should update the pod at all? Today Autoscaler just imports scheduler code and runs PreFilters/Filters, without any understanding of what they actually check. How would it know that a pod is unschedulable because of TopologySpreading and not some other constraint? - and I mean the question in the sense of how much of Autoscaler would we have to refactor, not just a high-level conceptual answer.

---

Alternatives that may be worth discussing

Finally, even if we solve all of those issues - scheduling pods in a way that respects topology spreading constraint still wouldn't guarantee equal spreading between zones. If one zone is temporarily stocked-out, the scheduler will not be able to restore the spreading after the instances become available again.

That brings me to the controversial question: wouldn't it be better to solve this at Deployment/ReplicaSet/StatefulSet/etc level instead? Any such controller could target pod to specific zones (ex. by setting nodeSelector for different zones) and it could continuously reconcile the number of pods in each zone. This would also address the problem of timeout - we could fallback to a different zone after a relatively short timeout, knowing that we can always restore the spreading as soon as the capacity becomes available.

This is the approach taken by https://github.com/kubernetes/autoscaler/blob/master/balancer/proposals/balancer.md proposal. I'm not sure balancer is the best way to implement this either - I'm not as familiar with whatever challenges this approach may be facing. But I think it would be good to start discussion from agreeing on what problems we're trying to solve and evaluating which component could best solve them, before jumping into any particular implementation.

[sanposhiho]

Thanks @MaciekPytel for getting involved!

I believe we should continue to discuss a detailed design in KEP PR instead of here, but let me roughly answer your questions.

In the issue, we're considering having new whenUnsatisfiable, DoNotSchedule_BestEffort and PreemptOrSchedule. (names can be officially decided in KEP)
DoNotSchedule_BestEffort is the one CA will be related.
When DoNotSchedule_BestEffort, the scheduler keeps trying to schedule the Pod while assuming TopologySpread is DoNotSchedule. But, when the scheduler somehow knows that CA cannot create Node for this Pod, then fallback to ScheduleAnyway so that the Pod will get scheduled without being blocked by PodTopologySpread.

Then, next it comes to how to know that CA cannot create Node for the Pod. We discussed two options:

• waits a certain period of time (it's like a timeout). If new Node isn't created for a certain period, then fallback to ScheduleAnyway. (ref)
• create new Pod condition TriggeredScaleUp and CA set it false if it cannot create new Node for the Pod. (ref)

In the issue, we kind of concluded to prefer option 2, new Pod condition. So, CA's responsibility is to add false to new Pod condition TriggeredScaleUp. The reason was, as you said, it's difficult to set an appropriate timeout. (ref)

It's the current status of our discussion, and I'm going to create the KEP based on this. So, answering your questions

My understanding is that the goal of this proposal is to address the problems with whenUnsatisfiable: DoNotSchedule spreading that I described above. Am I correct?

Yes. Exactly correct.

An alternative would be some sort of communication mechanism between Autoscaler and Scheduler

And yes it's exactly what we concluded (at least in the issue).

How would it know that a pod is unschedulable because of TopologySpreading and not some other constraint?

I believe, to make it simple, CA doesn't need to do something special for TopologySpread.
It only needs to set the condition when it cannot create a node for unschedulable Pod, regardless of the reason. And when the scheduler sees that condition, fallback to SchedulingAnyway in anyways. If unschedulable isn't caused by TopologySpread, the Pod should still be unschedulable even after the fallback.

If one zone is temporarily stocked-out, the scheduler will not be able to restore the spreading after the instances become available again.

Such "rescheduling", "rebalancing" is the responsibility of descheduler, not the scheduler. So, we don't need to concern much about the rebalancing in the scheduler.

[MaciekPytel]

I believe we should continue to discuss a detailed design in KEP PR instead of here

Sounds good to me. Please tag me on PR and also please feel free to ping me on slack if you want to discuss any CA-releated parts.

[sanposhiho]

@alculquicondor @MaciekPytel Can we target this enhancement to v1.29? (based on your bandwidth?)