KEP-5328: Node Capabilities

[pravk03]

• One-line PR description: Add the initial KEP for KEP 5328: Node Capabilities

• Issue link: Node Capabilities #5328

• Other comments:

[k8s-ci-robot]

Welcome @pravk03!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @pravk03. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[wojtek-t]

@dom4ha @sanposhiho @macsko - FYI

[pravk03]

/cc @tallclair @yujuhong

[sanposhiho]

/sig scheduling

[SergeyKanzhelev]

Can we have any examples listed that will justify this. Right now the KEP suggests to use it for FG-related capabilities, while not giving a good examples where it would be non-FG related.

The guaranteedQOSPodCPUResize example used in the KEP isn't purely a feature gate; it's a logical capability derived from a combination of feature gates and the Kubelet's cpuManagerPolicy configuration.

While this is still in early stages, this recent discussion about making the pod requirement for exclusive resources more explicit also indicates a need for non-FG capabilities. The API field itself should be forward-facing enough to support such potential use-cases ?.

Those are all examples of FG-related capabilities. Not the generic long-term capabilities.

[tallclair]

It seems like most of the concerns with this are around the specific capabilities being added, but this KEP doesn't actually propose adding any capabilities. The examples given are hypothetical examples based on features currently in development, but no new features will be able to depend on capabilities until it goes to beta. This creates a bit of a chicken-and-egg situation, where it's hard to point to exactly how capabilities will be used until we have users lined up, but we can't line up users yet.

[SergeyKanzhelev]

It seems like most of the concerns with this are around the specific capabilities being added, but this KEP doesn't actually propose adding any capabilities. The examples given are hypothetical examples based on features currently in development, but no new features will be able to depend on capabilities until it goes to beta. This creates a bit of a chicken-and-egg situation, where it's hard to point to exactly how capabilities will be used until we have users lined up, but we can't line up users yet.

we kind of need to know what will be expected use cases. Maybe past examples or hypothetical examples thought thru end-to-end. Right now this KEP is limited to just set of name/value pairs and a scenario of FG discoverability. But already we are thinking there MAY be need to support capabilities for node selection, ability to declare tolerations for capabilities, ability to have node-restricted capabilities. Knowing the scope would help to understand if API proposed is needed (among alternatives if the set of use cases is limited) and if needed, what shape should it have.

[pravk03]

Maybe past examples or hypothetical examples thought thru end-to-end

RuntimeClass was intended as a past example used to illustrate non-FG related runtime capabilities in the earlier version of the proposal. I agree that it had some missing details and thanks for highlighted them in your comment.

3. Runtime handlers as a list of handlers is also not a good fit. Default handler runc is not specified in pod spec. So it will not be used by scheduler and by definition must not be added to capabilities. Non-default handlers may need more details on what it is. And names list may not fit into the value length limits. Special object representing the runtime is a better choice here.

I have tried to address these the Case Study section.

[tallclair]

Maybe past examples or hypothetical examples thought thru end-to-end. Right now this KEP is limited to just set of name/value pairs and a scenario of FG discoverability.

I feel like we've discussed these options in depth already. Yes, these are all somewhat hypothetical because we've had to work around them in other ways. I'm sure we can dig up more examples from past KEPs, but is that necessary?

Capabilities that are not limited to just feature gates:

• swap enabled
• static CPU / memory manager enabled
• user namespace support

Feature gate capabilities:

• pod-level resources
• TLS for gRPC probes
• in-place resize (+IPPR for pod-level resources, IPPR for static CPU assignment, etc)

But already we are thinking there MAY be need to support capabilities for node selection, ability to declare tolerations for capabilities,

Not sure what node selection means, but we've explicitly said tolerations are out of scope.

ability to have node-restricted capabilities.

Where did this come in? Capabilities are just added by the node, so I'm not sure what this would even mean.

[pravk03]

We discussed this KEP today and decided to re-consider this for 1.35 release cycle. The primary reason is to get input fromsig-arch on using this capability-based framework as a general strategy for managing version skew.

Few more things discussed and that could be refined in the proposal:

1. Evaluate the strategy for managing capabilities with bounded lifetime. Define a clear lifecycle and deprecation path for capabilities tied to features that graduate to GA.
2. We would need a better use-case to consider long-term capabilities in-scope. It can be considered a future enhancement once a clear use case arises.
3. Further explore SemVer based filtering in Node Selectors as a potential alternative.

cc @tallclair @SergeyKanzhelev @dchen1107 @yujuhong

[pravk03]

This proposal was discussed in the SIG-Arch community meeting on June 26th (recording), It was generally seen as a beneficial strategy for managing version skew. The key takeaways and action items from the discussion are as follows:

1. The KEP can be scoped for temporary capabilities to solve version skew use cases. The configuration skew (like operating systems etc.) use-cases are less clear at the moment and may need explicit fields (like pod.spec.os).
2. Make the temporary nature of capabilities more obvious. This can prevent other components (webhooks etc.) from depending on this API. Obfuscating the fields can be explored if required.
3. A common library should be used to encapsulate the logic for inferring the capability requirements. This will ensure consistency between all consuming components, such as the kube-scheduler and admission controllers.
4. Autoscaler support: The "client-side" problem (determining a pod's requirements) can be solved with the proposed shared library. The problem of how to scale up nodes (specially scaling up from 0 nodes) is still a challenge and needs further exploration. This is already added in the future consideration section of the KEP.

I will incorporate this feedback into the KEP and reach out when its ready for review.

[pravk03]

I've updated the KEP based on the SIG Architecture feedback (#5347 (comment)). The new version focuses more on capabilities tied to the feature lifecycle and expands on the deprecation strategy.

@tallclair @SergeyKanzhelev @haircommander @wojtek-t PTAL when you get a chance.

[SergeyKanzhelev]

I'd suggest to add somebody from sig scheduling as approver here

[SergeyKanzhelev]

Suggested change

	5. Enable kubelet admission plugin to check the Pod is compatible with the Node's features

[SergeyKanzhelev]

is it the same? I thought that DRA is unique as DRA is not a part of a Node Status so it is harder to add those to the templates

[SergeyKanzhelev]

Also capabilities must be calculated BEFORE Pods admission. Otherwise pod admission will fail on node restart

[SergeyKanzhelev]

This requires clarification. Maybe with the specific versions on how to calculate supported version skews. Does this statement suggests that the capability will be removed only after GA + 3 versions? And after this, the logic is removed from both - control plane and kubelet at the same time?

[SergeyKanzhelev]

clarificaqtion is needed for the when it is removed from the control plane mostly

[SergeyKanzhelev]

this will delay adoption. Perhaps it can be solved in alpha