Skip to content

Commit 7f25e5f

Browse files
mortentmortent
committed
Update design with additional NodeSelection option on ResourceSlice
1 parent cd2aca2 commit 7f25e5f

File tree

1 file changed

+184
-162
lines changed
  • keps/sig-node/4815-dra-partitionable-devices

1 file changed

+184
-162
lines changed

‎keps/sig-node/4815-dra-partitionable-devices/README.md‎

Lines changed: 184 additions & 162 deletions
Original file line numberDiff line numberDiff line change
@@ -342,13 +342,15 @@ The basic idea is the following:
342342
to allocate it. This essentially removes that capacity from any referenced
343343
devices, rendering them unallocatable on their own.
344344

345-
1. The `NodeName` and `NodeSelector` fields describes the node or set of nodes
346-
where the device is available. This is similar to the `NodeName`, `NodeSelector`,
347-
and `AllNodes` properties in the `ResourceSlice` spec, but this allows for
348-
associating individual devices to node(s). That makes it possible to describe
349-
multi-host devices using the ResourceSlice API. The `NodeName` and `NodeSelector`
350-
fields are mutually exclusive and neither can be specified if the `Spec.NodeName` or
351-
`Spec.NodeSelector` fields are specified on the `ResourceSlice`.
345+
1. The `NodeSelectionDelegated` field on the `ResourceSlice` spec specifies
346+
whether the availability of the resources from nodes is defined on the
347+
`ResourceSlice` or on each individual device. If it is specified on the
348+
`ResourceSlice`, that choice applies to all devices in the slice.
349+
Three fields on the `CompositeDevice` object defines availability of the device
350+
from nodes. These fields are `NodeName`, `NodeSelector`, and `AllNodes` are
351+
mutually exclusive and mirrors the same fields on the `ResourceSlice` spec.
352+
These fields can only be set if `NodeSelectionDelegated` on the `ResourceSlice`
353+
spec is set to `true`.
352354

353355
With these additions in place, the scheduler has everything it needs to support
354356
the dynamic allocation of both full devices, their (possibly overlapping)
@@ -365,187 +367,207 @@ The exact set of proposed API changes can be seen below:
365367
```go
366368
// ResourceSliceSpec contains the information published by the driver in one ResourceSlice.
367369
type ResourceSliceSpec struct {
368-
...
369-
370-
// DeviceMixins represents a list of device mixins, i.e. a collection of
371-
// shared attributes and capacities that an actual device can "include"
372-
// to extend the set of attributes and capacities it already defines.
373-
//
374-
// The main purposes of these mixins is to reduce the memory footprint
375-
// of devices since they can reference the mixins provided here rather
376-
// than duplicate them.
377-
//
378-
// The total number of mixins, basic devices, and composite devices must be
379-
// less than 128.
380-
//
381-
// +optional
382-
// +listType=atomic
383-
DeviceMixins []DeviceMixin `json:"deviceMixins,omitempty"`
370+
...
371+
372+
// NodeSelectionDelegated defines whether the access from nodes to
373+
// resources in the pool is set on the ResourceSlice level or on each
374+
// device. If it is set to true, every device defined the ResourceSlice
375+
// must specify this individually.
376+
//
377+
// Exactly one of NodeName, NodeSelector, AllNodes, and NodeSelectionDelegated
378+
// must be set.
379+
//
380+
// +optional
381+
// +oneOf=NodeSelection
382+
NodeSelectionDelegated bool
383+
384+
// DeviceMixins represents a list of device mixins, i.e. a collection of
385+
// shared attributes and capacities that an actual device can "include"
386+
// to extend the set of attributes and capacities it already defines.
387+
//
388+
// The main purposes of these mixins is to reduce the memory footprint
389+
// of devices since they can reference the mixins provided here rather
390+
// than duplicate them.
391+
//
392+
// The total number of mixins, basic devices, and composite devices must be
393+
// less than 128.
394+
//
395+
// +optional
396+
// +listType=atomic
397+
DeviceMixins []DeviceMixin `json:"deviceMixins,omitempty"`
384398
}
385399

386400
// DeviceMixin defines a specific device mixin for each device type.
387401
// Besides the name, exactly one field must be set.
388402
type DeviceMixin struct {
389-
// Name is a unique identifier among all mixins managed by the driver
390-
// in the pool. It must be a DNS label.
391-
//
392-
// +required
393-
Name string `json:"name"`
394-
395-
// Composite defines a mixin usable by a composite device.
396-
//
397-
// +optional
398-
// +oneOf=deviceMixinType
399-
Composite *CompositeDeviceMixin `json:"composite,omitempty"`
403+
// Name is a unique identifier among all mixins managed by the driver
404+
// in the pool. It must be a DNS label.
405+
//
406+
// +required
407+
Name string `json:"name"`
408+
409+
// Composite defines a mixin usable by a composite device.
410+
//
411+
// +optional
412+
// +oneOf=deviceMixinType
413+
Composite *CompositeDeviceMixin `json:"composite,omitempty"`
400414
}
401415

402416
// CompositeDeviceMixin defines a mixin that a composite device can include.
403417
type CompositeDeviceMixin struct {
404-
// Attributes defines the set of attributes for this mixin.
405-
// The name of each attribute must be unique in that set.
406-
//
407-
// To ensure this uniqueness, attributes defined by the vendor
408-
// must be listed without the driver name as domain prefix in
409-
// their name. All others must be listed with their domain prefix.
410-
//
411-
// Conflicting attributes from those provided via other mixins are
412-
// overwritten by the ones provided here.
413-
//
414-
// The maximum number of attributes and capacities combined is 32.
415-
//
416-
// +optional
417-
Attributes map[QualifiedName]DeviceAttribute `json:"attributes,omitempty"`
418-
419-
// Capacity defines the set of capacities for this mixin.
420-
// The name of each capacity must be unique in that set.
421-
//
422-
// To ensure this uniqueness, capacities defined by the vendor
423-
// must be listed without the driver name as domain prefix in
424-
// their name. All others must be listed with their domain prefix.
425-
//
426-
// Conflicting capacities from those provided via other mixins are
427-
// overwritten by the ones provided here.
428-
//
429-
// The maximum number of attributes and capacities combined is 32.
430-
//
431-
// +optional
432-
Capacity map[QualifiedName]DeviceCapacity `json:"capacity,omitempty"`
418+
// Attributes defines the set of attributes for this mixin.
419+
// The name of each attribute must be unique in that set.
420+
//
421+
// To ensure this uniqueness, attributes defined by the vendor
422+
// must be listed without the driver name as domain prefix in
423+
// their name. All others must be listed with their domain prefix.
424+
//
425+
// Conflicting attributes from those provided via other mixins are
426+
// overwritten by the ones provided here.
427+
//
428+
// The maximum number of attributes and capacities combined is 32.
429+
//
430+
// +optional
431+
Attributes map[QualifiedName]DeviceAttribute `json:"attributes,omitempty"`
432+
433+
// Capacity defines the set of capacities for this mixin.
434+
// The name of each capacity must be unique in that set.
435+
//
436+
// To ensure this uniqueness, capacities defined by the vendor
437+
// must be listed without the driver name as domain prefix in
438+
// their name. All others must be listed with their domain prefix.
439+
//
440+
// Conflicting capacities from those provided via other mixins are
441+
// overwritten by the ones provided here.
442+
//
443+
// The maximum number of attributes and capacities combined is 32.
444+
//
445+
// +optional
446+
Capacity map[QualifiedName]DeviceCapacity `json:"capacity,omitempty"`
433447
}
434448

435449
// Device represents one individual hardware instance that can be selected based
436450
// on its attributes. Besides the name, exactly one field must be set.
437451
// +k8s:deepcopy-gen=true
438452
type Device struct {
439-
// Name is unique identifier among all devices managed by
440-
// the driver in the pool. It must be a DNS label.
441-
//
442-
// +required
443-
Name string `json:"name"`
444-
445-
// Basic defines one device instance.
446-
//
447-
// +optional
448-
// +oneOf=deviceType
449-
Basic *BasicDevice
450-
451-
// Composite defines one composite device instance.
452-
//
453-
// +optional
454-
// +oneOf=deviceType
455-
Composite *CompositeDevice `json:"composite,omitempty"`
453+
// Name is unique identifier among all devices managed by
454+
// the driver in the pool. It must be a DNS label.
455+
//
456+
// +required
457+
Name string `json:"name"`
458+
459+
// Basic defines one device instance.
460+
//
461+
// +optional
462+
// +oneOf=deviceType
463+
Basic *BasicDevice
464+
465+
// Composite defines one composite device instance.
466+
//
467+
// +optional
468+
// +oneOf=deviceType
469+
Composite *CompositeDevice `json:"composite,omitempty"`
456470
}
457471

458472
// CompositeDevice defines one device instance.
459473
type CompositeDevice struct {
460-
// Includes defines the set of device mixins that this device includes.
461-
//
462-
// The propertes of each included mixin are applied to this device in
463-
// order. Conflicting properties from multiple mixins are taken from the
464-
// last mixin listed that contains them.
465-
//
466-
// The maximum number of mixins that can be included is 8.
467-
//
468-
// +optional
469-
Includes []DeviceMixinRef `json:"includes,omitempty"`
470-
471-
// ConsumesCapacityFrom defines the set of devices where any capacity
472-
// consumed by this device should be pulled from. This applies recursively.
473-
// In cases where the device names itself as its source, the recursion is
474-
// halted.
475-
//
476-
// Conflicting capacities from multiple devices are taken from the
477-
// last device listed that contains them.
478-
//
479-
// The maximum number of devices that can be referenced is 8.
480-
//
481-
// +optional
482-
ConsumesCapacityFrom []DeviceRef `json:"consumesCapacityFrom,omitempty"`
483-
484-
// NodeName identifies the node where the device is available.
485-
//
486-
// Must only be set if Spec.AllNodes is set.
487-
// Only one or none of NodeName and NodeSelector must be set.
488-
//
489-
// +optional
490-
// +oneOf=DeviceNodeSelection
491-
NodeName string
492-
493-
// NodeSelector defines the nodes where the device is available.
494-
//
495-
// Must use exactly one term.
496-
//
497-
// Must only be set if Spec.AllNodes is set.
498-
// Only one or none of NodeName and NodeSelector must be set.
499-
//
500-
// +optional
501-
// +oneOf=DeviceNodeSelection
502-
NodeSelector *core.NodeSelector
503-
504-
// Attributes defines the set of attributes for this device.
505-
// The name of each attribute must be unique in that set.
506-
//
507-
// To ensure this uniqueness, attributes defined by the vendor
508-
// must be listed without the driver name as domain prefix in
509-
// their name. All others must be listed with their domain prefix.
510-
//
511-
// Conflicting attributes from those provided via mixins are
512-
// overwritten by the ones provided here.
513-
//
514-
// The maximum number of attributes and capacities combined is 32.
515-
//
516-
// +optional
517-
Attributes map[QualifiedName]DeviceAttribute `json:"attributes,omitempty"`
518-
519-
// Capacity defines the set of capacities for this device.
520-
// The name of each capacity must be unique in that set.
521-
//
522-
// To ensure this uniqueness, capacities defined by the vendor
523-
// must be listed without the driver name as domain prefix in
524-
// their name. All others must be listed with their domain prefix.
525-
//
526-
// Conflicting capacities from those provided via mixins are
527-
// overwritten by the ones provided here.
528-
//
529-
// The maximum number of attributes and capacities combined is 32.
530-
//
531-
// +optional
532-
Capacity map[QualifiedName]DeviceCapacity `json:"capacity,omitempty"`
474+
// Includes defines the set of device mixins that this device includes.
475+
//
476+
// The propertes of each included mixin are applied to this device in
477+
// order. Conflicting properties from multiple mixins are taken from the
478+
// last mixin listed that contains them.
479+
//
480+
// The maximum number of mixins that can be included is 8.
481+
//
482+
// +optional
483+
Includes []DeviceMixinRef `json:"includes,omitempty"`
484+
485+
// ConsumesCapacityFrom defines the set of devices where any capacity
486+
// consumed by this device should be pulled from. This applies recursively.
487+
// In cases where the device names itself as its source, the recursion is
488+
// halted.
489+
//
490+
// Conflicting capacities from multiple devices are taken from the
491+
// last device listed that contains them.
492+
//
493+
// The maximum number of devices that can be referenced is 8.
494+
//
495+
// +optional
496+
ConsumesCapacityFrom []DeviceRef `json:"consumesCapacityFrom,omitempty"`
497+
498+
// NodeName identifies the node where the device is available.
499+
//
500+
// Must only be set if Spec.AllNodes is set.
501+
// Only one or none of NodeName and NodeSelector must be set.
502+
//
503+
// +optional
504+
// +oneOf=DeviceNodeSelection
505+
NodeName string
506+
507+
// NodeSelector defines the nodes where the device is available.
508+
//
509+
// Must use exactly one term.
510+
//
511+
// Must only be set if Spec.AllNodes is set.
512+
// Only one or none of NodeName and NodeSelector must be set.
513+
//
514+
// +optional
515+
// +oneOf=DeviceNodeSelection
516+
NodeSelector *core.NodeSelector
517+
518+
// AllNodes indicates that all nodes have access to the device.
519+
//
520+
// Exactly one of NodeName, NodeSelector and AllNodes must be set.
521+
//
522+
// +optional
523+
// +oneOf=DeviceNodeSelection
524+
AllNodes bool
525+
526+
// Attributes defines the set of attributes for this device.
527+
// The name of each attribute must be unique in that set.
528+
//
529+
// To ensure this uniqueness, attributes defined by the vendor
530+
// must be listed without the driver name as domain prefix in
531+
// their name. All others must be listed with their domain prefix.
532+
//
533+
// Conflicting attributes from those provided via mixins are
534+
// overwritten by the ones provided here.
535+
//
536+
// The maximum number of attributes and capacities combined is 32.
537+
//
538+
// +optional
539+
Attributes map[QualifiedName]DeviceAttribute `json:"attributes,omitempty"`
540+
541+
// Capacity defines the set of capacities for this device.
542+
// The name of each capacity must be unique in that set.
543+
//
544+
// To ensure this uniqueness, capacities defined by the vendor
545+
// must be listed without the driver name as domain prefix in
546+
// their name. All others must be listed with their domain prefix.
547+
//
548+
// Conflicting capacities from those provided via mixins are
549+
// overwritten by the ones provided here.
550+
//
551+
// The maximum number of attributes and capacities combined is 32.
552+
//
553+
// +optional
554+
Capacity map[QualifiedName]DeviceCapacity `json:"capacity,omitempty"`
533555
}
534556

535557
// DeviceMixinRef defines a reference to a device mixin.
536558
type DeviceMixinRef struct {
537-
// Name refers to the name of a device mixin in the pool.
538-
//
539-
// +required
540-
Name string `json:"name"`
559+
// Name refers to the name of a device mixin in the pool.
560+
//
561+
// +required
562+
Name string `json:"name"`
541563
}
542564

543565
// DeviceRef defines a reference to a device.
544566
type DeviceRef struct {
545-
// Name refers to the name of a device in the pool.
546-
//
547-
// +required
548-
Name string `json:"name"`
567+
// Name refers to the name of a device in the pool.
568+
//
569+
// +required
570+
Name string `json:"name"`
549571
}
550572
```
551573

@@ -758,7 +780,7 @@ kind: ResourceSlice
758780
apiVersion: resource.k8s.io/v1beta1
759781
...
760782
spec:
761-
allNodes: true
783+
nodeSelectionDelegated: true
762784
pool:
763785
...
764786
driver: tpu.dra.example.com

0 commit comments

Comments
 (0)