feat: Introduce pluggable inter-flow dispatch policy framework #1167
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
k8s-ci-robot
added
needs-rebase
|
Hi @LukeAVanDrie. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/XXL
LukeAVanDrie
force-pushed
the
feat/flow-control-framework-plusings-inter-flow-dispatch
branch
from
b7e1f13 to
568fdff
Compare
k8s-ci-robot
removed
the
needs-rebase
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
This commit introduces the `InterFlowDispatchPolicy` framework, the third and final major component of the new pluggable flow control system. This framework decouples the logic for selecting which flow's queue to service next (fairness between flows) from the core controller. This completes the two-tier policy model, where the `InterFlowDispatchPolicy` makes the strategic decision about which flow to service, and the `IntraFlowDispatchPolicy` makes the tactical decision of which request to select from within that flow's queue. Key components include: - `framework.InterFlowDispatchPolicy`: The core interface that defines the contract for selecting a queue from a priority band. - `framework.PriorityBandAccessor`: A read-only interface that provides policies with safe access to the state of all queues within a priority level. - A factory and registration system for discovering and instantiating policy plugins by name. - A comprehensive functional conformance test suite to validate the contract for all policy plugins. - A `roundrobin` policy for fair, sequential queue selection. - A `besthead` policy for greedy, utilization-focused queue selection.
LukeAVanDrie
force-pushed
the
feat/flow-control-framework-plusings-inter-flow-dispatch
branch
from
568fdff to
b3d0cab
Compare
|
/lgtm |
k8s-ci-robot
added
the
lgtm
| This contrasts with the `framework.InterFlowDispatchPolicy` (not yet implemented), which is responsible for | ||
| **spatial fairness**: deciding *which flow's queue* gets the next opportunity to dispatch a request. The | ||
| `framework.IntraFlowDispatchPolicy` only operates *after* the inter-flow policy has selected a specific queue. | ||
| This contrasts with the `framework.InterFlowDispatchPolicy`, which is responsible for deciding *which flow's queue* |
There was a problem hiding this comment.
Luke makes a distinction between inter- & intra- flow. So those readme's would be distinct.
There was a problem hiding this comment.
Inter and intra flow plugins have fairly similar contracts (SelectQueue vs SelectItem) and are organized similarly, but I wanted a distinct contributor guide colocated near the actual implementations for each extension point.
I could merge these all into one doc under framework or framework/plugins, but since contributors do not need to understand every extension point to implement one (e.g., adding a new inter flow fairness policy does not require understanding the intra flow policies), I thought the current approach was less overwhelming.
|
/lgtm Exciting to see this come together! |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kfswain, LukeAVanDrie The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
This work tracks #674
This pull request introduces the
InterFlowDispatchPolicyframework, the third and final major component of the new pluggable flow control system outlined in our design proposals. This framework is responsible for inter-flow fairness, deciding which flow's queue gets the next opportunity to dispatch a request to the scheduler.This change completes the two-tier policy engine by adding the strategic scheduling layer that complements the existing
IntraFlowDispatchPolicy(which handles tactical, intra-queue scheduling). This allows operators to configure different fairness and utilization strategies based on their needs.What this PR does
Introduces
framework.InterFlowDispatchPolicy: The core contract that all inter-flow fairness plugins must implement. Its primary method,SelectQueue, inspects aPriorityBandAccessorand returns the chosen queue.Introduces
framework.PriorityBandAccessor: A safe, read-only abstraction that provides policies with a view into the state of all flow queues within a single priority level, without exposing mutable state.Provides a Plugin Factory: A standard registration and instantiation mechanism (
dispatch.MustRegisterPolicy,dispatch.NewPolicyFromName) allows new policies to be discovered and loaded by name, consistent with the existingSafeQueueandIntraFlowDispatchPolicyframeworks.Adds a Conformance Test Suite: A new functional test (
TestInterFlowDispatchPolicyConformance) automatically validates the basic interface contract for all registered policy plugins, ensuring they handle edge cases like nil or empty priority bands correctly.Implements
roundrobinPolicy: A stateful, fair-queuing policy that cycles through non-empty queues sequentially. It is resilient to dynamic changes in the set of available flows and serves as a robust default for ensuring fairness between tenants or models.Implements
bestheadPolicy: A stateless, greedy policy that selects the queue containing the single highest-priority item (based on each queue'sItemComparator). This policy is useful for maximizing resource utilization when strict fairness is not the primary concern. It also validates that it is only comparing items with compatible priority types.