Update lora affinity to be a scorer. #1121

rlakhtakia · 2025-07-08T17:18:56Z

Changes: Added new scorer for v2 scheduler.

Benchmark details:

Coefficient of Determination (R^2) between before and after runs:
avg_normalized_time_per_output_token_ms R^2 = 0.9887
p90_per_token_latency_ms R^2 = 0.9691
p90_latency_ms R^2 = 0.9633
num_prompts_attempted R^2 = 1.0000
num_prompts_succeeded R^2 = 0.9951

netlify · 2025-07-08T17:19:02Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`e3f2eb7`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/687ff1fb4f95700008dad953
😎 Deploy Preview	https://deploy-preview-1121--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-07-08T17:19:03Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rlakhtakia
Once this PR has been reviewed and has the lgtm label, please assign jeffwan for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-07-08T17:19:06Z

Hi @rlakhtakia. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go

nirrozenbaum · 2025-07-08T17:27:58Z

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go

+
+// LoraAffinityScorer scores list of candidate pods based on KV cache utilization.
+type LoraAffinityScorer struct {
+	name plugins.TypedName


pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go

nirrozenbaum · 2025-07-08T17:31:11Z

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go

+		if active {
+			scores[pod] = 1
+		} else if len(pod.GetMetrics().ActiveModels)+len(pod.GetMetrics().WaitingModels) < pod.GetMetrics().MaxActiveModels {
+			scores[pod] = 0.8


how did you select these numbers? is it based on some tests? performance comparison? intuition?

+1; how does this compare to the approach we have in the decision tree plugin.

@kaushikmitr pls take a look as well

nirrozenbaum · 2025-07-08T17:36:48Z

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity_test.go

+		name              string
+		request           *types.LLMRequest
+		pods              []types.Pod
+		expectedScoresPod map[int]float64 // Map of pod index to expected score


can you use a map[NamespacedName]float64 (map from pod identifier to expected score)?

That's a good suggestion. I wonder if map[string]float64 might be better. We can easily get the pod's name from the target pod directly, which would avoid needing to declare NamespacedName repeatedly in the test cases.

In Score function we get back map[Pod]float64, where the Pod representation includes NamespacedName, Address and Labels.
I assume we can either use NamespacedName.String() or Address. both will do the work.

ahg-g · 2025-07-08T17:49:32Z

/ok-to-test

nirrozenbaum · 2025-07-09T08:51:08Z

cc @mayabar

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go

ahg-g · 2025-07-09T15:35:50Z

This is great, this will help us move off of the decision tree plugin

kaushikmitr · 2025-07-09T22:58:22Z

In the current Lora affinity algorithm we have loraaffinitythreshold = 0.99. We use this to prefer a pod that already has the Lora loaded over a pod that does not have the Lora loaded but has room with 99% probability. This ensures that we maintain Lora affinity while spreading it out at high Lora traffic. I am guessing scoring can conceptually achieve that but it seems we are always picking the highest score.

elevran · 2025-07-17T13:38:36Z

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go

+var _ framework.Scorer = &LoraAffinityScorer{}
+
+// LoraAffinityScorerFactory defines the factory function for LoraAffinityScorer.
+func LoraAffinityScorerFactory(name string, _ json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {


Q: perhaps allow setting DefaultLoraAffinityScorerWeight via configuration parameters in the config file?
We're trying to avoid using environment variables (e.g., added in runner.go).

we are planning to have GIE default weight for scorers. if no weight specified in the config the weight will default to 1 automatically by the config api code.
this will be pushed in a separate, not in scope of this PR of course.

liu-cong · 2025-07-17T16:39:10Z

Do you have some benchmark numbers to share?

rlakhtakia · 2025-07-22T18:16:15Z

Benchmark details:

Coefficient of Determination (R^2) between before and after runs:
avg_normalized_time_per_output_token_ms R^2 = 0.9887
p90_per_token_latency_ms R^2 = 0.9691
p90_latency_ms R^2 = 0.9633
num_prompts_attempted R^2 = 1.0000
num_prompts_succeeded R^2 = 0.9951

ahg-g · 2025-07-23T15:58:58Z

Thanks @rlakhtakia; what exactly is the experiment setup? Can you please document that?

liu-cong · 2025-07-23T20:02:23Z

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go

+		// Determine the model server's suitability score based on adapter load status and capacity.
+		switch {
+		// Ideal: The adapter is already active on this model server.
+		case active:


The ideal case should be:

The requested lora is active AND there are no waiting loras (otherwise the waiting ones will be served first given the request FIFO queue).

liu-cong · 2025-07-23T20:08:17Z

@rlakhtakia

Thanks for the benchmark results, it looks very similar to the filter based lora affinity, great!

Can you provide more details on the benchmark?

The exact benchmark setup (I believe you followed the regression test guide, if so, pls be explicit). Also clarify the baseline EPP version.
Add a graph about throughput
The graphs show the aggregated results for all LoRAs? If so, can you also provide the specific graphs for a high traffic LoRA and low traffic LoRA?

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 8, 2025

k8s-ci-robot requested review from liu-cong and robscott July 8, 2025 17:19

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 8, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 8, 2025

nirrozenbaum reviewed Jul 8, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Jul 8, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go Outdated Show resolved Hide resolved

nirrozenbaum reviewed Jul 8, 2025

View reviewed changes

ahg-g reviewed Jul 9, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go Outdated Show resolved Hide resolved

ahg-g reviewed Jul 9, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go Outdated Show resolved Hide resolved

rlakhtakia force-pushed the scorer branch from 9c6a527 to 1bd67a6 Compare July 10, 2025 04:43

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 10, 2025

rlakhtakia force-pushed the scorer branch 3 times, most recently from f8f767d to a7214d7 Compare July 10, 2025 16:23

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 15, 2025

kfswain mentioned this pull request Jul 16, 2025

Feature: LoRA based filtering and scoring llm-d/llm-d-inference-scheduler#10

Open

elevran reviewed Jul 17, 2025

View reviewed changes

Update lora affinity to be a scorer.

e3f2eb7

rlakhtakia force-pushed the scorer branch from a7214d7 to e3f2eb7 Compare July 22, 2025 20:18

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 22, 2025

liu-cong reviewed Jul 23, 2025

View reviewed changes

Update lora affinity to be a scorer. #1121

Are you sure you want to change the base?

Update lora affinity to be a scorer. #1121

Conversation

rlakhtakia commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Jul 8, 2025

Uh oh!

k8s-ci-robot commented Jul 8, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahg-g commented Jul 8, 2025

Uh oh!

nirrozenbaum commented Jul 9, 2025

Uh oh!

Uh oh!

Uh oh!

ahg-g commented Jul 9, 2025

Uh oh!

kaushikmitr commented Jul 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liu-cong commented Jul 17, 2025

Uh oh!

rlakhtakia commented Jul 22, 2025

Uh oh!

ahg-g commented Jul 23, 2025

Uh oh!

liu-cong Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liu-cong commented Jul 23, 2025

Uh oh!

Uh oh!

rlakhtakia commented Jul 8, 2025 •

edited

Loading

netlify bot commented Jul 8, 2025 •

edited

Loading

liu-cong Jul 23, 2025 •

edited

Loading