Skip to content

Record EPP NormalizedTimePerOutputToken metric on streaming mode #1706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 27, 2025
Merged

Record EPP NormalizedTimePerOutputToken metric on streaming mode #1706

merged 1 commit into from
Oct 27, 2025
+24 −4

Conversation

@dharaneeshvrd
Copy link
Contributor

What type of PR is this?

/kind bug
/kind failing-test

What this PR does / why we need it:
Add code changes to record NormalizedTimePerOutputToken metric in EPP, which is expected in e2e epp test.

Which issue(s) this PR fixes:

Fixes #939

Does this PR introduce a user-facing change?:

NONE

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. labels Oct 13, 2025
@netlify
Copy link

netlify bot commented Oct 13, 2025 •
edited
Loading

✅ Deploy Preview for gateway-api-inference-extension ready!

Name Link
🔨 Latest commit 6c7ce3e
🔍 Latest deploy log https://app.netlify.com/projects/gateway-api-inference-extension/deploys/68ee07160be3260007fd9321
😎 Deploy Preview https://deploy-preview-1706--gateway-api-inference-extension.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 13, 2025
@dharaneeshvrd
Copy link
Contributor Author

/cc @delavet

@k8s-ci-robot
Copy link
Contributor

@dharaneeshvrd: GitHub didn't allow me to request PR reviews from the following users: delavet.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @delavet

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kfswain
Copy link
Collaborator

kfswain commented Oct 14, 2025

Hey @dharaneeshvrd! Thanks for the PR, do you mind adding this metric to our hermetic tests to validate the behavior?

gateway-api-inference-extension/test/integration/epp/hermetic_test.go

Line 705 in ef666c1

wantMetrics: map[string]string{`inference_objective_input_tokens`: `

@dharaneeshvrd dharaneeshvrd force-pushed the record-epp-ntpot-metric branch from 462c169 to e652c5a Compare October 14, 2025 07:55
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 14, 2025
@dharaneeshvrd dharaneeshvrd force-pushed the record-epp-ntpot-metric branch from e652c5a to 46c873a Compare October 14, 2025 08:16
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 14, 2025
@dharaneeshvrd
Record EPP NormalizedTimePerOutputToken metric on streaming mode
6c7ce3e 
Update e2e/epp/e2e_test & integration/epp/hermetic_test to validate inference_objective_normalized_time_per_output_token_seconds metric

Signed-off-by: Dharaneeshwaran Ravichandran <[email protected]>
@dharaneeshvrd dharaneeshvrd force-pushed the record-epp-ntpot-metric branch from 46c873a to 6c7ce3e Compare October 14, 2025 08:17
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 14, 2025
@dharaneeshvrd
Copy link
Contributor Author

@kfswain Updated the hermetic test. PTAL!

@dharaneeshvrd
Copy link
Contributor Author

@kfswain Can you please review this PR when you get a chance?

@danehans
Copy link
Contributor

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 27, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danehans, dharaneeshvrd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 27, 2025
@k8s-ci-robot k8s-ci-robot merged commit 8a869cc into kubernetes-sigs:main Oct 27, 2025
11 checks passed
sallyom pushed a commit to sallyom/gateway-api-inference-extension that referenced this pull request Oct 28, 2025
@dharaneeshvrd @sallyom
Record EPP NormalizedTimePerOutputToken metric on streaming mode (kub…
5064921 
…ernetes-sigs#1706)

Update e2e/epp/e2e_test & integration/epp/hermetic_test to validate inference_objective_normalized_time_per_output_token_seconds metric

Signed-off-by: Dharaneeshwaran Ravichandran <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kfswain kfswain Awaiting requested review from kfswain

@liu-cong liu-cong Awaiting requested review from liu-cong

Assignees

@danehans danehans

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

The normalized_time_per_output_token_seconds metrics is not recorded

4 participants

@dharaneeshvrd @k8s-ci-robot @kfswain @danehans