[Feature][APIServer v2] Support Compute Template in APIServer v2 #3959

machichima · 2025-08-15T23:50:51Z

Why are these changes needed?

APIServer v1 introduced a unique and mandatory feature, Compute Template, which requires users to predefine the amount of CPU and memory resources they need in templates.

APIServer v2 doesn’t require users to predefine templates anymore, but we still want to continue supporting the feature to help v1 users migrate to v2 more easily.

Implementation

Unmarshal the incoming request body in a new apiserversdk proxy handler.
Extract computeTemplate from the headGroupSpec and workerGroupSpec.
Mutate the request body according to the template before marshaling back.

We need 3 new proxy handlers for RayCluster, RayJob, and RayService, respectively. They may be able to share some parts of the implementation.

Added e2e test in compute_template_e2e_test.go

Related issue number

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: machichima <[email protected]>

…r-v2-compute-template Signed-off-by: machichima <[email protected]>

Signed-off-by: machichima <[email protected]>

…r-v2-compute-template Signed-off-by: machichima <[email protected]>

Signed-off-by: machichima <[email protected]>

Just a structure, test does not work now Signed-off-by: machichima <[email protected]>

Signed-off-by: machichima <[email protected]>

…r-v2-compute-template Signed-off-by: machichima <[email protected]>

Signed-off-by: machichima <[email protected]>

machichima · 2025-09-07T00:38:42Z

apiserver/pkg/http/client.go

 	//
 	// Store http request handling function for unit test purpose.
-	executeHttpRequest func(httpRequest *http.Request, URL string) ([]byte, *rpcStatus.Status, error)
+	ExecuteHttpRequest func(httpRequest *http.Request, URL string) ([]byte, *rpcStatus.Status, error)


Make this public so that we can use in apiserversdk

@machichima, can we revert this now?

I think we still need this to be public as we will use it in:

https://github.com/ray-project/kuberay/pull/3959/files#diff-5427afc42b84a19b5dc31e6751e58900f883eb8391e1839b790ff9a85e336b7cR273

machichima · 2025-09-07T00:49:06Z

apiserver/pkg/manager/resource_manager.go


 // Compute template
-func (r *ResourceManager) populateComputeTemplate(ctx context.Context, clusterSpec *api.ClusterSpec, nameSpace string) (map[string]*api.ComputeTemplate, error) {
+func (r *ResourceManager) PopulateComputeTemplate(ctx context.Context, clusterSpec *api.ClusterSpec, nameSpace string) (map[string]*api.ComputeTemplate, error) {


Make this public so that we can use in apiserversdk

machichima · 2025-09-07T02:45:48Z

apiserver/test/e2e/apiserversdk/exec.go

+}
+
+// execCommandWithCurlInPod executes a curl command inside the specified pod's container by `kubectl exec`
+func (rec *RemoteExecuteClient) execCommandWithCurlInPod(pod *corev1.Pod, url string, method string, body string, contentType string) ([]byte, error) {


Take this function from apiserver/test/e2e/exec.go and addcontentType args

machichima · 2025-09-07T02:51:53Z

apiserver/test/e2e/apiserversdk/types.go

 	return e2etc.ctx
 }

-func (e2etc *End2EndTestingContext) GetK8sHttpClient() *kubernetes.Clientset {


Originally there's duplicate k8sHttpClient and k8sClient that are both of type *kubernetes.Clientset. We removed k8sHttpClient and only keep k8sClient

machichima · 2025-09-07T14:49:20Z

apiserversdk/util/template.go

+)
+
+// compute_template_middleware.go
+func NewComputeTemplateMiddleware(clientManager manager.ClientManagerInterface) func(http.Handler) http.Handler {


Make clientManager as the args so that we can mock this out in unit test

Signed-off-by: machichima <[email protected]>

machichima · 2025-09-07T17:12:05Z

Not sure why in "build apiserversdk" check, the test failed. It works on my side

rueian · 2025-09-07T17:44:33Z

Not sure why in "build apiserversdk" check, the test failed. It works on my side

That is a new flaky failure happened frequently recently. I am wondering if it could be related to the newly added retry roundtripper. Could you help verify in another PR?

rueian · 2025-09-07T18:00:23Z

Or the flaky may be related to #4061

machichima · 2025-09-07T18:26:29Z

Or the flaky may be related to #4061

I'll try fixing this flaky test in another PR if the error persist after the #4061 PR merged

fscnick · 2025-09-08T13:19:12Z

apiserver/test/e2e/apiserversdk/compute_template_e2e_test.go

+}
+
+// verifyPodSpecResources verifies that the PodSpec has the expected resources from the compute template
+func verifyPodSpecResources(t *testing.T, podSpec *corev1.PodSpec, _, groupType string, computeTemplate *api.ComputeTemplate) {


Is any specific reason to put a _ in the function signature?

It's not needed, I think I forgot to remove it. Thank you for pointing this out!

Updated in edc99ad

fscnick · 2025-09-08T13:40:08Z

apiserversdk/util/template.go

+func convertRequestBodyToMap(requestBody []byte, contentType string) (map[string]any, error) {
+	var requestMap map[string]interface{}


nit: choose one between any and interface{} to make it consistent

Thank you! I changed interface{} to any as suggested in my linter in 8ad0573

fscnick · 2025-09-08T13:48:12Z

apiserversdk/util/template.go

+				} else {
+					tolerations = make([]interface{}, 0)
+				}


The else branch might not be necessary.

https://go.dev/play/p/ZB0ZrroHJl_C

Thank you! Removed the block in 559f2ae

…r-v2-compute-template

rueian · 2025-09-12T15:55:57Z

apiserversdk/util/template.go

+				computeTemplate, err := getComputeTemplate(context.Background(), resourceManager, headGroupMap, namespace)
+				if err != nil {
+					klog.Errorf("ComputeTemplate middleware: Failed to get compute template for head group: %v", err)
+					http.Error(w, err.Error(), http.StatusInternalServerError)


I feel like this should be a client error. Returning HTTP 422 will be better.

Fix in 967c314

rueian · 2025-09-12T15:57:45Z

apiserversdk/util/template.go

+func applyComputeTemplateToRequest(computeTemplate *api.ComputeTemplate, clusterSpecMap *map[string]any, group string) {
+	// calculate resources
+	cpu := fmt.Sprint(computeTemplate.GetCpu())
+	memory := fmt.Sprintf("%d%s", computeTemplate.GetMemory(), "Gi")


We need to support the new memory unit field.

Added in b56867d

…r-v2-compute-template

rueian · 2025-09-19T01:02:13Z

apiserver/test/e2e/apiserversdk/exec.go

+
+// RemoteExecuteClient allows executing HTTP requests against a service running inside a Kubernetes pod
+// by using `kubectl exec`-style command execution, without requiring a NodePort for external access.
+type RemoteExecuteClient struct {


Could we use the existing ProxyRoundTripper instead of introducing a RemoteExecuteClient?

Updated in 9561d29, thank you!

…r-v2-compute-template

…e client

rueian · 2025-09-26T13:56:09Z

apiserver/test/e2e/apiserversdk/compute_template_e2e_test.go

+	expectedMemory := fmt.Sprintf("%dGi", computeTemplate.GetMemory())
+	memoryLimit := rayContainer.Resources.Limits[corev1.ResourceMemory]
+	memoryRequest := rayContainer.Resources.Requests[corev1.ResourceMemory]
+	require.Equal(t, expectedMemory, memoryLimit.String(), "Expected memory limit to be 4Gi")


Could you remove the hard-coded values in these assertion messages in a follow-up?

Sure! no problem

rueian · 2025-09-26T14:00:26Z

apiserversdk/util/template.go

+
+			klog.Infof("ComputeTemplate middleware: Successfully processed request, sending to next handler")
+			// Update Content-Type to application/json and Content-Length header to match the new body size
+			r.Header.Set("Content-Type", "application/json")


Could you keep the original content type in a follow-up?

This is because after processing, we convert the map to JSON. Therefor I change the content-type here to be json

https://github.com/ray-project/kuberay/pull/3959/files#diff-a310e70ce3c41e0d4d682d053af475d9d67168224dacd5eaa7bf4d3e92479419R112

machichima added 20 commits August 16, 2025 07:44

feat: extract compute template from spec

cb6f1ea

Signed-off-by: machichima <[email protected]>

Merge branch 'master' of github.com:ray-project/kuberay into apiserve…

2edd8f0

…r-v2-compute-template Signed-off-by: machichima <[email protected]>

feat: add compute template middleware

7dd5a70

Signed-off-by: machichima <[email protected]>

fix: extract cluster spec from request correctly

7106c01

Signed-off-by: machichima <[email protected]>

Merge branch 'master' of github.com:ray-project/kuberay into apiserve…

7cc1d4e

…r-v2-compute-template Signed-off-by: machichima <[email protected]>

feat: convert request to map then to clusterSpec

4417877

Signed-off-by: machichima <[email protected]>

feat: parse anno/cpu/mem from compute template to request

c8a8f78

Signed-off-by: machichima <[email protected]>

fix: get compute template name without converting to ClusterSpec struct

3dc557c

Signed-off-by: machichima <[email protected]>

feat: apply gpu/extend/toleration

c4fc102

Signed-off-by: machichima <[email protected]>

feat: forward modified request body

fa7eae8

Signed-off-by: machichima <[email protected]>

feat: deal with case with no head/workerGroupSpec set

0f05fed

Signed-off-by: machichima <[email protected]>

feat: also support for rayjob and rayservice

ce17d2e

Signed-off-by: machichima <[email protected]>

test: structure for test compute template middleware

1f8db4c

Just a structure, test does not work now Signed-off-by: machichima <[email protected]>

fix: update field and match types

062d602

Signed-off-by: machichima <[email protected]>

Merge branch 'master' of github.com:ray-project/kuberay into apiserve…

d6e2cb5

…r-v2-compute-template Signed-off-by: machichima <[email protected]>

feat: use remote execute for apiserver sdk e2e test

abeaa5d

Signed-off-by: machichima <[email protected]>

feat: pass through if no spec & convert to json

b17b879

Signed-off-by: machichima <[email protected]>

feat: check request body type for unmarshal

740e8f4

Signed-off-by: machichima <[email protected]>

test: enable setitng content type in execCommandWithCurlInPod

20efd19

Trigger CI

b796540

Signed-off-by: machichima <[email protected]>

machichima marked this pull request as ready for review September 5, 2025 15:50

machichima added 5 commits September 6, 2025 17:34

fix: extract client manager for mocking

7695daf

Trigger CI

39e8919

Signed-off-by: machichima <[email protected]>

fix: add missing methods & clean up code

9c418f5

refactor: clean up print

0af361e

build: go mod tidy

44213ea

machichima commented Sep 7, 2025

View reviewed changes

machichima added 2 commits September 7, 2025 08:14

Trigger CI

4ba14e2

Signed-off-by: machichima <[email protected]>

Trigger CI

f772cd5

Signed-off-by: machichima <[email protected]>

fscnick reviewed Sep 8, 2025

View reviewed changes

machichima added 4 commits September 11, 2025 07:11

refactor: remove unused args

edc99ad

refactor: interface to any

8ad0573

refactor: remove redundant else block

559f2ae

Merge branch 'master' of github.com:ray-project/kuberay into apiserve…

65bbc6a

…r-v2-compute-template

rueian reviewed Sep 12, 2025

View reviewed changes

machichima added 3 commits September 15, 2025 19:57

refactor: status code 500 to 422

967c314

feat: support memory unit

b56867d

Merge branch 'master' of github.com:ray-project/kuberay into apiserve…

d9a9e2c

…r-v2-compute-template

rueian reviewed Sep 19, 2025

View reviewed changes

machichima added 3 commits September 23, 2025 19:29

Merge branch 'master' of github.com:ray-project/kuberay into apiserve…

bee12d0

…r-v2-compute-template

fix: use ProxyRoundTripper is apiserver client & remove remote execut…

9561d29

…e client

fix: base url use kubernetesConfig.Host

f4df39c

machichima requested a review from rueian September 24, 2025 12:58

rueian reviewed Sep 26, 2025

View reviewed changes

rueian approved these changes Sep 26, 2025

View reviewed changes

rueian merged commit 536ca35 into ray-project:master Sep 28, 2025
27 checks passed

		func convertRequestBodyToMap(requestBody []byte, contentType string) (map[string]any, error) {
		var requestMap map[string]interface{}

[Feature][APIServer v2] Support Compute Template in APIServer v2 #3959

[Feature][APIServer v2] Support Compute Template in APIServer v2 #3959

Uh oh!

Conversation

machichima commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Implementation

Related issue number

Checks

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

machichima commented Sep 7, 2025

Uh oh!

rueian commented Sep 7, 2025

Uh oh!

rueian commented Sep 7, 2025

Uh oh!

machichima commented Sep 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

machichima commented Aug 15, 2025 •

edited

Loading