[apiserver] Add retry and timeout to apiserver V2

[kenchung285]

Why are these changes needed?

Add retry logic and timeout to apiserver v2:

1. Timeout is set to long timeout (30 seconds) to prevent unbounded request
2. The retry logic determines if a http status code is retryable

Wrap errors with more informations

Related issue number

Closes #3606

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[kenchung285]

@dentiny PTAL, thanks!

[dentiny]

I don't see any unit tests, curious how did you test it?

[kenchung285]

I don't see any unit tests, curious how did you test it?

Currently my implementation logs the error message to stdout, so I can test my implementation by manual test with deploying a apiserver

I test with the following steps

cd apiserver
make start-local-apiserver

# An invalid path
curl http://localhost:31888/apis/ray.io/v1/namespaces/ray-system/invalid/path

# See stdout logs of the apiserver pod
kubectl logs -n ray-system pod/kuberay-apiserver-85744c5487-8csl4

And what I got is:

[kenchung285]

However, this may not work for the case that we wrapped error message into error

[kenchung285]

@dentiny PTAL, thanks!

[kenchung285]

@dentiny @rueian , PTAL, thanks!

[dentiny]

Thanks!

[fscnick]

LGTM

[rueian]

Do we still need to use req.GetBody()?

[kenchung285]

Because req.Body in go is an one-time streamer, which can only be used once, we need to regenerate a new body for the retries by req.GetBody()

[rueian]

You think can just use req.Body = io.NopCloser(bytes.NewReader(bodyBytes)) once you move bodyBytes, err = io.ReadAll(req.Body) out of the loop.

[rueian]

Suggested change

	if attempt > 0 && req.GetBody != nil {
	req.Body = io.NopCloser(bytes.NewReader(bodyBytes))
	}
	if bodyBytes != nil {
	req.Body = io.NopCloser(bytes.NewReader(bodyBytes))
	}

Could you add a test that uses req.Body as well?

[rueian]

Thanks @kenchung285!