Skip to content

Conversation

@KrishnanPrash
Copy link
Contributor

This PR is meant to address a silent error not being caught in tritonclient.http.is_server_live() and tritonclient.http.is_server_ready().

Currently if we start a tritonserver instance with a health restricted feature specification, with the following command:

tritonserver --model-repo ... --grpc-restricted-protocol=health:health-key=health-value  --http-restricted-api=health:health-key=health-value

Then if we use tritonclient to query for health information, we receive the following:

>>> import tritonclient.http as httpclient
>>> import tritonclient.grpc as grpcclient
>>> http_client = httpclient.InferenceServerClient("localhost:8000")
>>> grpc_client = grpcclient.InferenceServerClient("localhost:8001")
>>> grpc_client.is_server_ready(headers={"triton-grpc-protocol-health-key":"health-value"})
True
>>> http_client.is_server_ready(headers={"health-key":"health-value"})
True
>>> grpc_client.is_server_ready(headers={"invalid-key":"invalid-value"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/tritonclient/grpc/_client.py", line 344, in is_server_ready
    raise_error_grpc(rpc_error)
  File "/usr/local/lib/python3.10/dist-packages/tritonclient/grpc/_utils.py", line 77, in raise_error_grpc
    raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.UNAVAILABLE] This protocol is restricted, expecting header 'triton-grpc-protocol-health-key'
>>> http_client.is_server_ready(headers={"invalid-key":"invalid-value"}) # Should be erroring out with InferenceServerException: This API is restricted, expecting header 'health-key`
False 

From the above example, we can see that tritonclient provides different behavior for grpc and http in terms of erroring out when invalid credentials are presented.

@rmccorm4 rmccorm4 changed the title Making tritonclient.http and tritonclient.grpc health checks uniform refactor: Making tritonclient.http and tritonclient.grpc health checks uniform Dec 20, 2024
@rmccorm4 rmccorm4 self-assigned this Dec 20, 2024
@rmccorm4 rmccorm4 requested review from GuanLuo and rmccorm4 December 20, 2024 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants