🚀 The feature, motivation and pitch

Hello, we are hoping to allow our users to have a better understanding of what tokens are cached during their workloads. We would like to add KV cache metrics (e.g., cached tokens, tokens used from cache on a given request, etc.) as a part of the usage object so that the requestor can get a more detailed view of how their request interacted with the KV Cache. Ideally, this would extend to encapsulate metrics from lmcache also.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.