Add Open Telemetry instrumentation #526
Conversation
|
Traces that mix LLM model call spans with
|
|
Do we want to (optionally?) also include user and assistant messages, a la https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-events/ ? |
|
@hadley I do. But there's a ton of disagreement in the OTel LLM community about how to do that, and none of the existing instrumentation libraries work in the same way 😞. Plus the whole "structured body" mechanism the current spec proposes (1) isn't supported by the span API; and (2) is formally deprecated. So I kind of think we need to noodle on what to do there, and I suggest pushing it into follow-up work. I'm planning on writing up an issue describing what options we have. I also think we should have first-class support for tool call spans, because that's something that |
|
@atheriel ok, that makes sense. I'm sure there will be a lot of learning as we figure out exactly what is most useful to instrument across packages. |
|
Moving this back to draft because it has known issues (i.e. the concurrency does not work correctly). |
This commit instruments various operations with Open Telemetry spans that
abide by the (still nascent) semantic conventions for Generative AI
clients [0].
These conventions classify `ellmer` chatbots as "agents" due to their
ability to run tool calls, so in fact there are three types of span: (1)
a top-level `invoke_agent` span for each chat interaction; (2) `chat`
spans that wrap model API calls; and (3) `execute_tool` spans that wrap
tool calls on our end.
There's currently no community concensus for how to attach turns to
spans, so I've left that out for now.
Example code:
library(otelsdk)
Sys.setenv(OTEL_TRACES_EXPORTER = "stderr")
chat <- ellmer::chat_databricks(model = "databricks-claude-3-7-sonnet")
chat$chat("Tell me a joke in the form of an SQL query.")
Unit tests are included.
[0]: https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/
Signed-off-by: Aaron Jacobs <[email protected]>
atheriel
changed the title
|
This has been updated to support async operations and for changes in |
* main: (95 commits) fix(chat): Call `check_echo()` in `chat()` for consistent echo behavior (#742) Increment version number to 0.3.2.9000 Increment version number to 0.3.2 Don't run `content_image()` on CRAN (#739) feat(chat_): Add `params` and `model` to all `chat_` functions (#699) Fix spelling in `tool_prompt.md` (#730) Fix typos in source comments and regenerate documentation (#736) Fix news bullet Increment version number to 0.3.1.9000 Increment version number to 0.3.1 Update cran comments Check revdpes Re-build readme Typo fixes (#686) Polish news Update to latest Air settings and use `format-suggest.yaml` (#683) Use newer REST API base url (#726) Fix auth scope and API endpoints for Google Vertex (#704) Run `Rscript data-raw/prices.R` to update pricing info (#727) Improve error message for `batch_chat()` (#716) ...
Co-Authored-By: Aaron Jacobs <[email protected]>
Co-Authored-By: Aaron Jacobs <[email protected]>
…n activation for chat
| if (!is.null(tracer)) { | ||
| return(tracer) | ||
| } | ||
| if (testthat::is_testing()) { |
There was a problem hiding this comment.
Must check if {testthat} is installed (as it is currently only a Suggests pkg)
There was a problem hiding this comment.
You can just inline that function (i.e. copy it in into utils.R)
This commit wraps all LLM model calls in an Open Telemetry span that abides by the (still nascent) semantic conventions for Generative AI clients.
It's very similar in approach to what was done for
httr2, and in fact the two of them complement one another nicely:r-lib/httr2#729.
For example: