Improved Usage Tracking

[sirwolfgang]

This is to improved usage tracking and instrumentation for cost monitoring and APM integration. The original instrumentation was a slop coded prototype, that didn't actually work in a way that is useful. This cuts it down and restructures it for APM integration while adding usage normalization.

Multi-Turn Usage Tracking

Cumulative token tracking across tool calling cycles:

response = agent.generate("Research the weather in NYC")
# After 3 API calls (initial + 2 tool rounds):

response.usage.input_tokens   #=> 350 (cumulative across all calls)
response.usage.output_tokens  #=> 120 (cumulative across all calls)
response.usage.total_tokens   #=> 470 (cumulative across all calls)

# Access per-call details
response.usages.size          #=> 3
response.usages[0].input_tokens  #=> 100 (first call)
response.usages[1].input_tokens  #=> 150 (after tool execution)
response.usages[2].input_tokens  #=> 100 (final call)

Usage Statistics

Consistent token tracking across all providers:

response = agent.generate("Hello")
response.usage.input_tokens   #=> 8
response.usage.output_tokens  #=> 12
response.usage.total_tokens   #=> 20
response.usage.cached_tokens  #=> 0 (when available)

Provider-specific fields: reasoning_tokens, audio_tokens, cache_creation_tokens, service_tier, duration_ms

Combine usage objects:

total_usage = response1.usage + response2.usage
total_usage.input_tokens  #=> Combined from both

APM Integration

Unified instrumentation events with comprehensive payloads:

Top-level events (overall request lifecycle):

# Fires once per agent.generate call
ActiveSupport::Notifications.subscribe("prompt.active_agent") do |event|
  event.payload[:model]           #=> "gpt-4"
  event.payload[:temperature]     #=> 0.7
  event.payload[:message_count]   #=> 3
  event.payload[:stream]          #=> false
  
  # Cumulative across all API calls in multi-turn conversations
  event.payload[:usage]           #=> { input_tokens: 350, output_tokens: 120, total_tokens: 470 }
  event.payload[:finish_reason]   #=> "stop"
  event.payload[:response_id]     #=> "chatcmpl-123"
end

Provider-level events (per API call):

# Fires for each API call (multiple times during tool calling)
ActiveSupport::Notifications.subscribe("prompt.provider.active_agent") do |event|
  # Per-call usage for granular tracking
  event.payload[:usage]           #=> { input_tokens: 100, output_tokens: 50, total_tokens: 150 }
end

Event payloads include:

• Request params: model, temperature, max_tokens, top_p, stream, message_count, has_tools
• Usage data: input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens, audio_tokens, cache_creation_tokens
• Response metadata: finish_reason, response_model, response_id, embedding_count

Embedding events:

ActiveSupport::Notifications.subscribe("embed.active_agent") do |event|
  event.payload[:model]           #=> "text-embedding-ada-002"
  event.payload[:input_size]      #=> 5
  event.payload[:embedding_count] #=> 5
  event.payload[:usage]           #=> { input_tokens: 150, output_tokens: 0, total_tokens: 150 }
end

New response methods:

response.id              # Response ID
response.model           # Model used
response.finish_reason   # Why generation stopped
response.stop_reason     # Alias for Anthropic

Breaking Change

# Before
response.usage["prompt_tokens"]

# After
response.usage.input_tokens

[Copilot]

Pull Request Overview

This pull request introduces normalized usage statistics across all providers (OpenAI, Anthropic, Ollama, and OpenRouter) by adding a unified Usage model. The changes enable consistent access to token usage statistics regardless of which provider is being used.

Key changes:

• Added comprehensive unit tests for provider transform modules to verify gem-to-hash conversions and parameter normalization
• Implemented Usage model tests covering all provider formats with auto-detection capabilities
• Added integration tests with VCR cassettes for all providers to validate response handling
• Updated documentation and added inline documentation for model classes

Reviewed Changes

Copilot reviewed 214 out of 216 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/providers/open_router/transforms_test.rb	Unit tests for OpenRouter-specific transforms including parameter extraction, serialization, and response format handling
test/providers/open_ai/responses/transforms_test.rb	Unit tests for OpenAI Responses API transforms covering input normalization, response format handling, and message simplification
test/providers/open_ai/embedding/transforms_test.rb	Unit tests for OpenAI Embedding transforms including input normalization, parameter handling, and serialization cleanup
test/providers/open_ai/chat/transforms_test.rb	Comprehensive unit tests for OpenAI Chat transforms covering message normalization, content handling, and instruction processing
test/providers/ollama/embedding/transforms_test.rb	Unit tests for Ollama Embedding transforms including Ollama-specific parameter extraction and input validation
test/providers/ollama/chat/transforms_test.rb	Unit tests for Ollama Chat transforms covering message grouping and Ollama-specific parameter handling
test/providers/common/usage_test.rb	Core unit tests for Usage model covering all provider formats, auto-detection, and field normalization
test/providers/anthropic/transforms_test.rb	Unit tests for Anthropic transforms covering message normalization, content handling, and response cleanup
test/integration/*	Integration tests with VCR cassettes validating real API response handling across all providers
test/fixtures/vcr_cassettes/*	VCR cassettes capturing real API responses for reliable integration testing
test/docs/providers/anthropic_examples_test.rb	Fixed assertion argument order to follow Minitest conventions
lib/active_agent/providers/open_router/requests/prediction.rb	Added comprehensive inline documentation for Prediction model
lib/active_agent/providers/open_router/requests/message.rb	Removed unused require statement

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.