Handling orphaned tool_use blocks: exception handling vs. configurable automation

[dbschmigelski]

I'd like to start a discussion regarding the handling of orphaned tool_use blocks in our SDK, specifically addressing issue #495.

The issue occurs when a model doesn't complete tool results, particularly in cases like max_token limits or timeouts, leaving orphaned tool_use blocks without corresponding tool_result blocks. This leads to ValidationException errors that disrupt the conversation flow.

We're currently considering two main approaches to address this:

The first proposal suggests raising an exception when these orphaned blocks are detected. This would give users full control over how to handle these situations in their applications. Users could implement their own recovery strategies based on their specific needs and use cases.

The second approach involves introducing a configurable mechanism to automatically handle orphaned tool_use blocks. This could be implemented similar to the hook solution demonstrated in the issue, but as a proper part of the SDK's configuration. This would provide out-of-the-box handling while still maintaining flexibility. However, implementing default automatic handling behavior presents significant challenges as the root causes of these failures can vary. For instance, when max_tokens is reached, there are multiple potential remediation strategies: we could automatically increase the max_tokens limit, remove certain tools that might be causing the issue, modify the prompts, or take other corrective actions. While we could provide an interface for implementing these strategies, determining a sensible default behavior is non-obvious given the variety of use cases and potential failure modes. This complexity suggests that while we should provide the mechanism for automatic handling, users may need to implement their own handling logic based on their specific needs and understanding of their application's behavior. The option is still valid as the approach aims to provide a mechanism for the agent to continue without prematurely terminating the request.

An important consideration is that we can implement this in a backwards-compatible manner. We could default to raising an exception (maintaining current behavior), while allowing users to opt into automatic handling through configuration parameters. For example, we could introduce a configuration option that's None by default (raising exceptions), but when configured, would specify how to handle orphaned blocks.

We're looking for community input on these approaches. Which would better serve your use cases? Would you prefer explicit exception handling, or would automatic handling with configuration be more beneficial? Your feedback will help shape how we address this issue in the SDK.

Please share your thoughts and experiences to help us determine the best path forward.

[Avinm]

I have not had the ability to test this out extensively, but during my initial research, I had found quite a lot of frameworks similar to strands:

1. LangGraph
2. Autogen
3. Langroid
4. CrewAI
5. AgentSquad

It might be worth checking what some of these end up doing in similar situations?

[dbschmigelski]

Strands is going to move forward with the following solution:

We will be consistent with other failure types and fail hard by default with a MaxTokensReachedException.

To allow agents to recover without terminating the event loop a new Hook event will be added. This event will be triggered when max tokens is reached where the result of the hook will determine if we proceed or of if the exception is re-thrown.

Finally, to aid customers, an implementation of the HookProvider will be added to the SDK to solve this use case in a manner which we believe will work for the majority of use cases. The implementation will do the following:

Purge the invalid ToolUse message from the messages array
Insert a new message indicating that the previous call failed with max_tokens
The idea here is that on the next iteration of the cycle, the max_tokens message will bias the agent to use or not avoid certain problematic tools.