Skip to content

Conversation

kmw10693
Copy link

Overview

This PR fixes a JSON parsing error that occurs when handling streaming tool calls in Spring AI.

Related issue: spring-projects/spring-ai#4250


Why

  • In streaming mode, tool call JSON fragments can arrive incomplete.
  • This results in Jackson throwing errors such as: com.fasterxml.jackson.core.JsonParseException: Unexpected close marker ']': expected '}'
  • The issue only occurs in streaming (stream()) calls.
  • Synchronous calls (call()) are not affected.
  • This prevents users from reliably using tool callbacks with streaming responses.

What Changed

  • Added a new ToolExecutionEligibilityPredicate implementation: ToolOnFinishPredicate.
  • Ensures tool execution only runs after the assistant signals completion (finishReason = "tool_calls" or "stop").
  • Prevents premature execution and malformed JSON during streaming tool calls.

How to Use

Example usage:

ChatClient client = ChatClient.create(openAiChatModel);

String result = client.prompt()
  .user("Migrate node cluster xx")
  .toolCallbacks(ToolCallbacks.from(new MyTool()))
  .options(OpenAiChatOptions.builder()
      .toolExecutionEligibilityPredicate(new ToolOnFinishPredicate())
      .build())
  .stream()
  .content();

import java.util.List;
import java.util.function.BiPredicate;

import org.jetbrains.annotations.NotNull;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't use this annotation for not null

Copy link
Author

@kmw10693 kmw10693 Sep 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@markpollack I’ve removed ToolOnFinishPredicate.java and applied the fix directly in OpenAiChatModel.
when the LLM returns malformed or invalid JSON during streaming, the code explicitly throws an exception.

@markpollack markpollack modified the milestones: 1.1.0.M2, 1.1.0.M3 Sep 17, 2025
@markpollack
Copy link
Member

The PR looks incomplete. I've commented on the issue thread for further clarification. #4274

@kmw10693
Copy link
Author

The PR looks incomplete. I've commented on the issue thread for further clarification. #4274

@markpollack
On the issue, the problematic LLM was Qwen3-8B. they found that the chunk returned by the LLM was invalid.

However, the OpenAIChatModel was not properly detected when the LLM returned an invalid response.
I’ve updated OpenAiChatModel so that if the LLM sends an invalid or malformed JSON response (e.g., broken chunk). the code now explicitly throws an exception. I also added the OpenAiChatModelStreamingTest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants