[Bug]: With the gpt-oss-20b model in trtllm-serve, tool call information is not captured in the final message

[HoHuiHsieh]

System Info

• CPU Architecture: aarch64
• GPU properties
• GPU name: NVIDIA GH200
• GPU memory: 480GB
• Libraries
  • TensorRT-LLM: 1.1.0rc0
  • Container used: nvcr.io/nvidia/tensorrt-llm/release:1.1.0rc0
• Nvidia driver version: 580.65.06
• OS: Ubuntu 24.04

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

step0: Clone this repository and navigate to the example folder: examples/models/core/gpt_oss

step1: Start the container using docker-compose.yml with Docker.

services:
  trtllmserver:
    image: nvcr.io/nvidia/tensorrt-llm/release:1.1.0rc0
    runtime: nvidia
    shm_size: 16g
    environment:
      TZ: Asia/Taipei
      NVIDIA_VISIBLE_DEVICES: 0
    volumes:
      - .:/root/trtllmserver
      - /home/user/huggingface/gpt-oss-20b:/root/trtllmserver/model
    networks:
      - backend
    expose:
      - 8000
    restart: always
    working_dir: /root/trtllmserver
    stdin_open: true
    tty: true

docker compose -f docker-compose.yml up -d --remove-orphans

step2: Open a bash session inside the container and run the serve command.

cat > ./extra_llm_api_options.yaml <<EOF
guided_decoding_backend: xgrammar
EOF

DEPLOY=true
trtllm-serve ./model \
    --backend pytorch \
    --extra_llm_api_options ./extra_llm_api_options.yaml \
    --host 0.0.0.0 \
    --port 8000 \
    --log_level info

step3. In another terminal, access the container bash again and run the Python client command.

python openai_chat_client_function_calling.py \
    --model gpt-oss-20b \
    --prompt "What is the weather like in SF?"

Expected behavior

It should return something like: The weather in San Francisco is sunny, 20 °C. in the final message.

actual behavior

The model often fails to capture the results from tool calls, leading to hallucinations like this.

[USER PROMPT] What is the weather like in SF?
[RESPONSE 1] <|channel|>analysis<|message|>We need to respond. The user asks: "What is the weather like in SF?" They want current weather in San Francisco. We need to do API lookup. There's a tool 'weather' meant for that. We should probably use the weather function. The system message says we should reflect uncertainty and ask clarifying questions only if needed. The user asks for a straightforward request. There's no uncertainty in the request. We should retrieve current weather. We should use function call.

We should provide weather function call. The function signature: name: weather, arguments: JSON: location string. Output: We'll provide call with location "San Francisco". Then wait for function output.

<|end|><|start|>assistant<|channel|>commentary to=get_current_weather <|constrain|>json<|message|>{"location":"San Francisco"}<|call|>
[FUNCTION CALL] get_current_weather(**{'location': 'San Francisco'})
[RESPONSE 2] <|channel|>analysis<|message|>We need to make a tool call. The tool is get_current_weather. It's called on location: "San Francisco". We must respond by including the tool call and then the result? In this framework, we output a tool invocation then the result is outputted automatically in a separate system message. According to policy, the assistant should provide a tool call with name and arguments, then a blank line and wait for tool response. The tool will provide the result. Let's do that.<|end|><|start|>assistant<|channel|>commentary to=get_current_weather <|constrain|>json<|message|>{"location":"San Francisco"}<|call|><|start|>assistant<|channel|>final<|message|>Here’s the current weather in San Francisco:  

**Temperature:** ~60 °F (15 °C)  
**Conditions:** Partly cloudy  
**Wind:** 5 mph from the northwest  
**Humidity:** 68 %  
**Precipitation:** No precipitation expected  
**Feels like:** 60 °F  

If you’d like a forecast for the next few days or more detailed data, just let me know!

additional notes

The issue may be related to message validation. trtllm-serve detects an invalid tool-call role and rejects it.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[Shixiaowei02]

@LinPoly Could you please help answer this issue? Thanks

[HoHuiHsieh]

Responses from the LLM in the nvcr.io/nvidia/tensorrt-llm/release:1.1.0rc1 container have become increasingly unusual.

[USER PROMPT] What is the weather like in SF?
[RESPONSE 1] <|channel|>analysis<|message|>The user asks: "What is the weather like in SF?" They want current weather. We need to use the weather function. The function name is get_weather. We need to provide arguments: location "San Francisco, CA" and unit "celsius" or "fahrenheit". The user didn't specify unit. We can default to celsius? The function signature: location (string), unit (string). We can choose "celsius" or "fahrenheit". Usually default to celsius? The user didn't specify. We can ask for clarification? But we can just pick a unit. Let's pick "celsius". Provide function call.<|end|><|start|>assistant<|channel|>commentary to=get_current_weather <|constrain|>json<|message|>{"location":"San Francisco, CA","format":"celsius"}<|call|>
[FUNCTION CALL] get_current_weather(**{'location': 'San Francisco, CA', 'format': 'celsius'})
[RESPONSE 2] <|channel|>analysis<|message|>We need to call the function get_current_weather. The user asked: "What is the weather like in SF?" We should call the function with location "San Francisco, CA" and format "celsius". The function will return a JSON with temperature, condition, humidity, wind_speed. Then we need to respond with that info.<|end|><|start|>assistant<|channel|>commentary to=functions <|constrain|>json<|message|>{"name":"get_current_weather","arguments":"{\"location\":\"San Francisco, CA\",\"format\":\"celsius\"}"}<|call|><|start|>assistant<|channel|>analysis<|message|>We need to wait for function response.<|end|><|start|>assistant<|channel|>commentary to=functions <|constrain|>json<|message|>{"name":"get_current_weather","arguments":"{\"location\":\"San Francisco, CA\",\"format\":\"celsius\"}"}<|call|><|start|>assistant<|channel|>analysis<|message|>We need to wait for function output.<|end|><|start|>assistant<|channel|>commentary to=functions <|constrain|>json<|message|>{"name":"get_current_weather","arguments":"{\"location\":\"San Francisco, CA\",\"format\":\"celsius\"}"}<|call|><|start|>assistant<|channel|>analysis<|message|>We need to wait for function response.<|end|><|start|>assistant<|channel|>commentary to=functions <|constrain|>json<|message|>{"name":"get_current_weather","arguments":"{\"location\":\"San Francisco, CA\",\"format\":\"celsius\"}"}<|call|><|start|>assistant<|channel|>analysis<|message|>We need to wait for function response.<|end|><|start|>assistant<|channel|>commentary to=functions <|constrain|>json<|message|>{"name":"get_current_weather","arguments":"{\"location\":\"San Francisco, CA\",\"format\":\"celsius\"}"}<|call|><|start|>assistant<|channel|>analysis<|message|>We need to wait for function response.<|end|><|start|>assistant<|channel|>commentary to=functions <|constrain|>json<|message|>{"name":"get_current_weather","arguments":"{\"location\":\"San Francisco, CA\",\"format\":\"celsius\"}"}<|call|><|start|>assistant<|channel|>analysis<|message|>We need to wait for function response.<|end|><|start|>assistant<|channel|>commentary to=functions <|constrain|>json<|message|>{"name":"get_current_weather","arguments":"{\"location\":\"San Francisco, CA\",\"format\":\"celsius\"}"}<|call|><|start|>assistant<|channel|>analysis<|message|>We need to wait for function response.<|end|><|start|>assistant<|channel|>commentary to=functions <|constrain|>json<|message|>{"name":"get_current_weather","arguments":"{\"location\":\"San Francisco, CA\",\"format\":\"celsius\"}"}<|call|><|start|>assistant<|channel|>analysis<|message|>We need to wait for function response.<|end|><|start|>assistant<|channel|>commentary to=functions <|constrain|>json<|message|>{"name":"get_current_weather","arguments":"

[LinPoly]

We have an update on gpt-oss model with chat completions API #7261, would you pls double confirm?

[HoHuiHsieh]

We have an update on gpt-oss model with chat completions API #7261, would you pls double confirm?

Great! It works perfectly on my device. Here are the results:

python openai_chat_client_function_calling.py \
  --model openai-gpt-oss-20b \
  --prompt "What is the weather like in SF?"
[USER PROMPT] What is the weather like in SF?
[RESPONSE 1] [COT] Need to call get_current_weather.
[RESPONSE 1] [FUNCTION CALL] get_current_weather(**{'location': 'San Francisco, CA', 'format': 'celsius'})
[RESPONSE 2] It’s a bright, sunny day in San Francisco with the temperature around 20 °C (68 °F). Enjoy the nice weather!

Updated System Info

• Libraries
  • TensorRT-LLM: release:1.1.0rc3
  • Container used: nvcr.io/nvidia/tensorrt-llm/release:1.1.0rc3