Image generation tool #6749

ericstj · 2025-08-26T15:41:55Z

Fixes #6639

This adds a prototype implementation of ImageGenerationTool with support in OpenAI provider.

To use this requires using the OpenAI Responses chat client with OpenAI provider (not Azure OpenAI).

Still required:

Iterate on behavior of middle-ware to better support edit and remove workarounds.
More tests(currently testing with a sample)
Polish and document API
Update to latest OpenAI and remove private reflection.

Microsoft Reviewers: Open in CodeFlow

Copilot

Pull Request Overview

This PR adds a prototype implementation of ImageGenerationTool with support in the OpenAI provider, enabling AI services to perform image generation when specified as a tool.

Introduces ImageGenerationTool as a marker tool class that can be configured with ImageGenerationOptions
Adds experimental APIs for applying chat response updates to existing responses with configurable coalescing options
Implements reflection-based handling of OpenAI's internal image generation response types in the OpenAI provider

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
ImageGenerationTool.cs	New tool class for enabling image generation capabilities
ChatResponseUpdateCoalescingOptions.cs	Configuration options for how response updates are merged
ChatResponseExtensions.cs	Extension methods for applying updates to chat responses
OpenAIResponsesChatClient.cs	OpenAI provider implementation with image generation support via reflection
OpenAIJsonContext.cs	Added JSON serialization support for new types
ChatResponseUpdate.cs	Fixed documentation reference
ChatResponseUpdateExtensionsTests.cs	Comprehensive tests for new update coalescing functionality

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIResponsesChatClient.cs

src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatCompletion/ChatResponseExtensions.cs

PederHP · 2025-09-15T15:20:02Z

src/Libraries/Microsoft.Extensions.AI.Abstractions/Image/ImageGenerationTool.cs

+
+namespace Microsoft.Extensions.AI;
+
+/// <summary>Represents a hosted tool that can be specified to an AI service to enable it to perform image generation.</summary>


There's other hosted tools, like OpenAI's Code Interpreter. Other providers have something similar. Anthropic for example has Web Search, Fetch and Code Interpreter as "Server Tools". Maybe out of scope for this change, but a generalized abstraction for these would be great. AdditionalProperties seems to be common for all of them - including the Anthropic ones. So I think this would fit beautifully as a more general abstraction than ImageGenerationTool. ServerTool or HostedTool?

We already have HostedWebSearchTool, HostedCodeInterpreterTool, HostedFileSearchTool, and HostedMcpServerTool. Having a HostedImageGenerationTool makes sense to me. Different providers have different ways of exposing the same fundamental information, so being able to write e.g. HostedWebSearchTool, and have that map to the right thing for Gemini and Anthropic and OpenAI makes sense to me. AdditionalProperties can be used in each when there's some setting that's not exposed in a strongly-typed fashion on the tool type.

src/Libraries/Microsoft.Extensions.AI.Abstractions/Image/ImageGenerationTool.cs

src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatCompletion/ChatResponseExtensions.cs

stephentoub · 2025-09-15T15:37:10Z

src/Libraries/Microsoft.Extensions.AI.Abstractions/Image/ImageGenerationTool.cs

+
+namespace Microsoft.Extensions.AI;
+
+/// <summary>Represents a hosted tool that can be specified to an AI service to enable it to perform image generation.</summary>


We already have HostedWebSearchTool, HostedCodeInterpreterTool, HostedFileSearchTool, and HostedMcpServerTool. Having a HostedImageGenerationTool makes sense to me. Different providers have different ways of exposing the same fundamental information, so being able to write e.g. HostedWebSearchTool, and have that map to the right thing for Gemini and Anthropic and OpenAI makes sense to me. AdditionalProperties can be used in each when there's some setting that's not exposed in a strongly-typed fashion on the tool type.

ericstj · 2025-09-16T19:20:45Z

Forgot to mention that https://github.com/ericstj/imageGeneratorSample/tree/imageGeneratorTool is the sample that uses this.

…eGenerationTool

ericstj · 2025-10-01T15:08:56Z

I'm moving this to draft. I have private changes will pick up a self-built copy of OpenAI that has these types public (change is in main of OpenAI, but not yet on NuGet), which we won't be able to merge. I'm also still not happy with the middle-ware and need to experiment more with that.

ericstj · 2025-10-01T22:47:57Z

I'll also add that I know this middle-ware is not thread safe at the moment and it needs to be. I'm currently experimenting how to manage the state. Once I settle on that I'll make it thread-safe since we document that all IChatClient's need to be.

stephentoub · 2025-10-02T02:00:16Z

src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatCompletion/ChatResponseExtensions.cs

+        _ = Throw.IfNull(response);
+        _ = Throw.IfNull(updates);
+
+        if (updates is ICollection<ChatResponseUpdate> { Count: 0 })


Is the existing ToChatResponse then just:

ChatResponse response = new(); response.ApplyUpdates(updates); return response;

or is there a meaningful behavioral or performance difference with that?

stephentoub · 2025-10-02T02:01:25Z

src/Libraries/Microsoft.Extensions.AI.Abstractions/ChatCompletion/ChatResponseExtensions.cs

+                case DataContent dataContent when
+                    !string.IsNullOrEmpty(dataContent.Name):
+                    // Check if there's an existing DataContent with the same name to replace
+                    for (int i = 0; i < message.Contents.Count; i++)


Does this change the algorithm to be O(N^2)?

stephentoub · 2025-10-02T02:04:40Z

src/Libraries/Microsoft.Extensions.AI/ChatCompletion/ImageGeneratingChatClient.cs

+    // the following fields all have scope per-request. They are cleared at the start of each request.
+    private readonly Dictionary<string, List<AIContent>> _imageContentByCallId = [];
+    private readonly Dictionary<string, AIContent> _imageContentById = new(StringComparer.OrdinalIgnoreCase);
+    private ImageGenerationOptions? _imageGenerationOptions;


I would expect instead you'd create an object in Get{Streaming}ResponseAsync to hold the state for that request, and whatever tools you want to send would reference that object (e.g. they'd be instance methods on it).

stephentoub · 2025-10-02T02:06:16Z

src/Libraries/Microsoft.Extensions.AI/ChatCompletion/ImageGeneratingChatClient.cs

+            for (int contentIndex = 0; contentIndex < message.Contents.Count; contentIndex++)
+            {
+                var content = message.Contents[contentIndex];
+                if (content is DataContent dataContent && dataContent.MediaType.StartsWith("image/", StringComparison.OrdinalIgnoreCase))


DataContent.HasTopLevelMediaType is intended for this

ericstj added 6 commits August 19, 2025 08:38

Prototype of using ImageGenerationTool

ffe9a92

Handle DataContent returned from ImageGen

e5edc77

React to rename and improve metadata

2d19cce

Handle image_generation tool content from streaming

5eef474

Add handling for combining updates with images

ff80804

Add tests for new ChatResponseUpdateExtensions

1725ce1

Copilot AI review requested due to automatic review settings August 26, 2025 15:41

ericstj requested a review from a team as a code owner August 26, 2025 15:41

github-actions bot added the area-ai Microsoft.Extensions.AI libraries label Aug 26, 2025

dotnet-policy-service bot assigned ericstj Aug 26, 2025

Copilot AI reviewed Aug 26, 2025

View reviewed changes

PederHP reviewed Sep 15, 2025

View reviewed changes

stephentoub reviewed Sep 15, 2025

View reviewed changes

ericstj mentioned this pull request Sep 16, 2025

Python: OpenAI Responses Image Generation Tool + Added Samples microsoft/agent-framework#763

Closed

4 tasks

ericstj added 10 commits September 19, 2025 16:56

Merge branch 'main' of https://github.com/dotnet/extensions into Imag…

c44f5fb

…eGenerationTool

Rename ImageGenerationTool to HostedImageGenerationTool

b4fe94b

Remove ChatResponseUpdateCoalescingOptions

06bfa30

Add ImageGeneratingChatClient

ca8b15d

Fix namespace of tool

62e0ac5

Replace traces of function calling

81e6e5a

More namepsace fix

6559a66

Enable editing

398bbdb

Merge branch 'main' of https://github.com/dotnet/extensions into Imag…

ac2de35

…eGenerationTool

Update to preview OpenAI with image tool support

1d96532

ericstj marked this pull request as draft October 1, 2025 15:07

stephentoub reviewed Oct 2, 2025

View reviewed changes

ericstj added 4 commits October 3, 2025 13:32

Temporary OpenAI feed

6a6ffa2

Fix tests

94ceab2

Add integration tests for ImageGeneratingChatClient

96e9747

Remove ChatRole.Tool -> Assistant workaround

9ddc91a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Image generation tool #6749

Image generation tool #6749

Uh oh!

ericstj commented Aug 26, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PederHP Sep 15, 2025

Uh oh!

stephentoub Sep 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephentoub Sep 15, 2025

Uh oh!

ericstj commented Sep 16, 2025

Uh oh!

ericstj commented Oct 1, 2025 •

edited

Loading

Uh oh!

ericstj commented Oct 1, 2025

Uh oh!

stephentoub Oct 2, 2025

Uh oh!

stephentoub Oct 2, 2025

Uh oh!

stephentoub Oct 2, 2025

Uh oh!

stephentoub Oct 2, 2025

Uh oh!

Uh oh!


		namespace Microsoft.Extensions.AI;

		/// <summary>Represents a hosted tool that can be specified to an AI service to enable it to perform image generation.</summary>

Image generation tool #6749

Are you sure you want to change the base?

Image generation tool #6749

Uh oh!

Conversation

ericstj commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Microsoft Reviewers: Open in CodeFlow

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PederHP Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

stephentoub Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephentoub Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

ericstj commented Sep 16, 2025

Uh oh!

ericstj commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericstj commented Oct 1, 2025

Uh oh!

stephentoub Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

stephentoub Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

stephentoub Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

stephentoub Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ericstj commented Aug 26, 2025 •

edited

Loading

ericstj commented Oct 1, 2025 •

edited

Loading