Fix missing RAG response from generative_qa_parameters

[rithin-pullela-aws]

Description

Please refer to #4018 for detailed description of the issue.

Problem:

When using search templates with generative_qa_parameters, the RAG answer is missing from the response. This happens because:

1. Direct searches call GenerativeSearchResponse.toXContent() → includes ext block with RAG answer
2. Template searches call GenerativeSearchResponse.innerToXContent() → missing ext block

The core SearchTemplateResponse calls response.innerToXContent() instead of response.toXContent() to preserve the status field, but GenerativeSearchResponse only adds the ext block in toXContent().

Solution:

Override innerToXContent() in GenerativeSearchResponse to include the ext block, ensuring both direct and template searches return RAG responses.

Alternative Considered:
Modify core SearchTemplateResponse to use toXContent(), but this would lose the status field that templates require.

Related Issues

Resolves #4018

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[mingshl]

@rithin-pullela-aws nice way to compatible with search templates!!! can you add unit tests to verify your fix?

also @junqiu-lei can you help review this fix?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.80%. Comparing base (f75ae1a) to head (321c32c).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #4118      +/-   ##
============================================
- Coverage     81.81%   81.80%   -0.01%     
- Complexity     8847     8848       +1     
============================================
  Files           761      761              
  Lines         38099    38103       +4     
  Branches       4250     4250              
============================================
+ Hits          31170    31172       +2     
- Misses         5109     5110       +1     
- Partials       1820     1821       +1     

Flag	Coverage Δ
ml-commons	81.80% <100.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[junqiu-lei]

@rithin-pullela-aws nice way to compatible with search templates!!! can you add unit tests to verify your fix?

also @junqiu-lei can you help review this fix?

Agree, please also add integ test for template search with generative_qa_parameters.

[mingshl]

@rithin-pullela-aws if we are taking this approach, we need all custom search responses with search extensions to have the same fix.

can you add the fix for this search response class too？

ml-commons/plugin/src/main/java/org/opensearch/ml/processor/MLInferenceSearchResponse.java

Line 46 in 71d47e9

innerToXContent(builder, params);

[rithin-pullela-aws]

the fix for this search response class to

@rithin-pullela-aws if we are taking this approach, we need all custom search responses with search extensions to have the same fix.

can you add the fix for this search response class too？

ml-commons/plugin/src/main/java/org/opensearch/ml/processor/MLInferenceSearchResponse.java

Line 46 in 71d47e9

innerToXContent(builder, params);

@mingshl, already included in this PR :)

[mingshl]

the fix LGTM, is it difficult to add in UT and IT to verify the fix? you need create a search template and create a ml inference processor and rag processor and then verify in the search response has extension. Maybe you can add the new IT in here https://github.com/opensearch-project/ml-commons/blob/main/plugin/src/test/java/org/opensearch/ml/rest/RestMLInferenceSearchResponseProcessorIT.java

[rithin-pullela-aws]

the fix LGTM, is it difficult to add in UT and IT to verify the fix? you need create a search template and create a ml inference processor and rag processor and then verify in the search response has extension. Maybe you can add the new IT in here https://github.com/opensearch-project/ml-commons/blob/main/plugin/src/test/java/org/opensearch/ml/rest/RestMLInferenceSearchResponseProcessorIT.java

@mingshl, was busy with other tasks. I will add an IT as you suggested