Add ml-commons passthrough post process function

[q-andy]

Description

Adds a predefined post-process function to unwrap model output when making a remote call to second ml-commons cluster. When calling a remote second ml-commons predict API with a remote connector, the remote connector output will "double wrap" the second ml-commons. Tested compatibility with neural-search SparseEncodingProcessor and TextEmbeddingProcessor.

Predict call without post process function (double wrapped):

{
	"inference_results": [
		{
			"output": [
				{
					"name": "output",
					"dataAsMap": {
						"inference_results": [
							{
								"output": [
									{
										"name": "output",
										"dataAsMap": {
											"response": [
												{
													"this": 0.6228184700012207,
													"harrison": 0.5467907786369324,
													...
												},
												{
													"?": 0.1258760392665863,
													"day": 0.46702781319618225,
													...
												}
											]
										}
									}
								],
								"status_code": 200.0
							}
						]
					}
				}
			],
			"status_code": 200
		}
	]
}

After using post process function:

{
	"inference_results": [
		{
			"output": [
				{
					"name": "output",
					"dataAsMap": {
						"response": [
							{
								"increasingly": 0.028670792,
								"achievements": 0.4906937,
								...
							}
						]
					}
				},
				{
					"name": "output",
					"dataAsMap": {
						"response": [
							{
								"hesse": 0.07568395,
								"greeting": 0.296827,
								...
							}
						]
					}
				}
			],
			"status_code": 200
		}
	]
}

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dhrubo-os]

@q-andy fix spotless :)

[codecov]

Codecov Report

❌ Patch coverage is 88.67925% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.82%. Comparing base (d8c6208) to head (f2f6a78).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...RemoteMlCommonsPassthroughPostProcessFunction.java	88.00%	0 Missing and 6 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #4111      +/-   ##
============================================
+ Coverage     81.80%   81.82%   +0.01%     
- Complexity     8847     8866      +19     
============================================
  Files           761      762       +1     
  Lines         38099    38152      +53     
  Branches       4250     4263      +13     
============================================
+ Hits          31168    31217      +49     
+ Misses         5110     5109       -1     
- Partials       1821     1826       +5     

Flag	Coverage Δ
ml-commons	81.82% <88.67%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[q-andy]

Most CI passed, just 2 flaky integ test failures are unrelated to this change. Could you take another look @dhrubo-os @Zhangxunmt

[dhrubo-os]

instead of Map<String, Object> input, Since this parameter represents ML Commons inference results or response data, some better name suggestions would be:

mlCommonsResponse
inferenceResponse
responseData
modelResponse

[q-andy]

Sure, didn't realize you could rename the parameter variables of an overridden function. Changed to mlCommonsResponse

[dhrubo-os]

Can we add example in the doc for better understanding?

[dhrubo-os]

as this is a special kind, can we please add a comment here?

[dhrubo-os]

IllegalArgumentException is a 500 level error. Should we treat this as a 5xx error?

[dhrubo-os]

Also what is the reason for leaving as null?

[q-andy]

I left as null instead of treating like 5xx is because even if the model data type is invalid, there may be usecases where we can still parse and use the model data or dataAsMap. E.g. some model types like neural sparse or NER don't include data type as part of the model response, so null is still valid.

Right now since we're focused on neural sparse and dense, my thought process is its better to leave it flexible to be able to possible handle different model response formats. For example, in the future we me add a new datatype for dense models and we might inference that from an older version of ml-commons: perhaps we can still use the data by casting the data at the processor level. Updated the comment to explain this.

[dhrubo-os]

In the else we are sending null for shape, is that expected?

[q-andy]

Yes, e.g. neural sparse and dense models populate different fields, sparse leaves data_type, data, and shape as null.

Dense

{
	"name": "sentence_embedding",
	"data_type": "FLOAT32",
	"shape": [
		768
	],
	"data": [...]
}

Sparse

{
	"name": "output",
	"dataAsMap": {
		"response": [
			{ ... }
		]
	}
}

[dhrubo-os]

what happens if we send data as null to the model Tensor?

[q-andy]

See above comment, this is valid for neural sparse and other remote model format. Looks like data field is primarily used for vector/numerical info, and if a model output doesn't include that, then it uses dataAsMap instead and data field being null is valid.

[dhrubo-os]

Seems like both of the underlying logic could be put in a common method like:

private <T> T[] processNumericArray(Object obj, Class<T> type) {
    if (obj instanceof List<?> list) {
        T[] result = (T[]) Array.newInstance(type, list.size());
        // ... process the list
        return result;
    }
    return null;
}

[mingshl]

after using the post processing function,

the name seems changed, the name used to be "response" but after the post processing become "output". Is this change intended? I am sure if the name is used somewhere but better not change the name if not intentionally.

{
	"inference_results": [
		{
			"output": [
				{
					"name": "output",
					"dataAsMap": {
						"response": [

[q-andy]

the name seems changed, the name used to be "response" but after the post processing become "output". Is this change intended? I am sure if the name is used somewhere but better not change the name if not intentionally.

Fixed, the name is relevant for different model types. I changed it so the name will be passthroughed as well, this is just a typo in the PR description.