Support multi-tenancy for LocalRegexGuardrail

[yuancu]

Description

This PR Introduce add multi-tenancy support to LocalRegexGuardrail by introducing sdk client to perform search requests.

Related Issues

Resolves #4119

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[xinyual]

LGTM.

[zane-neo]

It seems the if...else can be merged, and check the other places using SdkClient, try to always use the same approach like:

 try (ThreadContext.StoredContext context = client.threadPool().getThreadContext().stashContext()) {
            sdkClient.getDataObjectAsync(getDataObjectRequest).whenComplete((

Without this try block, it could cause issue some edge cases.

[yuancu]

By merging the if...else, do you mean I can always stash and restore the thread context no matter whether it is a stop word system index or not?

Actually I don't really know why it has to stash and restore the thread context when the index is a system index, but does not have to do so when the index is not. Explanations will be helpful

[codecov]

Codecov Report

❌ Patch coverage is 35.13514% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.51%. Comparing base (a51e2a8) to head (82fccd4).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...pensearch/ml/common/model/LocalRegexGuardrail.java	21.73%	18 Missing ⚠️
...n/java/org/opensearch/ml/model/MLModelManager.java	14.28%	6 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #4120      +/-   ##
============================================
- Coverage     81.52%   81.51%   -0.01%     
- Complexity     8800     8806       +6     
============================================
  Files           761      761              
  Lines         38102    38108       +6     
  Branches       4248     4250       +2     
============================================
+ Hits          31061    31063       +2     
- Misses         5196     5199       +3     
- Partials       1845     1846       +1     

Flag	Coverage Δ
ml-commons	81.51% <35.13%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[zane-neo]

Code coverage failed: https://github.com/opensearch-project/ml-commons/pull/4120/checks?check_run_id=48455690124, please take a look and add more tests to cover.

[yuancu]

Hi @zane-neo , I updated implementations and fixed tests. Can you help approve the CI workflows?