new ad create enhanced tool to validate and create multiple detectors

[amitgalitz]

Description

This PR introduces CreateAnomalyDetectorToolEnhanced, a tool that leverages LLM capabilities to automatically generate, validate, and create anomaly detectors for different given indices

Implementation Overview

The tool implements a step-by-step processing flow that handles batch operations on indices while maintaining robust error handling and validation mechanisms.

Code Flow

1. Extract & validate indices →
2. Get index mappings →
3. LLM generates detector config →
4. Validate detector structure →
5. Optimize parameters (suggest API) →
6. Validate model requirements →
7. Create & start detector

The implementation includes utilizing an LLM configuration generation, and retrying on different stages with the validate and suggest api to get optimal configuration that will lead to detector initializing
detector reliability.

The tool processes indices sequentially to prevent LLM throttling and implements retry logic with LLM-assisted error correction. Each index undergoes mapping analysis, LLM configuration generation, detector validation, suggest api based changes, and model validation before final deployment.

Usage Example

Input:

{
  "indices": ["ecommerce-data", "server-logs"]
}

Output:

{
  "ecommerce-data": {
    "status": "success",
    "detectorId": "abc123",
    "detectorName": "ecommerce-data-detector-xyz",
    "createResponse": "Detector created successfully",
    "startResponse": "Detector started successfully"
  },
  "server-logs": {
    "status": "failed_validation",
    "error": "Insufficient data for model training"
  }
}

Processing Flow

The implementation follows a step-by-step flow for each index: extraction and validation, mapping retrieval, LLM configuration generation, detector structure validation, parameter optimization via suggest API, model requirement validation, and finally detector creation and activation. The tool maintains state consistency across batch operations and writes it to the detector result object that is returned as a string so an LLM can use this as one of its tools

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• Commits are signed per the DCO using --signoff.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dbwiddis]

I think we have better detection of System indices than the leading dot. Example usage.

[dbwiddis]

My knee-jerk reaction on seeing a loop that can be streamified:

Suggested change

	Set<String> result = new HashSet<>();
	for (Map.Entry<String, String> entry : fieldsToType.entrySet()) {
	String value = entry.getValue();
	if (value.equals("date") || value.equals("date_nanos")) {
	result.add(entry.getKey());
	}
	}
	return result;
	Set<String> dateTypes = Set.of("date", "date_nanos");
	return fieldsToType.entrySet().stream()
	.filter(e -> dateTypes.contains(e.getValue()))
	.map(Entry::getKey)
	.collect(Collectors.toSet());

But since you're just moving an existing method probably ok to leave as is.

[dbwiddis]

This hard-coded parsing and relying on exception handling seems like it could be improved. At least add a comment showing the expected format, so if this breaks in the future, the next maintainer (or future-you) can quickly identify the format change without walking through the code.

[dbwiddis]

No exception handling for empty outputs/tensors.

[dbwiddis]

Lots of duplicated code between retryDetectorValidation() and retryModelValidation, consider a helper method for the common parts.

[dbwiddis]

This seems to be a really strict regex that would fail on things like whitespace or the keys being in a different order (and cause retries which lose the benefit of the optimal date field.).

Given we're just extracting characters between known strings, it seems there's likely a better way of iterating between the | characters, splitting on =, and trimming the key/value into a map?

[dbwiddis]

This conditional (currentRetry >= MAX_DETECTOR_VALIDATION_RETRIES) should be a fast-fail at the top of the validateDetectorPhase method, otherwise you spend a lot of time validating the detector which, if validation succeeds, you suddenly can't use.

[dbwiddis]

Is starting with "detector" always blocking?

Is there some helper method somewhere which defines blocking errors that you can use?

[dbwiddis]

If detector.getIndices().get(0) throws an exception the listener will never be completed. Probably need some exception handling here.

[dbwiddis]

These seem like magic numbers. Looks like minutes in a day?

At a minimum needs a comment.

Perhaps move the strings into a constant:

// Time Series values from IntervalCalculation
private static final Set<String> KNOWN_HIGH_INTERVAL_STRINGS = 
    IntStream.of(4, 8, 12, 24)
        .map(hours -> hours * 60)
        .mapToObj(String::valueOf)
        .collect(Collectors.toUnmodifiableSet());

And then

boolean isUnreasonableInterval = validationError.contains("interval")
    && KNOWN_HIGH_INTERVAL_STRINGS.stream()
        .anyMatch(validationError::contains);

Alternately just parse the interval from the string and compare it >= 240.

[dbwiddis]

Also, highlighting the fact that magic numbers are generally bad, the INTERVAL_LADDER I linked to includes 720 and not 960. :)

[kaituo]

partial review

[kaituo]

Do you support remote index?

[kaituo]

operational monitoring
the same operational issue

It may not be just operational like UBI data for business/marketing.

[kaituo]

Mixed success/error fields

What if users want to monitor total traffic instead of just errors?

[kaituo]

e.g., user_agent.keyword, message.keyword

What if I want to monitor the message volume and the number of agents running?

[kaituo]

e.g., version.keyword, environment.keyword

We are not sure whether count is good or not until we know the meaning of environment.

[kaituo]

OpenSearch Anomaly Detection supports ONLY these 5 aggregation methods:

We can add cardinality aggregation that is the unique count.

[kaituo]

aggregation_field=field1,field2|aggregation_method=method1,method2

Should you be upfront on what happens for 1 feature (e.g., should it produce aggregation_method=mehtod1, or aggregation_method=method1?

[kaituo]

category_field=field_name_or_empty

Should we tell LLM what happens when there is no categorical field?

[kaituo]

interval=minutes

Should we tell LLM what happens when there are no suitable interval found?

[kaituo]

If validation suggests intervals >120 minutes

Where do you put validation suggested interval in the prompt? I only saw that in the error message of fix prompt, not the initial default prompt.

[kaituo]

can immediately investigate and resolve

Can you define what is the criteria by which the team can investigate and resolve?

[kaituo]

Does the agent have memory what you have asked before? If yes, where is the previous conversation stored and how does agent decide which conversation to give to LLM?

[kaituo]

The agent should be equipped with index search tool as well, right? If so, can you point me the code?

[kaituo]

It is hard to maintain two prompts with almost identical content. Can you try to generate the claude one programmatically by adding extra information?

[kaituo]

Users may have huge index mapping. Do you have a max limit?

[kaituo]

@SuppressWarnings should be outside of comment block. This may cause CI compile failure.

[kaituo]

Where do you use owasp?

[kaituo]

Would this override LLM's interval? If yes, is this on purpose? why do we need LLM's interval suggestion?

[kaituo]

Would this override LLM's interval? If yes, is this on purpose? why do we need LLM's interval suggestion?