[Fleet] Add graceful error handling for invalid sortField on GET api/fleet/agents

[Supplementing]

Summary

Closes https://github.com/elastic/ingest-dev/issues/5992

• Added error handling for all fleet endpoints that use ES. ES errors will now bubble up and be properly returned along with their status code response from ES. This will stop errors from being 500 when they are actually something else, we just werent properly handling them at the top level.
• Also added additional test coverage to verify the handler will throw/succeed when its supposed to, and added some simple integration tests to verify the entire flow works as intended.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
• Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• See some risk examples
• ...

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[Supplementing]

@elasticmachine merge upstream

[jen-huang]

I'm not loving this approach. it's extremely specific to this Fleet API & this type of ES error.

I can see from testing a direct ES query that this kind of error already returns 400 from ES. example curl:

curl -X GET -H "Content-Type: application/json" -d '{"sort":[{"invalid_field":{"order":"desc"}}]}' "http://elastic:changeme@localhost:9200/.kibana_ingest/_search" -v

can we explore a solution where the status code & error is just passed directly from ES? even better if we can handle it generically for all agent APIs (since they access ES directly instead of with soClient)

[Supplementing]

I'm not loving this approach. it's extremely specific to this Fleet API & this type of ES error.

can we explore a solution where the status code & error is just passed directly from ES? even better if we can handle it generically for all agent APIs (since they access ES directly instead of with soClient)

Yeah, I like the idea of making it less specific and more inclusive. I think the issue comes from that direct usage of ES and errors not being handled/bubbled up correctly. Let me do some digging to see what needs to be done to fix it higher up and across all the API's

[Supplementing]

I decided to remove the specific check on the handler level and instead extended the FleetError class with a new FleetElasticsearchValidationError that will get caught at the top level for all the endpoints if an error is thrown directly by ES.

I also kept some of the transformation logic as the ES errors can be very generic at the top level with things like 'all shards failed', which is not super helpful to determine the root cause. This approach returns nested errors a bit better.

Nevermind, I removed the opinionated casting and just the ES errors bubble as intended.

This can be tested with the sortField query param on the GET api/fleet/agents endpoint, and also other endpoints such as GET api/fleet/package_policies. They all should handle ES errors gracefully now cc: @jen-huang

[jen-huang]

A few cleanup areas, will come back to do some testing

[Supplementing]

@elasticmachine merge upstream

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 0ef25e1

Failed CI Steps

• FTR Configs #28
• FTR Configs #48

Test Failures

• [job] [logs] FTR Configs #48 / Package policies Package Policy - upgrade when upgrading to a version where a variable has been removed "after each" hook for "successfully upgrades package policy"
• [job] [logs] FTR Configs #28 / task_manager scheduling and running tasks should update schedule for existing task when calling ensureScheduled with a different schedule

Metrics [docs]

✅ unchanged

History

• 💚 Build #351052 succeeded 8eb0c07

[jen-huang]

Small nit wrt to name, but otherwise LGTM. Ty for the changes!

[jen-huang]

I didn't realize this had auto-merge on. @Supplementing see above about the naming ^

[Supplementing]

I didn't realize this had auto-merge on. @Supplementing see above about the naming ^

See quick PR here to rectify: #239640