[BUG] java.lang.IllegalArgumentException: Must contain at least one column and at least one row (got []/[indices:monitor/recovery])

[Mmuzaf]

What is the bug?

If the security plugin 2.19.3.0 is enabled, information about ongoing and completed shard recoveries cannot be obtained via the /_cat/recovery endpoint because all indices are closed (have been closed manually).

The request fails with:
java.lang.IllegalArgumentException: Must contain at least one column and at least one row (got []/[indices:monitor/recovery])

On 2.17 the same works fine, and setting plugins.security.privileges_evaluation.use_legacy_impl=true solves the issue.

How can one reproduce the bug?

• Start 3-node cluster with a security plugin 2.19.3.0 enabled
• Close all indices
• curl /_cat/recovery

The resolution for such a request yelds an empty list:

        Set<String> resolvedIndicesSet = resolvedIndices.getAllIndicesResolved(
            context.getClusterStateSupplier(),
            context.getIndexNameExpressionResolver());

        // resolvedIndicesSet - empty list

The request fails with the stacktrace in logs:

Unexpected exception java.lang.IllegalArgumentException: Must contain at least one column and at least one row (got []/[indices:monitor/recovery])
java.lang.IllegalArgumentException: Must contain at least one column and at least one row (got []/[indices:monitor/recovery])
        at com.selectivem.collections.CheckTableImpl.create(CheckTableImpl.java:81) ~[special-collections-complete-1.4.0.jar:?]
        at com.selectivem.collections.CheckTable.create(CheckTable.java:75) ~[special-collections-complete-1.4.0.jar:?]
        at org.opensearch.security.privileges.ActionPrivileges.hasIndexPrivilege(ActionPrivileges.java:210) ~[opensearch-security-2.19.3.0.jar:2.19.3.0]
        at org.opensearch.security.privileges.PrivilegesEvaluatorImpl.evaluate(PrivilegesEvaluatorImpl.java:603) ~[opensearch-security-2.19.3.0.jar:2.19.3.0]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:377) [opensearch-security-2.19.3.0.jar:2.19.3.0]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:166) [opensearch-security-2.19.3.0.jar:2.19.3.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.19.3.jar:2.19.3]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) [opensearch-2.19.3.jar:2.19.3]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) [opensearch-2.19.3.jar:2.19.3]

What is the expected behavior?

The request completes fine.

What is your host/environment?

• OS: Linux dev40 6.8.5-301.fc40.aarch64
• OpenSearch 2.19.3
• opensearch security plugin 2.19.3.0

Possible solution

The problem is probably in the following condition in the ActionPrivileges , when is actully not applied when isLocalAll == true (request containing "*").

The solution could be as simple as checking the resolved indices before passing them to the CheckTable:

    if (resolvedLocalIndices.isEmpty()) {
        return PrivilegesEvaluatorResponse.ok();
    }

Please, let me know what do you think and if I need prepare a fix for that.

[Mmuzaf]

@nibix could you please take a look?

[nibix]

@Mmuzaf

thank you for the report - interesting!

I will take a closer look tomorrow; yet, I don't think it's sufficient to check for if (resolvedLocalIndices.isEmpty()) as this cannot be relief on if isLocalAll() is true.

But, likely, one can do the check on resolvedIndices.getAllIndicesResolved(context.getClusterStateSupplier(), context.getIndexNameExpressionResolver()).

Just for the sake of completeness: could you share the non-authc/authz part of the config.yml you are using?

I'd be especially interested in the values for do_not_fail_on_forbidden, do_not_fail_on_forbidden_empty and respect_request_indices_options.

[Mmuzaf]

@nibix thank you for sharing these details.

I created a fix to address this problem: #5797 I assume it might be incomplete.

This is the config that I've used (wihtout authc section). The values do_not_fail_on_forbidden, do_not_fail_on_forbidden_empty and respect_request_indices_options are not used in the config.

[os@os-x23-26f14fa ~]# cat /etc/opensearch/opensearch-security/config.yml 
{
    "_meta": {
        "config_version": 2,
        "type": "config"
    },
    "config": {
        "dynamic": {
            "auth_failure_listeners": {
                "internal_authentication_backend_limiting": {
                    "allowed_tries": 10000,
                    "authentication_backend": "internal",
                    "block_expiry_seconds": 600,
                    "max_blocked_clients": 100000,
                    "max_tracked_clients": 100000,
                    "time_window_seconds": 3600,
                    "type": "username"
                }
            },
            "http": {
                "anonymous_auth_enabled": false,
                "xff": {
                    "enabled": false
                }
            },
            "kibana": {
                "index": ".kibana",
                "multitenancy_enabled": true,
                "server_username": "kibanaserver"
            }
        }
    }
}