Resolve Cluster API (#102726)

To improve cross-cluster search user experience, Kibana needs an endpoint that is accessible
by arbitrary Kibana dashboard search users and provides:

1. a listing of clusters in scope for a CCS query (based on the index expression and whether 
there are any indices on each cluster that the Kibana user has access to query).
2. whether that cluster is currently connected to the querying cluster (will it come back as 
skipped or failed in a CCS search)
3. showing the skip_unavailable setting for those clusters (so you can know whether it will
return skipped or failed in a CCS search)
4. the ES version of the cluster

Since no single Elasticsearch endpoint provides all of these features, this PR creates a new endpoint `_resolve/cluster` that works along side the existing `_resolve/index` endpoint 
(and leverages some of its features).

Example usage against a cluster with 2 remote clusters configured:

GET /_resolve/cluster/*,remote*:bl*

Response:

{
  "(local)": {
    "connected": true,
    "skip_unavailable": false,
    "matching_indices": true,
    "version": {
      "number": "8.12.0-SNAPSHOT",
      "build_flavor": "default",
      "minimum_wire_compatibility_version": "7.17.0",
      "minimum_index_compatibility_version": "7.0.0"
    }
  },
  "remote2": {
    "connected": true,
    "skip_unavailable": true,
    "matching_indices": true,
    "version": {
      "number": "8.12.0-SNAPSHOT",
      "build_flavor": "default",
      "minimum_wire_compatibility_version": "7.17.0",
      "minimum_index_compatibility_version": "7.0.0"
    }
  },
  "remote1": {
    "connected": true,
    "skip_unavailable": false,
    "matching_indices": false,
    "version": {
      "number": "8.12.0-SNAPSHOT",
      "build_flavor": "default",
      "minimum_wire_compatibility_version": "7.17.0",
      "minimum_index_compatibility_version": "7.0.0"
    }
  }
}

Almost all errors show up as "error" entries in the response.
Only the local SecurityException returns a 403 since that happens before the ResolveCluster
Transport code kicks in.

Author: quux00

docs/changelog/102726.yaml

@@ -0,0 +1,5 @@
+pr: 102726
+summary: Resolve Cluster API
+area: Search
+type: enhancement
+issues: []

docs/reference/indices.asciidoc

@@ -20,6 +20,7 @@ index settings, aliases, mappings, and index templates.
 * <<indices-rollover-index>>
 * <<unfreeze-index-api>>
 * <<indices-resolve-index-api>>
+* <<indices-resolve-cluster-api>>
 * <<indices-downsample-data-stream>>
 
 [discrete]
@@ -130,6 +131,7 @@ include::indices/dangling-indices-list.asciidoc[]
 include::indices/open-close.asciidoc[]
 include::indices/refresh.asciidoc[]
 include::indices/resolve.asciidoc[]
+include::indices/resolve-cluster.asciidoc[]
 include::indices/rollover-index.asciidoc[]
 include::indices/shrink-index.asciidoc[]
 include::indices/simulate-index.asciidoc[]

docs/reference/indices/resolve-cluster.asciidoc

@@ -0,0 +1,265 @@
+[[indices-resolve-cluster-api]]
+=== Resolve cluster API
+++++
+<titleabbrev>Resolve cluster</titleabbrev>
+++++
+
+Resolves the specified index expressions to return information about
+each cluster, including the local cluster, if included.
+
+This endpoint is useful before doing a <<modules-cross-cluster-search,{ccs}>> in
+order to determine which remote clusters should be included in a search.
+
+You use the same index expression with this endpoint as you would for cross-cluster
+search. Index and <<exclude-problematic-clusters,cluster exclusions>> are also supported
+with this endpoint.
+
+For each cluster in the index expression, information is returned about:
+
+1. whether the querying ("local") cluster is currently connected to each remote cluster
+   in the index expression scope
+2. whether each remote cluster is configured with `skip_unavailable` as `true` or `false`
+3. whether there are any indices, aliases or data streams on that cluster that match
+   the index expression
+4. whether the search is likely to have errors returned when you do the {ccs} (including any
+   authorization errors if your user does not have permission to query the index)
+5. cluster version information, including the Elasticsearch server version
+
+////
+[source,console]
+--------------------------------
+PUT _cluster/settings
+{
+  "persistent": {
+    "cluster": {
+      "remote": {
+        "cluster_one": {
+          "seeds": [
+            "35.238.149.1:9300"
+          ],
+          "skip_unavailable": true
+        },
+        "cluster_two": {
+          "seeds": [
+            "35.238.149.2:9300"
+          ],
+          "skip_unavailable": false
+        }
+      }
+    }
+  }
+}
+--------------------------------
+// TEST[setup:host]
+// TEST[s/35.238.149.\d+:930\d+/\${transport_host}/]
+////
+
+[source,console]
+----
+GET /_resolve/cluster/my-index-*,cluster*:my-index-*
+----
+// TEST[continued]
+
+This will return information about the local cluster and all remotely configured
+clusters that start with the alias `cluster*`. Each cluster will return information
+about whether it has any indices, aliases or data streams that match `my-index-*`.
+
+[[resolve-cluster-api-request]]
+==== {api-request-title}
+
+`GET /_resolve/cluster/<index_expression>`
+
+[[resolve-cluster-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the
+`view_index_metadata`, `read`, or `manage` <<privileges-list-indices,index
+privilege>> for the target data stream, index, or alias.
+
+[[resolve-cluster-api-path-params]]
+==== {api-path-parms-title}
+
+`<index_expression>`::
++
+--
+(Required, string) Comma-separated name(s) or index pattern(s) of the
+indices, aliases, and data streams to resolve, using <<api-multi-index>>.
+Resources on <<remote-clusters,remote clusters>> can be specified using the
+`<cluster>:<name>` syntax.
+--
+
+[[resolve-cluster-api-query-params]]
+==== {api-query-parms-title}
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
++
+Defaults to `open`.
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
++
+Defaults to `false`.
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
++
+Defaults to `true`.
+
+`ignore_throttled`::
+(Optional, Boolean) If `true`, concrete, expanded or aliased indices are
+ignored when frozen. Defaults to `false`.
++
+deprecated:[7.16.0]
+
+
+[discrete]
+[[usecases-for-resolve-cluster]]
+=== Examples of why this endpoint is useful before doing a {ccs}.
+
+You may want to exclude a cluster or index from a search when:
+
+1. A remote cluster is not currently connected and is configured with `skip_unavailable`=`false`.
+Executing a {ccs} under those conditions will cause
+<<cross-cluster-search-failures,the entire search to fail>>.
+
+2. A cluster has no matching indices, aliases or data streams for the index expression
+(or your user does not have permissions to search them). For example, suppose your
+index expression is `logs*,remote1:logs*` and the `remote1` cluster has no indices, aliases or data
+streams that match `logs*`. In that case, that cluster will return no results from that cluster if
+you include it in a {ccs}.
+
+3. The index expression (combined with any query parameters you specify) will likely cause an exception
+to be thrown when you do the search. In these cases, the "error" field in the `_resolve/cluster` response
+will be present. (This is also where security/permission errors will be shown.)
+
+4. A remote cluster is an older version that does not support the feature you want to
+use in your search.
+
+
+[[resolve-cluster-api-example]]
+==== {api-examples-title}
+
+[source,console]
+----
+GET /_resolve/cluster/my-index*,clust*:my-index*
+----
+// TEST[continued]
+// TEST[setup:my_index]
+
+The API returns the following response:
+
+[source,console-result]
+----
+{
+  "(local)": {          <1>
+    "connected": true,
+    "skip_unavailable": false,
+    "matching_indices": true,
+    "version": {
+      "number": "8.13.0",
+      "build_flavor": "default",
+      "minimum_wire_compatibility_version": "7.17.0",
+      "minimum_index_compatibility_version": "7.0.0"
+    }
+  },
+  "cluster_one": {
+    "connected": true,         <2>
+    "skip_unavailable": true,  <3>
+    "matching_indices": true,  <4>
+    "version": {
+      "number": "8.13.0",      <5>
+      "build_flavor": "default",
+      "minimum_wire_compatibility_version": "7.17.0",
+      "minimum_index_compatibility_version": "7.0.0"
+    }
+  },
+  "cluster_two": {
+    "connected": true,
+    "skip_unavailable": false,
+    "matching_indices": true,
+    "version": {
+      "number": "8.13.0",
+      "build_flavor": "default",
+      "minimum_wire_compatibility_version": "7.17.0",
+      "minimum_index_compatibility_version": "7.0.0"
+    }
+  }
+}
+----
+// TESTRESPONSE[s/"number": "8.13.0"/"number": "$body.$_path"/]
+// TESTRESPONSE[s/"minimum_wire_compatibility_version": "7.17.0"/"minimum_wire_compatibility_version": "$body.$_path"/]
+// TESTRESPONSE[s/"minimum_index_compatibility_version": "7.0.0"/"minimum_index_compatibility_version": "$body.$_path"/]
+
+<1> Each cluster has its own response section. The cluster you sent the request to is labelled as "(local)".
+<2> The querying cluster attempts to make a request to each remote cluster. If successful, `connected`=`true`.
+<3> The `skip_unavailable` setting for each remote cluster, as configured on the local cluster.
+<4> Indicates whether any index, alias or data stream matches the index expression specified for that cluster.
+<5> The Elasticsearch server version.
+
+
+[discrete]
+[[resolve-cluster-api-error-example]]
+==== An example showing potential problems with your search set up
+
+The following request shows examples of several reasons why you might want
+to change your query to avoid unsuccessful searches.
+
+[source,console]
+--------------------------------------------------
+GET /_resolve/cluster/not-present,clust*:my-index*,oldcluster:*?ignore_unavailable=false
+--------------------------------------------------
+// TEST[continued]
+// TEST[s/,oldcluster:*//]
+
+[source,console-result]
+--------------------------------------------------
+{
+  "(local)": {
+    "connected": true,
+    "skip_unavailable": false,
+    "error": "no such index [not_present]"  <1>
+  },
+  "cluster_one": {
+    "connected": true,
+    "skip_unavailable": true,
+    "matching_indices": false,    <2>
+    "version": {
+      "number": "8.13.0",
+      "build_flavor": "default",
+      "minimum_wire_compatibility_version": "7.17.0",
+      "minimum_index_compatibility_version": "7.0.0"
+    }
+  },
+  "cluster_two": {
+    "connected": false,           <3>
+    "skip_unavailable": false,
+    "matching_indices": true,
+    "version": {
+      "number": "8.13.0",
+      "build_flavor": "default",
+      "minimum_wire_compatibility_version": "7.17.0",
+      "minimum_index_compatibility_version": "7.0.0"
+    }
+  },
+  "oldcluster": {         <4>
+    "connected": true,
+    "skip_unavailable": false,
+    "matching_indices": true
+  }
+}
+--------------------------------------------------
+// TEST[skip: too many illustrative error variations to reproduce]
+
+<1> The local cluster has no index called `not_present` and if you try a search against it
+using the specified `ignore_unavailable=false` param, you will get an "no such index" error.
+Other types of errors can show up here as well, such as security permission errors when your user does
+not have authorization to search the specified index.
+<2> The `cluster_one` remote cluster does not have any indices that match the pattern `my-index*`.
+That could be due to there being no indices present that match the pattern or that the index
+is closed. (You can check this by using the <<indices-resolve-index-api,resolve index>> API.)
+<3> The `cluster_two` remote cluster is not connected (the attempt to connect failed). Since this
+cluster is marked as `skip_unavailable=false`, you should probably exclude this cluster from the
+search by adding `-cluster_two:*` to the search index expression.
+<4> The `oldcluster` remote cluster shows that it has matching_indices, but no version information
+is included. That means that it is running a version older than when the `_resolve/cluster` API
+was added (8.13.0), so you may want to exclude it from your {ccs}. (Note: the endpoint was able to tell
+there were matching indices because it fell back to using the <<indices-resolve-index-api,resolve index>>
+API.)