[GRPC] Publish transport-grpc-spi for QueryBuilderProtoConverter and QueryBuilderProtoConverterRegistry (#18949)

---------

Signed-off-by: Karen Xu <[email protected]>
Signed-off-by: Karen X <[email protected]>
Signed-off-by: Michael Froh <[email protected]>
Co-authored-by: Michael Froh <[email protected]>

Author: karenyrx

CHANGELOG.md

@@ -16,6 +16,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add overload constructor for Translog to accept Channel Factory as a parameter ([#18918](https://github.com/opensearch-project/OpenSearch/pull/18918))
 - Add subdirectory-aware store module with recovery support ([#19132](https://github.com/opensearch-project/OpenSearch/pull/19132))
 - Add a dynamic cluster setting to control the enablement of the merged segment warmer ([#18929](https://github.com/opensearch-project/OpenSearch/pull/18929))
+- Publish transport-grpc-spi exposing QueryBuilderProtoConverter and QueryBuilderProtoConverterRegistry ([#18949](https://github.com/opensearch-project/OpenSearch/pull/18949))
 - Support system generated search pipeline. ([#19128](https://github.com/opensearch-project/OpenSearch/pull/19128))
 
 ### Changed

gradle/libs.versions.toml

@@ -21,6 +21,7 @@ woodstox          = "6.4.0"
 kotlin            = "1.7.10"
 antlr4            = "4.13.1"
 guava             = "33.2.1-jre"
+opensearchprotobufs = "0.6.0"
 protobuf          = "3.25.8"
 jakarta_annotation = "1.3.5"
 google_http_client = "1.44.1"

modules/transport-grpc/build.gradle

@@ -21,6 +21,7 @@ testClusters {
 }
 
 dependencies {
+  api project('spi')
   compileOnly "com.google.code.findbugs:jsr305:3.0.2"
   runtimeOnly "com.google.guava:guava:${versions.guava}"
   implementation "com.google.errorprone:error_prone_annotations:2.24.1"

modules/transport-grpc/spi/README.md

@@ -0,0 +1,277 @@
+# transport-grpc-spi
+
+Service Provider Interface (SPI) for the OpenSearch gRPC transport module. This module provides interfaces and utilities that allow external plugins to extend the gRPC transport functionality.
+
+## Overview
+
+The `transport-grpc-spi` module enables plugin developers to:
+- Implement custom query converters for gRPC transport
+- Extend gRPC protocol buffer handling
+- Register custom query types that can be processed via gRPC
+
+## Key Components
+
+### QueryBuilderProtoConverter
+
+Interface for converting protobuf query messages to OpenSearch QueryBuilder objects.
+
+```java
+public interface QueryBuilderProtoConverter {
+    QueryContainer.QueryContainerCase getHandledQueryCase();
+    QueryBuilder fromProto(QueryContainer queryContainer);
+}
+```
+
+### QueryBuilderProtoConverterRegistry
+
+Interface for accessing the query converter registry. This provides a clean abstraction for plugins that need to convert nested queries without exposing internal implementation details.
+
+## Usage for Plugin Developers
+
+### 1. Add Dependency
+
+Add the SPI dependency to your plugin's `build.gradle`:
+
+```gradle
+dependencies {
+    compileOnly 'org.opensearch.plugin:transport-grpc-spi:${opensearch.version}'
+    compileOnly 'org.opensearch:protobufs:${protobufs.version}'
+}
+```
+
+### 2. Implement Custom Query Converter
+
+```java
+public class MyCustomQueryConverter implements QueryBuilderProtoConverter {
+
+    @Override
+    public QueryContainer.QueryContainerCase getHandledQueryCase() {
+        return QueryContainer.QueryContainerCase.MY_CUSTOM_QUERY;
+    }
+
+    @Override
+    public QueryBuilder fromProto(QueryContainer queryContainer) {
+        // Convert your custom protobuf query to QueryBuilder
+        MyCustomQuery customQuery = queryContainer.getMyCustomQuery();
+        return new MyCustomQueryBuilder(customQuery.getField(), customQuery.getValue());
+    }
+}
+```
+
+### 3. Register Your Converter
+
+In your plugin's main class, return the converter from createComponents:
+
+```java
+public class MyPlugin extends Plugin {
+
+    @Override
+    public Collection<Object> createComponents(Client client, ClusterService clusterService,
+                                             ThreadPool threadPool, ResourceWatcherService resourceWatcherService,
+                                             ScriptService scriptService, NamedXContentRegistry xContentRegistry,
+                                             Environment environment, NodeEnvironment nodeEnvironment,
+                                             NamedWriteableRegistry namedWriteableRegistry,
+                                             IndexNameExpressionResolver indexNameExpressionResolver,
+                                             Supplier<RepositoriesService> repositoriesServiceSupplier) {
+
+        // Return your converter instance - the transport-grpc plugin will discover and register it
+        return Collections.singletonList(new MyCustomQueryConverter());
+    }
+}
+```
+
+**Step 3b: Create SPI Registration File**
+
+Create a file at `src/main/resources/META-INF/services/org.opensearch.transport.grpc.spi.QueryBuilderProtoConverter`:
+
+```
+org.opensearch.mypackage.MyCustomQueryConverter
+```
+
+**Step 3c: Declare Extension in Plugin Descriptor**
+
+In your `plugin-descriptor.properties`, declare that your plugin extends transport-grpc:
+
+```properties
+extended.plugins=transport-grpc
+```
+
+### 4. Accessing the Registry (For Complex Queries)
+
+If your converter needs to handle nested queries (like k-NN's filter clause), you'll need access to the registry to convert other query types. The transport-grpc plugin will inject the registry into your converter.
+
+```java
+public class MyCustomQueryConverter implements QueryBuilderProtoConverter {
+
+    private QueryBuilderProtoConverterRegistry registry;
+
+    @Override
+    public void setRegistry(QueryBuilderProtoConverterRegistry registry) {
+        this.registry = registry;
+    }
+
+    @Override
+    public QueryBuilder fromProto(QueryContainer queryContainer) {
+        MyCustomQuery customQuery = queryContainer.getMyCustomQuery();
+
+        MyCustomQueryBuilder builder = new MyCustomQueryBuilder(
+            customQuery.getField(),
+            customQuery.getValue()
+        );
+
+        // Handle nested queries using the injected registry
+        if (customQuery.hasFilter()) {
+            QueryContainer filterContainer = customQuery.getFilter();
+            QueryBuilder filterQuery = registry.fromProto(filterContainer);
+            builder.filter(filterQuery);
+        }
+
+        return builder;
+    }
+}
+```
+
+**Registry Injection Pattern**
+
+**How k-NN Now Accesses Built-in Converters**:
+
+The gRPC plugin **injects the populated registry** into converters that need it:
+
+```java
+// 1. Converter interface has a default setRegistry method
+public interface QueryBuilderProtoConverter {
+    QueryBuilder fromProto(QueryContainer queryContainer);
+
+    default void setRegistry(QueryBuilderProtoConverterRegistry registry) {
+        // By default, converters don't need a registry
+        // Converters that handle nested queries should override this method
+    }
+}
+
+// 2. GrpcPlugin injects registry into loaded extensions
+for (QueryBuilderProtoConverter converter : queryConverters) {
+    // Inject the populated registry into the converter
+    converter.setRegistry(queryRegistry);
+
+    // Register the converter
+    queryRegistry.registerConverter(converter);
+}
+```
+
+**Registry Access Pattern for Converters with Nested Queries**:
+```java
+public class KNNQueryBuilderProtoConverter implements QueryBuilderProtoConverter {
+
+    private QueryBuilderProtoConverterRegistry registry;
+
+    @Override
+    public void setRegistry(QueryBuilderProtoConverterRegistry registry) {
+        this.registry = registry;
+        // Pass the registry to utility classes that need it
+        KNNQueryBuilderProtoUtils.setRegistry(registry);
+    }
+
+    @Override
+    public QueryBuilder fromProto(QueryContainer queryContainer) {
+        // The utility class can now convert nested queries using the injected registry
+        return KNNQueryBuilderProtoUtils.fromProto(queryContainer.getKnn());
+    }
+}
+```
+
+
+## Testing
+
+### Unit Tests
+
+```bash
+./gradlew :modules:transport-grpc:spi:test
+```
+
+### Testing Your Custom Converter
+
+```java
+@Test
+public void testCustomQueryConverter() {
+    MyCustomQueryConverter converter = new MyCustomQueryConverter();
+
+    // Create test protobuf query
+    QueryContainer queryContainer = QueryContainer.newBuilder()
+        .setMyCustomQuery(MyCustomQuery.newBuilder()
+            .setField("test_field")
+            .setValue("test_value")
+            .build())
+        .build();
+
+    // Convert and verify
+    QueryBuilder result = converter.fromProto(queryContainer);
+    assertThat(result, instanceOf(MyCustomQueryBuilder.class));
+
+    MyCustomQueryBuilder customQuery = (MyCustomQueryBuilder) result;
+    assertEquals("test_field", customQuery.fieldName());
+    assertEquals("test_value", customQuery.value());
+}
+```
+
+## Real-World Example: k-NN Plugin
+See the k-NN plugin https://github.com/opensearch-project/k-NN/pull/2833/files for an example on how to use this SPI, including handling nested queries.
+
+**1. Dependency in build.gradle:**
+```gradle
+compileOnly "org.opensearch.plugin:transport-grpc-spi:${opensearch.version}"
+compileOnly "org.opensearch:protobufs:0.8.0"
+```
+
+**2. Converter Implementation with Registry Access:**
+```java
+public class KNNQueryBuilderProtoConverter implements QueryBuilderProtoConverter {
+
+    private QueryBuilderProtoConverterRegistry registry;
+
+    @Override
+    public void setRegistry(QueryBuilderProtoConverterRegistry registry) {
+        this.registry = registry;
+    }
+
+    @Override
+    public QueryContainer.QueryContainerCase getHandledQueryCase() {
+        return QueryContainer.QueryContainerCase.KNN;
+    }
+
+    @Override
+    public QueryBuilder fromProto(QueryContainer queryContainer) {
+        KnnQuery knnQuery = queryContainer.getKnn();
+
+        KNNQueryBuilder builder = new KNNQueryBuilder(
+            knnQuery.getField(),
+            knnQuery.getVectorList().toArray(new Float[0]),
+            knnQuery.getK()
+        );
+
+        // Handle nested filter query using injected registry
+        if (knnQuery.hasFilter()) {
+            QueryContainer filterContainer = knnQuery.getFilter();
+            QueryBuilder filterQuery = registry.fromProto(filterContainer);
+            builder.filter(filterQuery);
+        }
+
+        return builder;
+    }
+}
+```
+
+**3. Plugin Registration:**
+```java
+// In KNNPlugin.createComponents()
+KNNQueryBuilderProtoConverter knnQueryConverter = new KNNQueryBuilderProtoConverter();
+return ImmutableList.of(knnStats, knnQueryConverter);
+```
+
+**4. SPI File:**
+```
+# src/main/resources/META-INF/services/org.opensearch.transport.grpc.spi.QueryBuilderProtoConverter
+org.opensearch.knn.grpc.proto.request.search.query.KNNQueryBuilderProtoConverter
+```
+
+**Why k-NN needs the registry:**
+The k-NN query's `filter` field is a `QueryContainer` protobuf type that can contain any query type (MatchAll, Term, Terms, etc.). The k-NN converter needs access to the registry to convert these nested queries to their corresponding QueryBuilder objects.

modules/transport-grpc/spi/build.gradle

@@ -0,0 +1,22 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+apply plugin: 'opensearch.build'
+apply plugin: 'opensearch.publish'
+
+base {
+  group = 'org.opensearch.plugin'
+  archivesName = 'transport-grpc-spi'
+}
+
+dependencies {
+  api project(":server")
+  api "org.opensearch:protobufs:${versions.opensearchprotobufs}"
+
+  testImplementation project(":test:framework")
+}

modules/transport-grpc/spi/licenses/protobufs-0.6.0.jar.sha1

@@ -0,0 +1 @@
+1675c5085e1376fd1a107b87f7e325944ab5b4dc