Skip to content

Commit f383882

Browse files
committed
[GRPC] Documentation for 3.2 GRPC GA (Bulk + KNN Search)
Signed-off-by: Karen Xu <[email protected]>
1 parent 4a36395 commit f383882

File tree

5 files changed

+299
-155
lines changed

5 files changed

+299
-155
lines changed

_api-reference/grpc-apis/bulk.md

Lines changed: 36 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,10 @@ nav_order: 20
99
**Introduced 3.0**
1010
{: .label .label-purple }
1111

12-
This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/16787).
13-
{: .warning}
12+
**Generally available 3.2**
13+
{: .label .label-green }
14+
15+
The gRPC Bulk API is generally available starting with OpenSearch 3.2. For OpenSearch 3.0 and 3.1, this was an experimental feature.
1416

1517
The gRPC Bulk API provides an efficient, binary-encoded alternative to the [HTTP Bulk API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/bulk/) for performing multiple document operations—such as indexing, updating, and deleting—in a single call. This service uses protocol buffers and mirrors the REST API in terms of parameters and structure.
1618

@@ -20,70 +22,70 @@ To submit gRPC requests, you must have a set of protobufs on the client side. Fo
2022

2123
## gRPC service and method
2224

23-
gRPC Document APIs reside in the [DocumentService](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/services/document_service.proto#L23C12-L23C23).
25+
gRPC Document APIs reside in the [DocumentService](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/services/document_service.proto#L23C12-L23C23).
2426

25-
You can submit bulk requests by invoking the [`Bulk`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/services/document_service.proto#L23) gRPC method within the `DocumentService`. The method takes in a [`BulkRequest`](#bulkrequest-fields) and returns a [`BulkResponse`](#bulkresponsebody-fields).
27+
You can submit bulk requests by invoking the [`Bulk`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/services/document_service.proto#L23) gRPC method within the `DocumentService`. The method takes in a [`BulkRequest`](#bulkrequest-fields) and returns a [`BulkResponse`](#bulkresponsebody-fields).
2628

2729
## Document format
2830

2931
In gRPC, documents must be provided and returned as bytes. Use Base64 encoding to provide documents in a gRPC request.
3032
{: .note }
3133

32-
For example, consider the following document in a regular Bulk API request:
34+
For example, consider the following document in a regular Bulk API request:
3335

3436
```json
3537
"doc": "{\"title\": \"Inception\", \"year\": 2010}"
3638
```
3739

38-
For a gRPC Bulk API request, provide the same document in Base64 encoding:
40+
For a gRPC Bulk API request, provide the same document in Base64 encoding:
3941

4042
```json
4143
"doc": "eyJ0aXRsZSI6ICJJbmNlcHRpb24iLCAieWVhciI6IDIwMTB9"
4244
```
4345

4446
## BulkRequest fields
4547

46-
The [`BulkRequest`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L16) message is the top-level container for a gRPC bulk operation. It accepts the following fields.
48+
The [`BulkRequest`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L16) message is the top-level container for a gRPC bulk operation. It accepts the following fields.
4749

4850
| Field | Protobuf type | Description |
4951
| :---- | :---- | :---- |
50-
| `request_body` | `repeated `[`BulkRequestBody`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L53) | The list of bulk operations (`index`/`create`/`update`/`delete`). Required. |
52+
| `request_body` | `repeated `[`BulkRequestBody`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L53) | The list of bulk operations (`index`/`create`/`update`/`delete`). Required. |
5153
| `index` | `string` | The default index for all operations unless overridden in `request_body`. Specifying the `index` in the `BulkRequest` means that you don't need to include it in the [BulkRequestBody](#bulkrequestbody-fields). Optional. |
52-
| `source` | [`SourceConfigParam`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L154) | Controls whether to return the full `_source`, no `_source`, or only specific fields from `_source` in the response. Optional. |
54+
| `source` | [`SourceConfigParam`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L154) | Controls whether to return the full `_source`, no `_source`, or only specific fields from `_source` in the response. Optional. |
5355
| `source_excludes` | `repeated string` | Fields to exclude from `source`. Optional. |
5456
| `source_includes` | `repeated string` | Fields to include from `source`. Optional. |
5557
| `pipeline` | `string` | The preprocessing ingest pipeline ID. Optional. |
56-
| `refresh` | [`Refresh`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L307) | Whether to refresh shards after indexing. Optional. |
58+
| `refresh` | [`Refresh`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L307) | Whether to refresh shards after indexing. Optional. |
5759
| `require_alias` | `bool` | If `true`, actions must target an alias. Optional. |
5860
| `routing` | `string` | The routing value for shard assignment. Optional. |
5961
| `timeout` | `string` | The timeout duration (for example, `1m`). Optional. |
6062
| `type` (Deprecated) | `string` | The document type (always `_doc`). Optional. |
61-
| `wait_for_active_shards` | [`WaitForActiveShards`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L13) | The minimum number of active shards to wait for. Optional. |
63+
| `wait_for_active_shards` | [`WaitForActiveShards`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L13) | The minimum number of active shards to wait for. Optional. |
6264

6365

6466
## BulkRequestBody fields
6567

66-
The [`BulkRequestBody`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L53) message represents a single document-level operation within a `BulkRequest`. It accepts the following fields. All fields are optional, but exactly one of `index`, `create`, `update`, or `delete` must be set in the `BulkRequestBody`.
68+
The [`BulkRequestBody`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L53) message represents a single document-level operation within a `BulkRequest`. It accepts the following fields. All fields are optional, but exactly one of `index`, `create`, `update`, or `delete` must be set in the `BulkRequestBody`.
6769

6870
| Field | Protobuf type | Description |
6971
| :---- | :---- | :---- |
70-
| `index` | [`IndexOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L107) | Index a document. Replaces the document if it already exists. Optional. |
71-
| `create` | [`CreateOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L141) | Create a new document. Fails if the document already exists. Optional. |
72-
| `update` | [`UpdateOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L171) | Partially update a document or use upsert/script options. Optional. |
73-
| `delete` | [`DeleteOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L190) | Delete a document by ID. Optional. |
72+
| `index` | [`IndexOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L107) | Index a document. Replaces the document if it already exists. Optional. |
73+
| `create` | [`CreateOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L141) | Create a new document. Fails if the document already exists. Optional. |
74+
| `update` | [`UpdateOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L171) | Partially update a document or use upsert/script options. Optional. |
75+
| `delete` | [`DeleteOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L190) | Delete a document by ID. Optional. |
7476
| `detect_noop` | `bool` | If `true`, skips the update if the document content hasn't changed. Optional. Default is `true`. |
7577
| `doc` | `bytes` | Partial or full document data for `update` or `index` operations. Optional. |
7678
| `doc_as_upsert` | `bool` | If `true`, treats the document as the full upsert document if the target document doesn't exist. Only valid for the `update` operation. Optional. |
77-
| `script` | [`Script`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L27) | A script to apply to the document (used with `update`). Optional. |
79+
| `script` | [`Script`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L27) | A script to apply to the document (used with `update`). Optional. |
7880
| `scripted_upsert` | `bool` | If `true`, executes the script whether or not the document exists. Optional. |
79-
| `source` | [`SourceConfig`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L176) | Controls how the document source is fetched or filtered. Optional. |
81+
| `source` | [`SourceConfig`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L176) | Controls how the document source is fetched or filtered. Optional. |
8082
| `upsert` | `bytes` | The full document to use if the target does not exist. Used with `script`. Optional. |
8183
| `object` | `bytes` | The full document content used with `create`. Optional. |
8284

8385

8486
### Create
8587

86-
`CreateOperation` adds a new document only if it doesn't already exist.
88+
`CreateOperation` adds a new document only if it doesn't already exist.
8789

8890
The document itself must be provided in the `object` field, outside of the `CreateOperation` message.
8991

@@ -97,7 +99,7 @@ The following optional fields can also be provided.
9799
| `if_primary_term` | `int64` | Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
98100
| `if_seq_no` | `int64` | Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
99101
| `version` | `int64` | The explicit document version for concurrency control. Optional. |
100-
| `version_type` | [`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L99) | Controls version matching behavior. Optional. |
102+
| `version_type` | [`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L99) | Controls version matching behavior. Optional. |
101103
| `pipeline` | `string` | The preprocessing ingest pipeline ID. Optional. |
102104
| `require_alias` | `bool` | Enforces the use of index aliases only. Optional. |
103105

@@ -132,7 +134,7 @@ The `DeleteOperation` removes a document by ID. It accepts the following fields.
132134
| `if_primary_term` | `int64` | Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
133135
| `if_seq_no` | `int64` | Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
134136
| `version` | `int64` | The explicit document version for concurrency control. Optional. |
135-
| `version_type` | [`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L99) | Controls version matching behavior. Optional. |
137+
| `version_type` | [`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L99) | Controls version matching behavior. Optional. |
136138

137139

138140
#### Example request
@@ -170,9 +172,9 @@ The following optional fields can also be provided.
170172
| `routing` | `string` | A custom routing value used to control shard placement. Optional. |
171173
| `if_primary_term` | `int64` | Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
172174
| `if_seq_no` | `int64` | Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
173-
| `op_type` | [`OpType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L91) | The operation type. Controls the overwriting behavior. Valid values are `index` (default) and `create`. Optional. |
175+
| `op_type` | [`OpType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L91) | The operation type. Controls the overwriting behavior. Valid values are `index` (default) and `create`. Optional. |
174176
| `version` | `int64` | The explicit document version for concurrency control. Optional. |
175-
| `version_type` | [`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L99) | Controls version matching behavior. Optional. |
177+
| `version_type` | [`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L99) | Controls version matching behavior. Optional. |
176178
| `pipeline` | `string` | The preprocessing ingest pipeline ID. Optional. |
177179
| `require_alias` | `bool` | Enforces the use of index aliases only. Optional. |
178180

@@ -199,11 +201,11 @@ The following example shows a bulk request with an `index` operation. It indexes
199201

200202
### Update
201203

202-
The `UpdateOperation` performs partial document updates.
204+
The `UpdateOperation` performs partial document updates.
203205

204206
The document itself is provided in the `doc` field, outside of the `UpdateOperation` message.
205207

206-
All `UpdateOperation` fields, listed in the following table, are optional except for `id`.
208+
All `UpdateOperation` fields, listed in the following table, are optional except for `id`.
207209

208210
| Field | Protobuf type | Description |
209211
| ----- | ----- | ----- |
@@ -268,9 +270,9 @@ The following example shows a bulk request with an `upsert` operation. It update
268270
### Script
269271

270272
Run a stored or inline script to modify a document.
271-
272-
273-
To specify a script, provide an `UpdateOperation` and a `script` field outside of the `UpdateOperation`.
273+
274+
275+
To specify a script, provide an `UpdateOperation` and a `script` field outside of the `UpdateOperation`.
274276

275277
#### Example request
276278

@@ -302,7 +304,7 @@ The gRPC Bulk API provides the following response fields.
302304

303305
### BulkResponseBody fields
304306

305-
The [`BulkResponse`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L211) message wraps either a `BulkResponseBody` for successful requests, or a `BulkErrorResponse` for failed requests. The `BulkResponseBody` provides a summary and per-item result of a bulk operation and contains the following fields.
307+
The [`BulkResponse`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L211) message wraps either a `BulkResponseBody` for successful requests, or a `BulkErrorResponse` for failed requests. The `BulkResponseBody` provides a summary and per-item result of a bulk operation and contains the following fields.
306308

307309
| Field | Protobuf type | Description |
308310
| ----- | ----- | ----- |
@@ -331,18 +333,18 @@ Each `ResponseItem` corresponds to a single operation in the request. It contain
331333
| Field | Protobuf type | Description |
332334
|------------------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
333335
| `type` | `string` | The document type. |
334-
| `id` | [`ResponseItem.Id`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L254) | The document ID associated with the operation. Can be `null`. |
336+
| `id` | [`ResponseItem.Id`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L254) | The document ID associated with the operation. Can be `null`. |
335337
| `index` | `string` | The name of the index associated with the operation. If a data stream was targeted, this is the backing index. |
336338
| `status` | `int32` | The HTTP status code returned for the operation. *(Note: This field may be replaced with a gRPC code in the future.)* |
337-
| `error` | [`ErrorCause`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L239) | Contains additional information about a failed operation. |
339+
| `error` | [`ErrorCause`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L239) | Contains additional information about a failed operation. |
338340
| `primary_term` | `int64` | The primary term assigned to the document. |
339341
| `result` | `string` | The operation result. Valid values are `created`, `deleted`, and `updated`. |
340342
| `seq_no` | `int64` | A sequence number assigned to the document to maintain version order. |
341-
| `shards` | [`ShardInfo`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L286) | Shard information for the operation (only returned for successful actions). |
343+
| `shards` | [`ShardInfo`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L286) | Shard information for the operation (only returned for successful actions). |
342344
| `version` | `int64` | The document version (only returned for successful actions). |
343345
| `forced_refresh` | `bool` | If `true`, forces the document to become visible immediately after the operation. |
344-
| `get` | [`InlineGetDictUserDefined`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L290) | Contains the document `source` returned from an inline get, if requested. |
345-
346+
| `get` | [`InlineGetDictUserDefined`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L290) | Contains the document `source` returned from an inline get, if requested. |
347+
346348
## Example response
347349

348350
```json

0 commit comments

Comments
 (0)