You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _api-reference/grpc-apis/bulk.md
+36-34Lines changed: 36 additions & 34 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,8 +9,10 @@ nav_order: 20
9
9
**Introduced 3.0**
10
10
{: .label .label-purple }
11
11
12
-
This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/16787).
13
-
{: .warning}
12
+
**Generally available 3.2**
13
+
{: .label .label-green }
14
+
15
+
The gRPC Bulk API is generally available starting with OpenSearch 3.2. For OpenSearch 3.0 and 3.1, this was an experimental feature.
14
16
15
17
The gRPC Bulk API provides an efficient, binary-encoded alternative to the [HTTP Bulk API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/bulk/) for performing multiple document operations—such as indexing, updating, and deleting—in a single call. This service uses protocol buffers and mirrors the REST API in terms of parameters and structure.
16
18
@@ -20,70 +22,70 @@ To submit gRPC requests, you must have a set of protobufs on the client side. Fo
20
22
21
23
## gRPC service and method
22
24
23
-
gRPC Document APIs reside in the [DocumentService](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/services/document_service.proto#L23C12-L23C23).
25
+
gRPC Document APIs reside in the [DocumentService](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/services/document_service.proto#L23C12-L23C23).
24
26
25
-
You can submit bulk requests by invoking the [`Bulk`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/services/document_service.proto#L23) gRPC method within the `DocumentService`. The method takes in a [`BulkRequest`](#bulkrequest-fields) and returns a [`BulkResponse`](#bulkresponsebody-fields).
27
+
You can submit bulk requests by invoking the [`Bulk`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/services/document_service.proto#L23) gRPC method within the `DocumentService`. The method takes in a [`BulkRequest`](#bulkrequest-fields) and returns a [`BulkResponse`](#bulkresponsebody-fields).
26
28
27
29
## Document format
28
30
29
31
In gRPC, documents must be provided and returned as bytes. Use Base64 encoding to provide documents in a gRPC request.
30
32
{: .note }
31
33
32
-
For example, consider the following document in a regular Bulk API request:
34
+
For example, consider the following document in a regular Bulk API request:
The [`BulkRequest`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L16) message is the top-level container for a gRPC bulk operation. It accepts the following fields.
48
+
The [`BulkRequest`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L16) message is the top-level container for a gRPC bulk operation. It accepts the following fields.
47
49
48
50
| Field | Protobuf type | Description |
49
51
| :---- | :---- | :---- |
50
-
|`request_body`|`repeated `[`BulkRequestBody`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L53)| The list of bulk operations (`index`/`create`/`update`/`delete`). Required. |
52
+
|`request_body`|`repeated `[`BulkRequestBody`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L53)| The list of bulk operations (`index`/`create`/`update`/`delete`). Required. |
51
53
|`index`|`string`| The default index for all operations unless overridden in `request_body`. Specifying the `index` in the `BulkRequest` means that you don't need to include it in the [BulkRequestBody](#bulkrequestbody-fields). Optional. |
52
-
|`source`|[`SourceConfigParam`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L154)| Controls whether to return the full `_source`, no `_source`, or only specific fields from `_source` in the response. Optional. |
54
+
|`source`|[`SourceConfigParam`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L154)| Controls whether to return the full `_source`, no `_source`, or only specific fields from `_source` in the response. Optional. |
53
55
|`source_excludes`|`repeated string`| Fields to exclude from `source`. Optional. |
54
56
|`source_includes`|`repeated string`| Fields to include from `source`. Optional. |
55
57
|`pipeline`|`string`| The preprocessing ingest pipeline ID. Optional. |
56
-
|`refresh`|[`Refresh`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L307)| Whether to refresh shards after indexing. Optional. |
58
+
|`refresh`|[`Refresh`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L307)| Whether to refresh shards after indexing. Optional. |
57
59
|`require_alias`|`bool`| If `true`, actions must target an alias. Optional. |
58
60
|`routing`|`string`| The routing value for shard assignment. Optional. |
59
61
|`timeout`|`string`| The timeout duration (for example, `1m`). Optional. |
60
62
|`type` (Deprecated) |`string`| The document type (always `_doc`). Optional. |
61
-
|`wait_for_active_shards`|[`WaitForActiveShards`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L13)| The minimum number of active shards to wait for. Optional. |
63
+
|`wait_for_active_shards`|[`WaitForActiveShards`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L13)| The minimum number of active shards to wait for. Optional. |
62
64
63
65
64
66
## BulkRequestBody fields
65
67
66
-
The [`BulkRequestBody`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L53) message represents a single document-level operation within a `BulkRequest`. It accepts the following fields. All fields are optional, but exactly one of `index`, `create`, `update`, or `delete` must be set in the `BulkRequestBody`.
68
+
The [`BulkRequestBody`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L53) message represents a single document-level operation within a `BulkRequest`. It accepts the following fields. All fields are optional, but exactly one of `index`, `create`, `update`, or `delete` must be set in the `BulkRequestBody`.
67
69
68
70
| Field | Protobuf type | Description |
69
71
| :---- | :---- | :---- |
70
-
|`index`|[`IndexOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L107)| Index a document. Replaces the document if it already exists. Optional. |
71
-
|`create`|[`CreateOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L141)| Create a new document. Fails if the document already exists. Optional. |
72
-
|`update`|[`UpdateOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L171)| Partially update a document or use upsert/script options. Optional. |
73
-
|`delete`|[`DeleteOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L190)| Delete a document by ID. Optional. |
72
+
|`index`|[`IndexOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L107)| Index a document. Replaces the document if it already exists. Optional. |
73
+
|`create`|[`CreateOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L141)| Create a new document. Fails if the document already exists. Optional. |
74
+
|`update`|[`UpdateOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L171)| Partially update a document or use upsert/script options. Optional. |
75
+
|`delete`|[`DeleteOperation`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L190)| Delete a document by ID. Optional. |
74
76
|`detect_noop`|`bool`| If `true`, skips the update if the document content hasn't changed. Optional. Default is `true`. |
75
77
|`doc`|`bytes`| Partial or full document data for `update` or `index` operations. Optional. |
76
78
|`doc_as_upsert`|`bool`| If `true`, treats the document as the full upsert document if the target document doesn't exist. Only valid for the `update` operation. Optional. |
77
-
|`script`|[`Script`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L27)| A script to apply to the document (used with `update`). Optional. |
79
+
|`script`|[`Script`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L27)| A script to apply to the document (used with `update`). Optional. |
78
80
|`scripted_upsert`|`bool`| If `true`, executes the script whether or not the document exists. Optional. |
79
-
|`source`|[`SourceConfig`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L176)| Controls how the document source is fetched or filtered. Optional. |
81
+
|`source`|[`SourceConfig`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L176)| Controls how the document source is fetched or filtered. Optional. |
80
82
|`upsert`|`bytes`| The full document to use if the target does not exist. Used with `script`. Optional. |
81
83
|`object`|`bytes`| The full document content used with `create`. Optional. |
82
84
83
85
84
86
### Create
85
87
86
-
`CreateOperation` adds a new document only if it doesn't already exist.
88
+
`CreateOperation` adds a new document only if it doesn't already exist.
87
89
88
90
The document itself must be provided in the `object` field, outside of the `CreateOperation` message.
89
91
@@ -97,7 +99,7 @@ The following optional fields can also be provided.
97
99
|`if_primary_term`|`int64`| Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
98
100
|`if_seq_no`|`int64`| Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
99
101
|`version`|`int64`| The explicit document version for concurrency control. Optional. |
100
-
|`version_type`|[`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L99)| Controls version matching behavior. Optional. |
102
+
|`version_type`|[`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L99)| Controls version matching behavior. Optional. |
101
103
|`pipeline`|`string`| The preprocessing ingest pipeline ID. Optional. |
102
104
|`require_alias`|`bool`| Enforces the use of index aliases only. Optional. |
103
105
@@ -132,7 +134,7 @@ The `DeleteOperation` removes a document by ID. It accepts the following fields.
132
134
|`if_primary_term`|`int64`| Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
133
135
|`if_seq_no`|`int64`| Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
134
136
|`version`|`int64`| The explicit document version for concurrency control. Optional. |
135
-
|`version_type`|[`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L99)| Controls version matching behavior. Optional. |
137
+
|`version_type`|[`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L99)| Controls version matching behavior. Optional. |
136
138
137
139
138
140
#### Example request
@@ -170,9 +172,9 @@ The following optional fields can also be provided.
170
172
|`routing`|`string`| A custom routing value used to control shard placement. Optional. |
171
173
|`if_primary_term`|`int64`| Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
172
174
|`if_seq_no`|`int64`| Used for concurrency control. The operation only runs if the document's primary term matches this value. Optional. |
173
-
|`op_type`|[`OpType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L91)| The operation type. Controls the overwriting behavior. Valid values are `index` (default) and `create`. Optional. |
175
+
|`op_type`|[`OpType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L91)| The operation type. Controls the overwriting behavior. Valid values are `index` (default) and `create`. Optional. |
174
176
|`version`|`int64`| The explicit document version for concurrency control. Optional. |
175
-
|`version_type`|[`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L99)| Controls version matching behavior. Optional. |
177
+
|`version_type`|[`VersionType`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L99)| Controls version matching behavior. Optional. |
176
178
|`pipeline`|`string`| The preprocessing ingest pipeline ID. Optional. |
177
179
|`require_alias`|`bool`| Enforces the use of index aliases only. Optional. |
178
180
@@ -199,11 +201,11 @@ The following example shows a bulk request with an `index` operation. It indexes
199
201
200
202
### Update
201
203
202
-
The `UpdateOperation` performs partial document updates.
204
+
The `UpdateOperation` performs partial document updates.
203
205
204
206
The document itself is provided in the `doc` field, outside of the `UpdateOperation` message.
205
207
206
-
All `UpdateOperation` fields, listed in the following table, are optional except for `id`.
208
+
All `UpdateOperation` fields, listed in the following table, are optional except for `id`.
207
209
208
210
| Field | Protobuf type | Description |
209
211
| ----- | ----- | ----- |
@@ -268,9 +270,9 @@ The following example shows a bulk request with an `upsert` operation. It update
268
270
### Script
269
271
270
272
Run a stored or inline script to modify a document.
271
-
272
-
273
-
To specify a script, provide an `UpdateOperation` and a `script` field outside of the `UpdateOperation`.
273
+
274
+
275
+
To specify a script, provide an `UpdateOperation` and a `script` field outside of the `UpdateOperation`.
274
276
275
277
#### Example request
276
278
@@ -302,7 +304,7 @@ The gRPC Bulk API provides the following response fields.
302
304
303
305
### BulkResponseBody fields
304
306
305
-
The [`BulkResponse`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L211) message wraps either a `BulkResponseBody` for successful requests, or a `BulkErrorResponse` for failed requests. The `BulkResponseBody` provides a summary and per-item result of a bulk operation and contains the following fields.
307
+
The [`BulkResponse`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L211) message wraps either a `BulkResponseBody` for successful requests, or a `BulkErrorResponse` for failed requests. The `BulkResponseBody` provides a summary and per-item result of a bulk operation and contains the following fields.
306
308
307
309
| Field | Protobuf type | Description |
308
310
| ----- | ----- | ----- |
@@ -331,18 +333,18 @@ Each `ResponseItem` corresponds to a single operation in the request. It contain
|`id`|[`ResponseItem.Id`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L254)| The document ID associated with the operation. Can be `null`. |
336
+
|`id`|[`ResponseItem.Id`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L254)| The document ID associated with the operation. Can be `null`. |
335
337
|`index`|`string`| The name of the index associated with the operation. If a data stream was targeted, this is the backing index. |
336
338
|`status`|`int32`| The HTTP status code returned for the operation. *(Note: This field may be replaced with a gRPC code in the future.)*|
337
-
|`error`|[`ErrorCause`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L239)| Contains additional information about a failed operation. |
339
+
|`error`|[`ErrorCause`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L239)| Contains additional information about a failed operation. |
338
340
|`primary_term`|`int64`| The primary term assigned to the document. |
339
341
|`result`|`string`| The operation result. Valid values are `created`, `deleted`, and `updated`. |
340
342
|`seq_no`|`int64`| A sequence number assigned to the document to maintain version order. |
341
-
|`shards`|[`ShardInfo`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/common.proto#L286)| Shard information for the operation (only returned for successful actions). |
343
+
|`shards`|[`ShardInfo`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L286)| Shard information for the operation (only returned for successful actions). |
342
344
|`version`|`int64`| The document version (only returned for successful actions). |
343
345
|`forced_refresh`|`bool`| If `true`, forces the document to become visible immediately after the operation. |
344
-
|`get`|[`InlineGetDictUserDefined`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.3.0/protos/schemas/document.proto#L290)| Contains the document `source` returned from an inline get, if requested. |
345
-
346
+
|`get`|[`InlineGetDictUserDefined`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/document.proto#L290)| Contains the document `source` returned from an inline get, if requested. |
0 commit comments