You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The `scripted_metric`metric is a multi-value metric aggregation that returns metrics calculated from a specified script.
12
+
The `scripted_metric`aggregation is a multi-value metric aggregation that returns metrics calculated from a specified script. A script has four phases, `init`, `map`, `combine`, and `reduce`, which are run in order by each aggregation and allow you to combine results from your documents.
13
13
14
-
A script has four stages: the initial stage, the map stage, the combine stage, and the reduce stage.
14
+
All four scripts share a mutable object called `state` that is defined by you. The `state` is local to each shard during the `init`, `map`, and `combine` phases. The result is passed into the `states` array for the `reduce` phase. Therefore, each shard's `state` is independent until the shards are combined in the `reduce` step.
15
15
16
-
*`init_script`: (OPTIONAL) Sets the initial state and executes before any collection of documents.
17
-
*`map_script`: Checks the value of the `type` field and executes the aggregation on the collected documents.
18
-
*`combine_script`: Aggregates the state returned from every shard. The aggregated value is returned to the coordinating node.
19
-
*`reduce_script`: Provides access to the variable states; this variable combines the results from the `combine_script` on each shard into an array.
16
+
## Parameters
20
17
21
-
The following example aggregates the different HTTP response types in web log data:
18
+
The `scripted_metric` aggregation takes the following parameters.
19
+
20
+
| Parameter | Data type | Required/Optional | Description |
|`init_script`| String | Optional | A script that executes once per shard before any documents are processed. Used to set up an initial `state` (for example, initialize counters or lists in a `state` object). If not provided, the `state` starts as an empty object on each shard. |
23
+
|`map_script`| String | Required | A script that executes for each document collected by the aggregation. This script updates the `state` based on the document's data. For example, you might check the field's value and then increment a counter or calculate a running sum in the `state`. |
24
+
|`combine_script`| String | Required | A script that executes once per shard after all documents on that shard have been processed by the `map_script`. This script aggregates the shard's `state` into a single result to be sent back to the coordinating node. This script is used to finalize the computation for one shard (for example, summing up counters or totals stored in the `state`). The script should return the consolidated value or structure for its shard. |
25
+
|`reduce_script`| String | Required | A script that executes once on the coordinating node after receiving combined results from all shards. This script receives a special variable `states`, which is an array containing each shard's output from the `combine_script`. The `reduce_script` iterates over states and produces the final aggregation output (for example, adding shard sums or merging maps of counts). The value returned by the `reduce_script` is the value reported in the aggregation results. |
26
+
|`params`| Object | Optional | User-defined parameters accessible from all scripts except `reduce_script`. |
27
+
28
+
## Allowed return types
29
+
30
+
Scripts can use any valid operation and object internally. However, the data you store in `state` or return from any script must be of one of the allowed types. This restriction exists because the intermediate `state` needs to be sent between nodes. The following types are allowed:
- Map (with keys and values of only allowed types: primitive types, string, map, or array)
35
+
- Array (containing only allowed types: primitive types, string, map, or array)
36
+
37
+
The `state` can be a number, a string, a `map` (object) or an array (list), or a combination of these. For example, you can use a `map` to accumulate multiple counters, an array to collect values, or a single number to keep a running sum. If you need to return multiple metrics, you can store them in a `map` or array. If you return a `map` as the final value from the `reduce_script`, the aggregation result contains an object. If you return a single number or string, the result is a single value.
38
+
39
+
## Using parameters in scripts
40
+
41
+
You can optionally pass custom parameters to your scripts using the `params` field. This is a user-defined object whose contents become variables available in your `init_script`, `map_script`, and `combine_script`. The `reduce_script` does not directly receive `params` because by the `reduce` phase, all needed data must be in the `states` array. If you need a constant in the `reduce` phase, you can include it as part of each shard's `state` or use a stored script. All parameters must be defined inside the global `params` object. This ensures that they are shared across the different script phases. If you do not specify any `params`, the `params` object is empty.
42
+
43
+
For example, you can supply a `threshold` or `field` name in `params` and then reference `params.threshold` or `params.field` in your scripts:
22
44
23
45
```json
24
-
GET opensearch_dashboards_sample_data_logs/_search
46
+
"scripted_metric": {
47
+
"params": {
48
+
"threshold": 100,
49
+
"field": "amount"
50
+
},
51
+
"init_script": "...",
52
+
"map_script": "...",
53
+
"combine_script": "...",
54
+
"reduce_script": "..."
55
+
}
56
+
```
57
+
58
+
## Examples
59
+
60
+
The following examples demonstrate different ways to use `scripted_metric`.
61
+
62
+
### Calculating net profit from transactions
63
+
64
+
The following example demonstrates the use of the `scripted_metric` aggregation to compute a custom metric that is not directly supported by built-in aggregations. The dataset represents financial transactions, in which each document is classified as either a `sale` (income) or a `cost` (expense) and includes an `amount` field. The objective is to calculate the total net profit by subtracting the total cost from the total sales across all documents.
65
+
66
+
Create an index:
67
+
68
+
```json
69
+
PUT transactions
25
70
{
26
-
"size": 0,
71
+
"mappings": {
72
+
"properties": {
73
+
"type": { "type": "keyword" },
74
+
"amount": { "type": "double" }
75
+
}
76
+
}
77
+
}
78
+
```
79
+
{% include copy-curl.html %}
80
+
81
+
Index four transactions, two sales (amounts `80` and `130`), and two costs (`10` and `30`):
82
+
83
+
```json
84
+
PUT transactions/_bulk?refresh=true
85
+
{ "index": {} }
86
+
{ "type": "sale", "amount": 80 }
87
+
{ "index": {} }
88
+
{ "type": "cost", "amount": 10 }
89
+
{ "index": {} }
90
+
{ "type": "cost", "amount": 30 }
91
+
{ "index": {} }
92
+
{ "type": "sale", "amount": 130 }
93
+
```
94
+
{% include copy-curl.html %}
95
+
96
+
To run a search with a `scripted_metric` aggregation to calculate the profit, use the following scripts:
97
+
98
+
- The `init_script` creates an empty list used to store transaction values for each shard.
99
+
- The `map_script` adds each document's amount to the `state.transactions` list as a positive number if the type is `sale` or as a negative number if the type is `cost`. By the end of the `map` phase, each shard has a `state.transactions` list representing its income and expenses.
100
+
- The `combine_script` processes the `state.transactions` list and computes a single `shardProfit` value for the shard. The `shardProfit` is then returned as the shard's output.
101
+
- The `reduce_script` runs on the coordinating node, receiving the `states` array, which holds the `shardProfit` value from each shard. It checks for null entries, adds these values to compute the overall profit, and returns the final result.
102
+
103
+
The following request contains all described scripts:
"combine_script": "double shardProfit = 0; for (t in state.transactions) { shardProfit += t; } return shardProfit;",
115
+
"reduce_script": "double totalProfit = 0; for (p in states) { if (p != null) { totalProfit += p; }} return totalProfit;"
116
+
}
117
+
}
118
+
}
119
+
}
120
+
```
121
+
{% include copy-curl.html %}
122
+
123
+
124
+
The response returns the `total_profit`:
125
+
126
+
```json
127
+
{
128
+
...
129
+
"hits": {
130
+
"total": {
131
+
"value": 4,
132
+
"relation": "eq"
133
+
},
134
+
"max_score": null,
135
+
"hits": []
136
+
},
27
137
"aggregations": {
28
-
"responses.counts": {
138
+
"total_profit": {
139
+
"value": 170
140
+
}
141
+
}
142
+
}
143
+
```
144
+
145
+
### Categorizing HTTP response codes
146
+
147
+
The following example demonstrates a more advanced use of the `scripted_metric` aggregation for returning multiple values within a single aggregation. The dataset consists of web server log entries, each containing an HTTP response code. The goal is to classify the responses into three categories: successful responses (2xx status codes), client or server errors (4xx or 5xx status codes), and other responses (1xx or 3xx status codes). This classification is implemented by maintaining counters within a map-based aggregation `state`.
148
+
149
+
Create a sample index:
150
+
151
+
```json
152
+
PUT logs
153
+
{
154
+
"mappings": {
155
+
"properties": {
156
+
"response": { "type": "keyword" }
157
+
}
158
+
}
159
+
}
160
+
```
161
+
{% include copy-curl.html %}
162
+
163
+
Add sample documents with a variety of response codes:
164
+
165
+
```json
166
+
PUT logs/_bulk?refresh=true
167
+
{ "index": {} }
168
+
{ "response": "200" }
169
+
{ "index": {} }
170
+
{ "response": "201" }
171
+
{ "index": {} }
172
+
{ "response": "404" }
173
+
{ "index": {} }
174
+
{ "response": "500" }
175
+
{ "index": {} }
176
+
{ "response": "304" }
177
+
```
178
+
{% include copy-curl.html %}
179
+
180
+
The `state` (on each shard) is a `map` with three counters: `error`, `success`, and `other`.
181
+
182
+
To run a scripted metric aggregation that counts the categories, use the following scripts:
183
+
184
+
- The `init_script` initializes counters for `error`, `success`, and `other` to `0`.
185
+
- The `map_script` examines each document's response code and increments the relevant counter based on the response code.
186
+
- The `combine_script` returns the `state.responses map` for that shard.
187
+
- The `reduce_script` merges the array of maps (`states`) from all shards. Thus, it creates a new combined `map` and adds the `error`, `success`, and `other` counts from each shard's `map`. This combined `map` is returned as the final result.
188
+
189
+
The following request contains all described scripts:
@@ -55,18 +231,38 @@ GET opensearch_dashboards_sample_data_logs/_search
55
231
```
56
232
{% include copy-curl.html %}
57
233
58
-
#### Example response
234
+
235
+
The response returns three values in the `value` object, demonstrating how a scripted metric can return multiple metrics at once by using a `map` in the `state`:
59
236
60
237
```json
61
-
...
62
-
"aggregations" : {
63
-
"responses.counts" : {
64
-
"value" : {
65
-
"other" : 0,
66
-
"success" : 12832,
67
-
"error" : 1242
238
+
{
239
+
...
240
+
"hits": {
241
+
"total": {
242
+
"value": 5,
243
+
"relation": "eq"
244
+
},
245
+
"max_score": null,
246
+
"hits": []
247
+
},
248
+
"aggregations": {
249
+
"responses_by_type": {
250
+
"value": {
251
+
"other": 1,
252
+
"success": 2,
253
+
"error": 2
254
+
}
68
255
}
69
256
}
70
-
}
71
257
}
72
258
```
259
+
260
+
## Managing empty buckets (no documents)
261
+
262
+
When using a `scripted_metric` aggregation as a subaggregation within a bucket aggregation (such as `terms`), it is important to account for buckets that contain no documents on certain shards. In such cases, those shards return a `null` value for the aggregation `state`. During the `reduce_script` phase, the `states` array may therefore include `null` entries corresponding to these shards. To ensure reliable execution, the `reduce_script` must be designed to handle `null` values gracefully. A common approach is to include a conditional check, such as `if (state != null)`, before accessing or operating on each `state`. Failure to implement such checks can result in runtime errors when processing empty buckets across shards.
263
+
264
+
265
+
## Performance considerations
266
+
267
+
Because scripted metrics run custom code for every document and therefore potentially accumulate a large in-memory `state`, they can be slower than built-in aggregations. The intermediate `state` from each shard must be serialized in order to send it to the coordinating node. Therefore if your `state` is very large, it can consume a lot of memory and network bandwidth. To keep your searches efficient, make your scripts as lightweight as possible and avoid accumulating unnecessary data in the `state`. Use the combine stage to shrink `state` data before sending, as demonstrated in [Calculating net profit from transactions](#calculating-net-profit-from-transactions), and only collect the values that you truly need to produce the final metric.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4&size=40){kind=avatar}
0 commit comments