Skip to content

adding the new config for aws lambda and defaults #11509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+24 −23
Open

adding the new config for aws lambda and defaults #11509

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5cfa68e
adding the new config for aws lambda and defaults
AntonEliatra Nov 10, 2025
db8e3ac
removing invocation_type #9872
AntonEliatra Nov 10, 2025
7fd3853
addressing PR comments
AntonEliatra Nov 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 24 additions & 23 deletions _data-prepper/pipelines/configuration/processors/aws-lambda.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,49 +20,50 @@ You can configure the processor using the following configuration options.

Field | Type | Required | Description
-------------------- | ------- | -------- | ----------------------------------------------------------------------------
`function_name` | String | Required | The name of the AWS Lambda function to invoke.
`invocation_type` | String | Required | Specifies the invocation type, either `request-response` or `event`. Default is `request-response`.
`aws.region` | String | Required | The AWS Region in which the Lambda function is located.
`aws.sts_role_arn` | String | Optional | The Amazon Resource Name (ARN) of the role to assume before invoking the Lambda function.
`function_name` | String | Required | The name of the AWS Lambda function to invoke.
`aws.region` | String | Required | The AWS Region in which the Lambda function is located. If no configuration is provided, value in `data-prepper-config.yaml` file is used. If still not found, default AWS credentials on the host are used.
`aws.sts_role_arn` | String | Optional | The Amazon Resource Name (ARN) of the role to assume before invoking the Lambda function. If no configuration is provided, value in `data-prepper-config.yaml` file is used. If still not found, default AWS credentials on the host are used.
`max_retries` | Integer | Optional | The maximum number of retries for failed invocations. Default is `3`.
`batch` | Object | Optional | The batch settings for the Lambda invocations. Default is `key_name = "events"`. Default threshold is `event_count=100`, `maximum_size="5mb"`, and `event_collect_timeout = 10s`.
`lambda_when` | String | Optional | A conditional expression that determines when to invoke the Lambda processor.
`batch` | Object | Optional | The batch settings for the Lambda invocations. Default is `key_name = "events"`, threshold `event_count=100`, `maximum_size="5mb"`, and `event_collect_timeout = 10s`.
`lambda_when` | String | Optional | A conditional expression that determines when to invoke the Lambda processor. By default all events are processed.
`response_codec` | Object | Optional | A codec configuration for parsing Lambda responses. Default is `json`.
`tags_on_match_failure` | List | Optional | A list of tags to add to events when Lambda matching fails or encounters an unexpected error.
`sdk_timeout` | Duration| Optional | Configures the SDK's client connection timeout period. Default is `60s`.
`tags_on_match_failure` | List | Optional | A list of tags to add to events when Lambda matching fails or encounters an unexpected error. Default is `[]`.
`response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`.
`client` | Object | Optional | The client configuration.
`api_call_timeout` | Duration | Optional | The amount of time that the SDK maintains the API call before timing out, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations).
`base_delay` | Duration | Optional | The base delay for exponential backoff, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations).
`connection_timeout` | Duration | Optional | The amount of time that the SDK maintains the connection to the client before timing out, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations).
`max_backoff` | Duration | Optional | The maximum backoff time for exponential backoff, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations).
`max_concurrency` | Integer | Optional | The maximum concurrency defined on the client side.
`max_retries` | Integer | Optional | The maximum number of retries before failing.
`api_call_timeout` | Duration | Optional | The total time that the SDK will attempt connection, including all retries. A Data Prepper duration can be expressed either as full [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) or as a simplified form in seconds or milliseconds, for example `60s` or `60000ms`. Default is `60s`.
`base_delay` | Duration | Optional | The base delay for exponential backoff, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations). Default is `100ms`.
`connection_timeout` | Duration | Optional | The amount of time that the SDK maintains the connection to the client before timing out, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations). Default is `60s`.
`max_backoff` | Duration | Optional | The maximum backoff time for exponential backoff, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations). Default is `20s`.
`max_concurrency` | Integer | Optional | The maximum concurrency defined on the client side. Default is `200`.
`max_retries` | Integer | Optional | The maximum number of retries before failing. Default is `3`.
`circuit_breaker_retries` | Integer | Optional | The number of retry attempts to perform when the circuit breaker is open before resuming normal processing. Default is `0`.
`circuit_breaker_wait_interval` | Integer | Optional | The wait interval, in milliseconds, between circuit breaker retries. Default is `1000`.

## Example configuration

#### Example configuration
The following is example configuration:

```
processors:
- aws_lambda:
function_name: "my-lambda-function"
invocation_type: "request-response"
function_name: my-lambda-function
response_events_match: false
client:
connection_timeout: PT5M
api_call_timeout: PT5M
aws:
region: "us-east-1"
sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-role"
region: us-east-1
sts_role_arn: arn:aws:iam::123456789012:role/my-lambda-role
max_retries: 3
batch:
key_name: "events"
key_name: events
threshold:
event_count: 100
maximum_size: "5mb"
maximum_size: 5mb
event_collect_timeout: PT10S
circuit_breaker_retries: 30
circuit_breaker_wait_interval: 1000
lambda_when: "event['status'] == 'process'"

```
{% include copy.html %}

Expand Down Expand Up @@ -101,4 +102,4 @@ Integration tests for this plugin are executed separately from the main Data Pre
```
./gradlew :data-prepper-plugins:aws-lambda:integrationTest -Dtests.processor.lambda.region="us-east-1" -Dtests.processor.lambda.functionName="lambda_test_function" -Dtests.processor.lambda.sts_role_arn="arn:aws:iam::123456789012:role/dataprepper-role
```
{% include copy.html %}
{% include copy.html %}