Skip to content

Commit a674683

Browse files
bryantbiggsantonbabenko
bryantbiggs
and
antonbabenko
authored
feat!: Upgrade AWS provider and min required Terraform version to 6.19 and 1.5.7 respectively (#45)
Co-authored-by: Anton Babenko <[email protected]>
1 parent 127ab1e commit a674683

File tree

33 files changed

+1814
-691
lines changed

33 files changed

+1814
-691
lines changed

‎README.md‎

Lines changed: 41 additions & 32 deletions
Large diffs are not rendered by default.

‎docs/UPGRADE-3.0.md‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
# Upgrade from v2.x to v3.x
2+
3+
If you have any questions regarding this upgrade process, please consult the [`examples`](https://github.com/terraform-aws-modules/terraform-aws-emr/tree/master/examples) directory:
4+
If you find a bug, please open an issue with supporting configuration to reproduce.
5+
6+
## List of backwards incompatible changes
7+
8+
- Terraform `v1.5.7` is now minimum supported version
9+
- AWS provider `v6.19` is now minimum supported version
10+
- Kubernetes provider `v2.38` is now minimum supported version (EMR on EKS virtual cluster sub-module)
11+
- `aws_security_group_rule` resources have been split into `aws_vpc_security_group_ingress_rule` and `aws_vpc_security_group_egress_rule` resources to better match the AWS API and allow for more flexibility in defining security group rules. Prior variable names of `*_security_group_rules` have been split into `*_security_group_ingress_rules` and `*_security_group_egress_rules` to match.
12+
13+
## Additional changes
14+
15+
### Added
16+
17+
- Support for `region` parameter to specify the AWS region for the resources created if different from the provider region.
18+
19+
### Modified
20+
21+
- Variable definitions now contain detailed `object` types in place of the previously used any type.
22+
- Ensure data sources are gated behind `create` flags to prevent unnecessary API calls.
23+
- `release_label_filters.prefix` now defaults to `emr-7`, was previously `emr-6`.
24+
- `unhealthy_node_replacement` now defaults to `true`
25+
- `aws_service_principal` data source is now used to fetch the correct service principals (instead of trying to construct them psuedo-manually with the DNS suffix).
26+
27+
### Variable and output changes
28+
29+
1. Removed variables:
30+
31+
- `serverless` sub-module
32+
- None
33+
34+
- `studio` sub-module
35+
- None
36+
37+
- `virtual_cluster` sub-module
38+
39+
2. Renamed variables:
40+
41+
- `master_security_group_rules` -> `master_security_group_ingress_rules` and `master_security_group_egress_rules`
42+
- `slave_security_group_rules` -> `slave_security_group_ingress_rules` and `slave_security_group_egress_rules`
43+
- `service_security_group_rules` -> `service_security_group_ingress_rules` and `service_security_group_egress_rules`
44+
45+
- `serverless` sub-module
46+
- `security_group_rules` -> `security_group_ingress_rules` and `security_group_egress_rules`
47+
- `release_label_prefix` -> `release_label_filters`
48+
49+
- `studio` sub-module
50+
- `engine_security_group_rules` -> `engine_security_group_ingress_rules` and `engine_security_group_egress_rules`
51+
- `workspace_security_group_rules` -> `workspace_security_group_ingress_rules` and `workspace_security_group_egress_rules`
52+
53+
- `virtual_cluster` sub-module
54+
- `eks_cluster_id` -> `eks_cluster_name` to better match API of EKS module/resources
55+
- `oidc_provider_arn` -> `eks_oidc_provider_arn` for clarity to show its related to EKS authentication
56+
57+
3. Added variables:
58+
59+
- `os_release_label`
60+
61+
- `serverless` sub-module
62+
- `monitoring_configuration`
63+
- `runtime_configuration`
64+
- `scheduler_configuration`
65+
66+
- `studio` sub-module
67+
- None
68+
69+
- `virtual_cluster` sub-module
70+
- `cloudwatch_log_group_class`
71+
72+
4. Removed outputs:
73+
74+
- `serverless` sub-module
75+
- None
76+
77+
- `studio` sub-module
78+
- None
79+
80+
- `virtual_cluster` sub-module
81+
82+
5. Renamed outputs:
83+
84+
- `serverless` sub-module
85+
- None
86+
87+
- `studio` sub-module
88+
- None
89+
90+
- `virtual_cluster` sub-module
91+
- None
92+
93+
6. Added outputs:
94+
95+
- `serverless` sub-module
96+
- None
97+
98+
- `studio` sub-module
99+
- None
100+
101+
- `virtual_cluster` sub-module
102+
- None
103+
104+
## Upgrade Migration
105+
106+
### Before v2.x Example
107+
108+
```hcl
109+
module "emr" {
110+
source = "terraform-aws-modules/emr/aws"
111+
version = "~> 2.0"
112+
113+
# Only the affected attributes are shown
114+
115+
bootstrap_action = {
116+
example = {
117+
name = "Just an example",
118+
path = "file:/bin/echo",
119+
args = ["Hello World!"]
120+
}
121+
}
122+
}
123+
```
124+
125+
### After v3.x Example
126+
127+
```hcl
128+
module "emr" {
129+
source = "terraform-aws-modules/emr/aws"
130+
version = "~> 3.0"
131+
132+
# Only the affected attributes are shown
133+
134+
# Copy and paste from output to maintain backwards compatibility
135+
# This was added by the AWS EMR API and provider in v6.x
136+
os_release_label = "2023.9.20251014.0"
137+
138+
bootstrap_action = [
139+
{
140+
name = "Just an example",
141+
path = "file:/bin/echo",
142+
args = ["Hello World!"]
143+
}
144+
]
145+
}
146+
```
147+
148+
### State Changes
149+
150+
Due to the change from `aws_security_group_rule` to `aws_vpc_security_group_ingress_rule` and `aws_vpc_security_group_egress_rule`, the following reference state changes are required to maintain the current security group rules. (Note: these are different resources so they cannot be moved with `terraform mv ...`)
151+
152+
#### Instance Group
153+
154+
```sh
155+
# Master Security Group
156+
terraform state rm 'module.emr_instance_group.aws_security_group_rule.master["default"]'
157+
terraform state import 'module.emr_instance_group.aws_vpc_security_group_egress_rule.master["all-traffic"]' 'sg-xxx'
158+
159+
# Slave Security Group
160+
terraform state rm 'module.emr_instance_group.aws_security_group_rule.slave["default"]'
161+
terraform state import 'module.emr_instance_group.aws_vpc_security_group_egress_rule.slave["all-traffic"]' 'sg-xxx'
162+
163+
# Service Security Group
164+
terraform state rm 'module.emr_instance_group.aws_security_group_rule.service["master_9443_ingress"]'
165+
terraform state import 'module.emr_instance_group.aws_vpc_security_group_ingress_rule.service["master_9443"]' 'sg-xxx'
166+
167+
terraform state rm 'module.emr_instance_group.aws_security_group_rule.service["master_9443_egress"]'
168+
terraform state import 'module.emr_instance_group.aws_vpc_security_group_egress_rule.service["master_8443"]' 'sg-xxx'
169+
170+
terraform state rm 'module.emr_instance_group.aws_security_group_rule.service["core_task_8443_egress"]'
171+
terraform state import 'module.emr_instance_group.aws_vpc_security_group_egress_rule.service["core_task_8443"]' 'sg-xxx'
172+
```
173+
174+
#### Instance Fleet
175+
176+
```sh
177+
# Master Security Group
178+
terraform state rm 'module.emr_instance_fleet.aws_security_group_rule.master["default"]'
179+
terraform state import 'module.emr_instance_fleet.aws_vpc_security_group_egress_rule.master["all-traffic"]' 'sg-xxx'
180+
181+
# Slave Security Group
182+
terraform state rm 'module.emr_instance_fleet.aws_security_group_rule.slave["default"]'
183+
terraform state import 'module.emr_instance_fleet.aws_vpc_security_group_egress_rule.slave["all-traffic"]' 'sg-xxx'
184+
185+
# Service Security Group
186+
terraform state rm 'module.emr_instance_fleet.aws_security_group_rule.service["master_9443_ingress"]'
187+
terraform state import 'module.emr_instance_fleet.aws_vpc_security_group_ingress_rule.service["master_9443"]' 'sg-xxx'
188+
189+
terraform state rm 'module.emr_instance_fleet.aws_security_group_rule.service["master_9443_egress"]'
190+
terraform state import 'module.emr_instance_fleet.aws_vpc_security_group_egress_rule.service["master_8443"]' 'sg-xxx'
191+
192+
terraform state rm 'module.emr_instance_fleet.aws_security_group_rule.service["core_task_8443_egress"]'
193+
terraform state import 'module.emr_instance_fleet.aws_vpc_security_group_egress_rule.service["core_task_8443"]' 'sg-xxx'
194+
```
195+
196+
#### Serverless sub-module
197+
198+
```sh
199+
terraform state rm 'module.emr_serverless_spark.aws_security_group_rule.this["egress_all"]'
200+
terraform state import 'module.emr_serverless_spark.aws_vpc_security_group_egress_rule.this["all-traffic"]' 'sg-xxx'
201+
```

‎examples/private-cluster/README.md‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,17 @@ Configuration in this directory creates:
88
- S3 bucket for EMR logs
99
- VPC endpoints for EMR, STS, and S3
1010

11-
Note: The private subnets will need to be tagged with `{ "for-use-with-amazon-emr-managed-policies" = true }` ([Reference](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-iam-policies.html#manually-tagged-resources))
11+
> [!NOTE]
12+
> The private subnets will need to be tagged with `{ "for-use-with-amazon-emr-managed-policies" = true }` ([Reference](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-iam-policies.html#manually-tagged-resources))
1213
1314
## Usage
1415

1516
To run this example you need to execute:
1617

1718
```bash
18-
$ terraform init
19-
$ terraform plan
20-
$ terraform apply
19+
terraform init
20+
terraform plan
21+
terraform apply
2122
```
2223

2324
Note that this example may create resources which will incur monetary charges on your AWS bill. Run `terraform destroy` when you no longer need these resources.
@@ -28,13 +29,13 @@ Note that this example may create resources which will incur monetary charges on
2829
| Name | Version |
2930
|------|---------|
3031
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.5.7 |
31-
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.83 |
32+
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 6.19 |
3233

3334
## Providers
3435

3536
| Name | Version |
3637
|------|---------|
37-
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.83 |
38+
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 6.19 |
3839

3940
## Modules
4041

@@ -46,7 +47,6 @@ Note that this example may create resources which will incur monetary charges on
4647
| <a name="module_s3_bucket"></a> [s3\_bucket](#module\_s3\_bucket) | terraform-aws-modules/s3-bucket/aws | ~> 5.0 |
4748
| <a name="module_vpc"></a> [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 6.0 |
4849
| <a name="module_vpc_endpoints"></a> [vpc\_endpoints](#module\_vpc\_endpoints) | terraform-aws-modules/vpc/aws//modules/vpc-endpoints | ~> 6.0 |
49-
| <a name="module_vpc_endpoints_sg"></a> [vpc\_endpoints\_sg](#module\_vpc\_endpoints\_sg) | terraform-aws-modules/security-group/aws | ~> 5.0 |
5050

5151
## Resources
5252

@@ -61,7 +61,6 @@ Note that this example may create resources which will incur monetary charges on
6161
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
6262
| [aws_iam_policy_document.assume](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
6363
| [aws_iam_policy_document.autoscaling](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
64-
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
6564

6665
## Inputs
6766

0 commit comments

Comments
 (0)