Skip to content

feat!: Add support for placement group config and unhealthy node replacement; raise AWS provider MSV to v5.44 #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 7, 2024
Merged

feat!: Add support for placement group config and unhealthy node replacement; raise AWS provider MSV to v5.44 #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
47de6ef
placement group config and unhealthy node replacement
magreenbaum Apr 7, 2024
dfbcd2d
add commented out example for placement group config
magreenbaum Apr 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,13 +330,13 @@ Examples codified under the [`examples`](https://github.com/terraform-aws-module
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.42 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.44 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.42 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.44 |

## Modules

Expand Down Expand Up @@ -421,6 +421,7 @@ No modules.
| <a name="input_master_security_group_description"></a> [master\_security\_group\_description](#input\_master\_security\_group\_description) | Description of the security group created | `string` | `"Managed master security group"` | no |
| <a name="input_master_security_group_rules"></a> [master\_security\_group\_rules](#input\_master\_security\_group\_rules) | Security group rules to add to the security group created | `any` | <pre>{<br> "default": {<br> "cidr_blocks": [<br> "0.0.0.0/0"<br> ],<br> "description": "Allow all egress traffic",<br> "from_port": 0,<br> "ipv6_cidr_blocks": [<br> "::/0"<br> ],<br> "protocol": "-1",<br> "to_port": 0,<br> "type": "egress"<br> }<br>}</pre> | no |
| <a name="input_name"></a> [name](#input\_name) | Name of the job flow | `string` | `""` | no |
| <a name="input_placement_group_config"></a> [placement\_group\_config](#input\_placement\_group\_config) | The specified placement group configuration | `any` | `{}` | no |
| <a name="input_release_label"></a> [release\_label](#input\_release\_label) | Release label for the Amazon EMR release | `string` | `null` | no |
| <a name="input_release_label_filters"></a> [release\_label\_filters](#input\_release\_label\_filters) | Map of release label filters use to lookup a release label | `any` | <pre>{<br> "default": {<br> "prefix": "emr-6"<br> }<br>}</pre> | no |
| <a name="input_scale_down_behavior"></a> [scale\_down\_behavior](#input\_scale\_down\_behavior) | Way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized | `string` | `"TERMINATE_AT_TASK_COMPLETION"` | no |
Expand All @@ -443,6 +444,7 @@ No modules.
| <a name="input_task_instance_fleet"></a> [task\_instance\_fleet](#input\_task\_instance\_fleet) | Configuration block to use an [Instance Fleet](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-fleet.html) for the task node type. Cannot be specified if any `task_instance_group` configuration blocks are set | `any` | `{}` | no |
| <a name="input_task_instance_group"></a> [task\_instance\_group](#input\_task\_instance\_group) | Configuration block to use an [Instance Group](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-group-configuration.html#emr-plan-instance-groups) for the [task node type](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html#emr-plan-master) | `any` | `{}` | no |
| <a name="input_termination_protection"></a> [termination\_protection](#input\_termination\_protection) | Switch on/off termination protection (default is `false`, except when using multiple master nodes). Before attempting to destroy the resource when termination protection is enabled, this configuration must be applied with its value set to `false` | `bool` | `null` | no |
| <a name="input_unhealthy_node_replacement"></a> [unhealthy\_node\_replacement](#input\_unhealthy\_node\_replacement) | Whether whether Amazon EMR should gracefully replace core nodes that have degraded within the cluster. Default value is `false` | `bool` | `null` | no |
| <a name="input_visible_to_all_users"></a> [visible\_to\_all\_users](#input\_visible\_to\_all\_users) | Whether the job flow is visible to all IAM users of the AWS account associated with the job flow. Default value is `true` | `bool` | `null` | no |
| <a name="input_vpc_id"></a> [vpc\_id](#input\_vpc\_id) | The ID of the Amazon Virtual Private Cloud (Amazon VPC) where the security groups will be created | `string` | `""` | no |

Expand Down
4 changes: 2 additions & 2 deletions examples/private-cluster/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,13 @@ Note that this example may create resources which will incur monetary charges on
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.42 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.44 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.42 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.44 |

## Modules

Expand Down
17 changes: 13 additions & 4 deletions examples/private-cluster/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,10 +146,11 @@ module "emr_instance_fleet" {
list_steps_states = ["PENDING", "RUNNING", "CANCEL_PENDING", "CANCELLED", "FAILED", "INTERRUPTED", "COMPLETED"]
log_uri = "s3://${module.s3_bucket.s3_bucket_id}/"

scale_down_behavior = "TERMINATE_AT_TASK_COMPLETION"
step_concurrency_level = 3
termination_protection = false
visible_to_all_users = true
scale_down_behavior = "TERMINATE_AT_TASK_COMPLETION"
step_concurrency_level = 3
termination_protection = false
unhealthy_node_replacement = true
visible_to_all_users = true

tags = local.tags
}
Expand Down Expand Up @@ -192,6 +193,14 @@ module "emr_instance_group" {
}
])

# Example placement group config for multiple primary node clusters
# placement_group_config = [
# {
# instance_role = "MASTER"
# placement_strategy = "SPREAD"
# }
# ]

master_instance_group = {
name = "master-group"
instance_count = 1
Expand Down
2 changes: 1 addition & 1 deletion examples/private-cluster/versions.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.42"
version = ">= 5.44"
}
}
}
4 changes: 2 additions & 2 deletions examples/public-cluster/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,13 @@ Note that this example may create resources which will incur monetary charges on
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.42 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.44 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.42 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.44 |

## Modules

Expand Down
2 changes: 1 addition & 1 deletion examples/public-cluster/versions.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.42"
version = ">= 5.44"
}
}
}
4 changes: 2 additions & 2 deletions examples/serverless-cluster/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,13 @@ Note that this example may create resources which will incur monetary charges on
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.42 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.44 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.42 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.44 |

## Modules

Expand Down
2 changes: 1 addition & 1 deletion examples/serverless-cluster/versions.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.42"
version = ">= 5.44"
}
}
}
4 changes: 2 additions & 2 deletions examples/studio/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,13 @@ $ terraform apply
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.42 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.44 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.42 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.44 |

## Modules

Expand Down
2 changes: 1 addition & 1 deletion examples/studio/versions.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.42"
version = ">= 5.44"
}
}
}
4 changes: 2 additions & 2 deletions examples/virtual-cluster/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ aws emr-containers list-virtual-clusters --region us-west-2 --states ARRESTED \
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.42 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.44 |
| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.17 |
| <a name="requirement_null"></a> [null](#requirement\_null) | >= 3.0 |
| <a name="requirement_time"></a> [time](#requirement\_time) | >= 0.7 |
Expand All @@ -54,7 +54,7 @@ aws emr-containers list-virtual-clusters --region us-west-2 --states ARRESTED \

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.42 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.44 |
| <a name="provider_null"></a> [null](#provider\_null) | >= 3.0 |
| <a name="provider_time"></a> [time](#provider\_time) | >= 0.7 |

Expand Down
2 changes: 1 addition & 1 deletion examples/virtual-cluster/versions.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.42"
version = ">= 5.44"
}
kubernetes = {
source = "hashicorp/kubernetes"
Expand Down
16 changes: 13 additions & 3 deletions main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,15 @@ resource "aws_emr_cluster" "this" {
}
}

dynamic "placement_group_config" {
for_each = var.placement_group_config

content {
instance_role = placement_group_config.value.instance_role
placement_strategy = try(placement_group_config.value.placement_strategy, null)
}
}

name = var.name
release_label = try(coalesce(var.release_label, element(data.aws_emr_release_labels.this[0].release_labels, 0)), "")
scale_down_behavior = var.scale_down_behavior
Expand Down Expand Up @@ -299,9 +308,10 @@ resource "aws_emr_cluster" "this" {
}
}

step_concurrency_level = var.step_concurrency_level
termination_protection = var.termination_protection
visible_to_all_users = var.visible_to_all_users
step_concurrency_level = var.step_concurrency_level
termination_protection = var.termination_protection
unhealthy_node_replacement = var.unhealthy_node_replacement
visible_to_all_users = var.visible_to_all_users

tags = merge(
local.tags,
Expand Down
4 changes: 2 additions & 2 deletions modules/serverless/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,13 +134,13 @@ Examples codified under the [`examples`](https://github.com/terraform-aws-module
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.62 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.44 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.62 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.44 |

## Modules

Expand Down
2 changes: 1 addition & 1 deletion modules/serverless/versions.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.62"
version = ">= 5.44"
}
}
}
4 changes: 2 additions & 2 deletions modules/studio/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,13 @@ Examples codified under the [`examples`](https://github.com/terraform-aws-module
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.42 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.44 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.42 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.44 |

## Modules

Expand Down
2 changes: 1 addition & 1 deletion modules/studio/versions.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.42"
version = ">= 5.44"
}
}
}
4 changes: 2 additions & 2 deletions modules/virtual-cluster/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,14 +82,14 @@ Examples codified under the [`examples`](https://github.com/terraform-aws-module
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.42 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.44 |
| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.10 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.42 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.44 |
| <a name="provider_kubernetes"></a> [kubernetes](#provider\_kubernetes) | >= 2.10 |

## Modules
Expand Down
2 changes: 1 addition & 1 deletion modules/virtual-cluster/versions.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.42"
version = ">= 5.44"
}
kubernetes = {
source = "hashicorp/kubernetes"
Expand Down
12 changes: 12 additions & 0 deletions variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,12 @@ variable "name" {
default = ""
}

variable "placement_group_config" {
description = "The specified placement group configuration"
type = any
default = {}
}

variable "release_label" {
description = "Release label for the Amazon EMR release"
type = string
Expand Down Expand Up @@ -169,6 +175,12 @@ variable "termination_protection" {
default = null
}

variable "unhealthy_node_replacement" {
description = "Whether whether Amazon EMR should gracefully replace core nodes that have degraded within the cluster. Default value is `false`"
type = bool
default = null
}

variable "visible_to_all_users" {
description = "Whether the job flow is visible to all IAM users of the AWS account associated with the job flow. Default value is `true`"
type = bool
Expand Down
2 changes: 1 addition & 1 deletion versions.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.42"
version = ">= 5.44"
}
}
}