feat: Add support for multi-vpc private connectivity (#13)

Co-authored-by: Bryant Biggs <[email protected]>

Author: carldjohnston

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/antonbabenko/pre-commit-terraform
-    rev: v1.81.0
+    rev: v1.83.2
     hooks:
       - id: terraform_fmt
       - id: terraform_validate

README.md

@@ -128,13 +128,13 @@ Examples codified under the [`examples`](https://github.com/terraform-aws-module
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.12 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.0 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.12 |
 
 ## Modules
 
@@ -152,6 +152,7 @@ No modules.
 | [aws_msk_cluster.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_cluster) | resource |
 | [aws_msk_configuration.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_configuration) | resource |
 | [aws_msk_scram_secret_association.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_scram_secret_association) | resource |
+| [aws_msk_vpc_connection.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_vpc_connection) | resource |
 | [aws_mskconnect_custom_plugin.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mskconnect_custom_plugin) | resource |
 | [aws_mskconnect_worker_configuration.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mskconnect_worker_configuration) | resource |
 
@@ -209,6 +210,7 @@ No modules.
 | <a name="input_storage_mode"></a> [storage\_mode](#input\_storage\_mode) | Controls storage mode for supported storage tiers. Valid values are: `LOCAL` or `TIERED` | `string` | `null` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | A map of tags to assign to the resources created | `map(string)` | `{}` | no |
 | <a name="input_timeouts"></a> [timeouts](#input\_timeouts) | Create, update, and delete timeout configurations for the cluster | `map(string)` | `{}` | no |
+| <a name="input_vpc_connections"></a> [vpc\_connections](#input\_vpc\_connections) | Map of VPC Connections to create | `any` | `{}` | no |
 
 ## Outputs
 
@@ -233,6 +235,7 @@ No modules.
 | <a name="output_schema_registries"></a> [schema\_registries](#output\_schema\_registries) | A map of output attributes for the schema registries created |
 | <a name="output_schemas"></a> [schemas](#output\_schemas) | A map of output attributes for the schemas created |
 | <a name="output_scram_secret_association_id"></a> [scram\_secret\_association\_id](#output\_scram\_secret\_association\_id) | Amazon Resource Name (ARN) of the MSK cluster |
+| <a name="output_vpc_connections"></a> [vpc\_connections](#output\_vpc\_connections) | A map of output attributes for the VPC connections created |
 | <a name="output_zookeeper_connect_string"></a> [zookeeper\_connect\_string](#output\_zookeeper\_connect\_string) | A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster. The returned values are sorted alphabetically |
 | <a name="output_zookeeper_connect_string_tls"></a> [zookeeper\_connect\_string\_tls](#output\_zookeeper\_connect\_string\_tls) | A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster via TLS. The returned values are sorted alphabetically |
 <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

examples/basic/README.md

@@ -23,13 +23,13 @@ Note that this example may create resources which will incur monetary charges on
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.12 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.0 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.12 |
 
 ## Modules
 

examples/basic/versions.tf

@@ -4,7 +4,7 @@ terraform {
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 5.0"
+      version = ">= 5.12"
     }
   }
 }

examples/complete/README.md

@@ -22,14 +22,14 @@ Note that this example may create resources which will incur monetary charges on
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.12 |
 | <a name="requirement_random"></a> [random](#requirement\_random) | >= 2.0 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.0 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.12 |
 | <a name="provider_random"></a> [random](#provider\_random) | >= 2.0 |
 
 ## Modules
@@ -40,6 +40,8 @@ Note that this example may create resources which will incur monetary charges on
 | <a name="module_s3_logs_bucket"></a> [s3\_logs\_bucket](#module\_s3\_logs\_bucket) | terraform-aws-modules/s3-bucket/aws | ~> 3.0 |
 | <a name="module_security_group"></a> [security\_group](#module\_security\_group) | terraform-aws-modules/security-group/aws | ~> 5.0 |
 | <a name="module_vpc"></a> [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 5.0 |
+| <a name="module_vpc_connection"></a> [vpc\_connection](#module\_vpc\_connection) | terraform-aws-modules/vpc/aws | ~> 5.0 |
+| <a name="module_vpc_connection_security_group"></a> [vpc\_connection\_security\_group](#module\_vpc\_connection\_security\_group) | terraform-aws-modules/security-group/aws | ~> 5.0 |
 
 ## Resources
 

examples/complete/main.tf

@@ -8,8 +8,9 @@ locals {
   name   = "ex-${basename(path.cwd)}"
   region = "us-east-1"
 
-  vpc_cidr = "10.0.0.0/16"
-  azs      = slice(data.aws_availability_zones.available.names, 0, 3)
+  vpc_cidr            = "10.0.0.0/16"
+  vpc_connection_cidr = "10.1.0.0/16"
+  azs                 = slice(data.aws_availability_zones.available.names, 0, 3)
 
   secrets = ["producer", "consumer"]
 
@@ -32,13 +33,36 @@ module "msk_cluster" {
   number_of_broker_nodes = 3
   enhanced_monitoring    = "PER_TOPIC_PER_PARTITION"
 
-  broker_node_client_subnets  = module.vpc.private_subnets
-  broker_node_instance_type   = "kafka.t3.small"
+  broker_node_client_subnets = module.vpc.private_subnets
+  broker_node_connectivity_info = {
+    public_access = {
+      type = "DISABLED"
+    }
+    vpc_connectivity = {
+      client_authentication = {
+        tls = false
+        sasl = {
+          iam   = false
+          scram = true
+        }
+      }
+    }
+  }
+  broker_node_instance_type   = "kafka.m5.large"
   broker_node_security_groups = [module.security_group.security_group_id]
   broker_node_storage_info = {
     ebs_storage_info = { volume_size = 100 }
   }
 
+  vpc_connections = {
+    connection_one = {
+      authentication  = "SASL_SCRAM"
+      vpc_id          = module.vpc_connection.vpc_id
+      client_subnets  = module.vpc_connection.private_subnets
+      security_groups = [module.vpc_connection_security_group.security_group_id]
+    }
+  }
+
   encryption_in_transit_client_broker = "TLS"
   encryption_in_transit_in_cluster    = true
 
@@ -233,3 +257,43 @@ module "s3_logs_bucket" {
 
   tags = local.tags
 }
+
+################################################################################
+# VPC Connections
+################################################################################
+
+module "vpc_connection_security_group" {
+  source  = "terraform-aws-modules/security-group/aws"
+  version = "~> 5.0"
+
+  name        = "${local.name}-vpc-connection"
+  description = "Security group for ${local.name} VPC Connection"
+  vpc_id      = module.vpc_connection.vpc_id
+
+  ingress_cidr_blocks = module.vpc_connection.private_subnets_cidr_blocks
+  ingress_rules = [
+    "kafka-broker-tcp",
+    "kafka-broker-tls-tcp"
+  ]
+
+  tags = local.tags
+}
+
+module "vpc_connection" {
+  source  = "terraform-aws-modules/vpc/aws"
+  version = "~> 5.0"
+
+  name = "${local.name}-vpc-connection"
+  cidr = local.vpc_connection_cidr
+
+  azs              = local.azs
+  public_subnets   = [for k, v in local.azs : cidrsubnet(local.vpc_connection_cidr, 8, k)]
+  private_subnets  = [for k, v in local.azs : cidrsubnet(local.vpc_connection_cidr, 8, k + 3)]
+  database_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_connection_cidr, 8, k + 6)]
+
+  create_database_subnet_group = true
+  enable_nat_gateway           = true
+  single_nat_gateway           = true
+
+  tags = local.tags
+}

examples/complete/versions.tf

@@ -4,7 +4,7 @@ terraform {
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 5.0"
+      version = ">= 5.12"
     }
     random = {
       source  = "hashicorp/random"

examples/connect/README.md

@@ -22,14 +22,14 @@ Note that this example may create resources which will incur monetary charges on
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.12 |
 | <a name="requirement_null"></a> [null](#requirement\_null) | >= 3.0 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.0 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.12 |
 | <a name="provider_null"></a> [null](#provider\_null) | >= 3.0 |
 
 ## Modules

examples/connect/versions.tf

@@ -4,7 +4,7 @@ terraform {
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 5.0"
+      version = ">= 5.12"
     }
     null = {
       source  = "hashicorp/null"

main.tf

@@ -20,6 +20,29 @@ resource "aws_msk_cluster" "this" {
             type = try(public_access.value.type, null)
           }
         }
+
+        dynamic "vpc_connectivity" {
+          for_each = try([connectivity_info.value.vpc_connectivity], [])
+
+          content {
+            dynamic "client_authentication" {
+              for_each = try([vpc_connectivity.value.client_authentication], [])
+
+              content {
+                dynamic "sasl" {
+                  for_each = try([client_authentication.value.sasl], [])
+
+                  content {
+                    iam   = try(sasl.value.iam, null)
+                    scram = try(sasl.value.scram, null)
+                  }
+                }
+
+                tls = try(client_authentication.value.tls, null)
+              }
+            }
+          }
+        }
       }
     }
 
@@ -145,6 +168,22 @@ resource "aws_msk_cluster" "this" {
   tags = var.tags
 }
 
+################################################################################
+# VPC Connection
+################################################################################
+
+resource "aws_msk_vpc_connection" "this" {
+  for_each = { for k, v in var.vpc_connections : k => v if var.create }
+
+  authentication     = each.value.authentication
+  client_subnets     = each.value.client_subnets
+  security_groups    = each.value.security_groups
+  target_cluster_arn = aws_msk_cluster.this[0].arn
+  vpc_id             = each.value.vpc_id
+
+  tags = merge(var.tags, try(each.value.tags, {}))
+}
+
 ################################################################################
 # Configuration
 ################################################################################

outputs.tf

@@ -52,6 +52,15 @@ output "zookeeper_connect_string_tls" {
   value       = try(aws_msk_cluster.this[0].zookeeper_connect_string_tls, null)
 }
 
+################################################################################
+# VPC Connection
+################################################################################
+
+output "vpc_connections" {
+  description = "A map of output attributes for the VPC connections created"
+  value       = aws_msk_vpc_connection.this
+}
+
 ################################################################################
 # Configuration
 ################################################################################

variables.tf

@@ -158,6 +158,16 @@ variable "timeouts" {
   default     = {}
 }
 
+################################################################################
+# VPC Connection
+################################################################################
+
+variable "vpc_connections" {
+  description = "Map of VPC Connections to create"
+  type        = any
+  default     = {}
+}
+
 ################################################################################
 # Configuration
 ################################################################################

versions.tf

@@ -4,7 +4,7 @@ terraform {
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 5.0"
+      version = ">= 5.12"
     }
   }
 }