feat!: Refactor count/lists into for_each/maps

[bmurphey]

Description

• Add an "accepter" resource for VPC attachments, to avoid the "auto accept shared attachments" feature when using Resource Access Manager (RAM)
• Add support for TGW Peering attachments
• Add Flow Logs for whole-TGW and/or individual TGW Peering/VPC attachments, publishing to S3 and/or CloudWatch Logs
• Convert tgw_routes from a list of maps to a map of maps, to avoid potential downtime associated with destroying routes when adding new ones
• Enable multiple TGW route tables to allow for more granular network segmentation
• Allow for adding multiple CIDR blocks to VPC route tables per-attachment, and rename the parameter from tgw_destination_cidrs to vpc_route_table_destination_cidrs to reflect its true purpose
• Add parameters to help transform implementation steps into a more cohesive order of operations
• Convert TGW route destination CIDR block to list, to allow multiple CIDR blocks per attachment
• Allow for disabling non-default route table propagation, to ensure VPC CIDR block can be left out of TGW route table when only certain subnets should be routable

Motivation and Context

This module was missing quite a few features that are necessary for true multi-account operation, as well as operational concerns like logging.

• Closes fix: Change count for for_each on ram_principal_association due resource recreation #95
• Resolves fix: Use IPv6 CIDR block destination on route when IPv6 support is enabled #102
• Resolves Significant refactor #109
• Resolves Execution fails when using vpc_attachments[].vpc_route_table_ids #111
• Closes feat: Allow specifying additional TGW routes in attached VPCs #132
• Closes feat: Add support for security group referencing to transit-gateway module #133
• Resolves Support Security Group Referencing #135
• Resolves Default association and propagation are always enabled #136

Breaking Changes

This PR includes breaking changes due to changes in data structures. The changes to data structures (converting lists/sets to maps, mostly) were necessary to prevent downtime during applies due to deleting/re-creating routes.

How Has This Been Tested?

• I have updated at least one of the examples/* to demonstrate and validate my change(s)
• I have tested and validated these changes using one or more of the provided examples/* projects
• I have executed pre-commit run -a on my pull request

I am running my refactor in production currently, using one AWS account as the hub, and several other AWS accounts as the spokes.

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[bmurphey]

@bryantbiggs anything I can help with to move this along?

[dash-aug]

Could we have a separate PR for adding flow logs? They're quite useful, and it's a shame they're "stuck" in this PR.

[ronaldour]

Does it make sense to change the behavior of the route-table submodule to decouple the associations from the propagations? I see the only way to create route_table_propagations is with:

associations = {
    vpc1 = {
      transit_gateway_attachment_id = module.transit_gateway.vpc_attachments["vpc1"].id
      propagate_route_table         = true
    }
    vpc2 = {
      transit_gateway_attachment_id = module.transit_gateway.vpc_attachments["vpc2"].id
      propagate_route_table         = true
    }
  }

but attachments can only be associated to one route table but propagated to multiple route tables.

[ronaldour]

Is it possible to make route_table_id and destination_cidr_block lists instead of a single value? in most cases there are multiple route tables that need routes to the transit gateway, and multiple cidr blocks adds flexibility too

[bryantbiggs]

absolutely! any details on the usage would be appreciated - Anton and I heavily use various services so we are quite familiar with them, but on something like this module, we're mostly just "Terraform experts" and have less insight into how folks use TGW or how they want to configure TGW using this module. That insight is valuable in helping design the module to support those use cases (where possible)

[flah00]

@bryantbiggs good to know :)

We're trying to use the module to create infra a lot like this

We are trying to setup transit gateways so we can route OU VPCs to our VPN. We would like to connect organizational units using transit gateway.

Details

We are using AWS organizations. We have a network service account and we have "spoke" accounts as well.

A simplified network map, which only includes nonprod (excludes prod and uat)
https://viewer.diagrams.net/?tags=%7B%7D&lightbox=1&highlight=B3B3B3&edit=_blank&layers=1&nav=1#G1BsHvhzBNZI_-grnqDZNiJv6dCk_A1K8-#%7B%22pageId%22%3A%22-E4dg9jivi15Yi2I4Ksi%22%7D

Our goal was to create a single tgw in each of the regions we use in the network service account. Each transit gateway would have three routing tables called prod, nonpod, and UAT.

Within the network service account we would also create three VPCs called prod, nonprod, and UAT. Each of these VPC subnets would attach to the tgw and associate with the route table of the same name.

We also created a RAM share for our OU VPCs, like sandbox.

The problems

Maybe I've misunderstood something, but I think the multi account example could be improved? It does not make use of the non-default route table. In my experience the use of the non-default route table is more complex. A working example would be helpful.

We are not currently able to manage multiple tgw route tables using the module as it is.

Managing spoke attachments (eg sandbox vpc in a different OU) seems opaque within the module. I can create the attachment in the sandbox OU using the module.

EDIT
Removing this observation: But I don't think I can use the module in the network service account to manage the sandbox attachment association/propagation

[flah00]

In my experience, using terraform to manage spoke attachments in different OU that do not use the default route table requires a specific configuration.. But maybe I've missed some critical documentation? I don't think I've seen this use case documented anywhere?

terraform apply 1

1. TF provider account 1: Create tgw and disable default route table association and propagation, RAM share it with other OU
2. TF provider account 1: Create tgw route tables for tgw
3. TF provider account 1: Create tgw attachment association & route propagation for VPCs within account 1

terraform apply 2

1. TF provider account 2: Create tgw attachment
2. TF provider alias to account 1: Create tgw attachment association & route propagation

It would be great if there were a module that handled the second apply.
The critical part is gracefully supporting calling out to the OU where the route table lives.

[flah00]

@bryantbiggs here's an example that demonstrates how we handle terraform apply number 2 described above (spoiler we settled on a provider alias using a different profile)

https://gist.github.com/flah00/fa840591fbb6b131f2c23d020bdc9c30

[flah00]

@ronaldour it seems like this branch is seeing some activity again.
I'd love to get my hands on these features.
What are the odds we can seize the momentum and get this merged?

[ronaldour]

@flah00 well I'm not part of the maintainers but happy to help with anything to move this forward. @bryantbiggs anything we can help improve here?

[ronaldour]

I've been working on an opinionated refactor using this as the base. I focused on simplifying how de module is used since Transit Gateways are fairly complex. https://github.com/ronaldour/terraform-aws-transit-gateway/tree/refactor
Some of the features I added:

• Multiple Route Tables and Routes
• Easier to define attachments
• Multiple VPC routes to Transit Gateway
• For_each instead of counts
• Multi-account multi-region example

If you want to give it a try, feedback is welcome. Check out the Examples

[bryantbiggs]

we'll proceed once AWS provider v6 arrives - this was originally meant to arrive in April but seems to be taking a bit longer