feat!: update TPG version constraints to 4.0 (#1129)

* feat: update TPG version constraints to allow 4.0

* Removes basic auth, renames namespace_identity

* Regenerates modules and documentation

* Updates tests to use latest Google provider

* addresses warning about multiple provider blocks

* Updates network module for Google provider 4.0 compatibility

* Temporarily uses "main" for gcloud module (until next release is cut)

* Comments out version constraint (temporary change)

* fetches main branch by default?

* Uses master branch for gcloud module (until release is cut)

* Uses kubectl-wrapper where appropriate

* Uses released version of gcloud module

* Returns instance group URLs per node pool

* Extends use of cluster_output_node_pools_ variables

* Fixes documentation

* Updates more modules

* Updates READMEs to match variables

* Uses master branch of bastion

* temporary change until new version is released

* Updates node pools versions description

* Adds locals for node pool instance group URLs

* Uses master branch of terraform-google-project-factory

* temporary change until new version of that dependency is released

* Updates project version ready for release

* Updates pinned version of Google provider for example

* Updates pinned version of Google provider in example

* Addresses code review comments

* Temporarily applies an empty source_tags setting.

* this should be removed once hashicorp/terraform-provider-google#10494 is addressed

* Fixes indentation

* Uses newly-released version of project factory

* Uses released version of bastion host

* Removes use of SECURE mode (deprecated)

* test empty source tag workaround

* fix wi test

* refactor IAM test for loose match

* map old node meta value, add validations

* update docs

* Update autogen/main/variables.tf.tmpl

Co-authored-by: Morgante Pell <[email protected]>

* remove local

Co-authored-by: cloud-foundation-bot <[email protected]>
Co-authored-by: Jack Whelpton <[email protected]>
Co-authored-by: Morgante Pell <[email protected]>

Author: 4 people

README.md

@@ -128,8 +128,6 @@ Then perform the following commands on the root folder:
 | add\_master\_webhook\_firewall\_rules | Create master\_webhook firewall rules for ports defined in `firewall_inbound_ports` | `bool` | `false` | no |
 | add\_shadow\_firewall\_rules | Create GKE shadow firewall (the same as default firewall rules with firewall logs enabled). | `bool` | `false` | no |
 | authenticator\_security\_group | The name of the RBAC security group for use with Google security groups in Kubernetes RBAC. Group name must be in format [email protected] | `string` | `null` | no |
-| basic\_auth\_password | The password to be used with Basic Authentication. | `string` | `""` | no |
-| basic\_auth\_username | The username to be used with Basic Authentication. An empty value will disable Basic Authentication, which is the recommended configuration. | `string` | `""` | no |
 | cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br>    enabled       = bool<br>    min_cpu_cores = number<br>    max_cpu_cores = number<br>    min_memory_gb = number<br>    max_memory_gb = number<br>    gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))<br>  })</pre> | <pre>{<br>  "enabled": false,<br>  "gpu_resources": [],<br>  "max_cpu_cores": 0,<br>  "max_memory_gb": 0,<br>  "min_cpu_cores": 0,<br>  "min_memory_gb": 0<br>}</pre> | no |
 | cluster\_ipv4\_cidr | The IP address range of the kubernetes pods in this cluster. Default is an automatically assigned CIDR. | `any` | `null` | no |
 | cluster\_resource\_labels | The GCE resource labels (a map of key/value pairs) to be applied to the cluster | `map(string)` | `{}` | no |
@@ -151,7 +149,7 @@ Then perform the following commands on the root folder:
 | grant\_registry\_access | Grants created cluster-specific service account storage.objectViewer and artifactregistry.reader roles. | `bool` | `false` | no |
 | horizontal\_pod\_autoscaling | Enable horizontal pod autoscaling addon | `bool` | `true` | no |
 | http\_load\_balancing | Enable httpload balancer addon | `bool` | `true` | no |
-| identity\_namespace | Workload Identity namespace. (Default value of `enabled` automatically sets project based namespace `[project_id].svc.id.goog`) | `string` | `"enabled"` | no |
+| identity\_namespace | The workload pool to attach all Kubernetes service accounts to. (Default value of `enabled` automatically sets project-based pool `[project_id].svc.id.goog`) | `string` | `"enabled"` | no |
 | impersonate\_service\_account | An optional service account to impersonate for gcloud commands. If this service account is not specified, the module will use Application Default Credentials. | `string` | `""` | no |
 | initial\_node\_count | The number of nodes to create in this cluster's default node pool. | `number` | `0` | no |
 | ip\_masq\_link\_local | Whether to masquerade traffic to the link-local prefix (169.254.0.0/16). | `bool` | `false` | no |
@@ -170,7 +168,7 @@ Then perform the following commands on the root folder:
 | network\_policy | Enable network policy addon | `bool` | `false` | no |
 | network\_policy\_provider | The network policy provider. | `string` | `"CALICO"` | no |
 | network\_project\_id | The project ID of the shared VPC's host (for shared vpc support) | `string` | `""` | no |
-| node\_metadata | Specifies how node metadata is exposed to the workload running on the node | `string` | `"GKE_METADATA_SERVER"` | no |
+| node\_metadata | Specifies how node metadata is exposed to the workload running on the node | `string` | `"GKE_METADATA"` | no |
 | node\_pools | List of maps containing node pools | `list(map(string))` | <pre>[<br>  {<br>    "name": "default-node-pool"<br>  }<br>]</pre> | no |
 | node\_pools\_labels | Map of maps containing node labels by node-pool name | `map(map(string))` | <pre>{<br>  "all": {},<br>  "default-node-pool": {}<br>}</pre> | no |
 | node\_pools\_metadata | Map of maps containing node metadata by node-pool name | `map(map(string))` | <pre>{<br>  "all": {},<br>  "default-node-pool": {}<br>}</pre> | no |
@@ -202,7 +200,7 @@ Then perform the following commands on the root folder:
 | endpoint | Cluster endpoint |
 | horizontal\_pod\_autoscaling\_enabled | Whether horizontal pod autoscaling enabled |
 | http\_load\_balancing\_enabled | Whether http load balancing enabled |
-| identity\_namespace | Workload Identity namespace |
+| identity\_namespace | Workload Identity pool |
 | instance\_group\_urls | List of GKE generated instance groups |
 | location | Cluster location (region if regional cluster, zone if zonal cluster) |
 | logging\_service | Logging service used |
@@ -213,7 +211,7 @@ Then perform the following commands on the root folder:
 | name | Cluster name |
 | network\_policy\_enabled | Whether network policy enabled |
 | node\_pools\_names | List of node pools names |
-| node\_pools\_versions | List of node pools versions |
+| node\_pools\_versions | Node pool versions by node pool name |
 | region | Cluster region |
 | release\_channel | The release channel of this cluster |
 | service\_account | The service account to default running nodes as if not overridden in `node_pools`. |

autogen/main/cluster.tf.tmpl

@@ -161,9 +161,6 @@ resource "google_container_cluster" "primary" {
   }
 
   master_auth {
-    username = var.basic_auth_username
-    password = var.basic_auth_password
-
     client_certificate_config {
       issue_client_certificate = var.issue_client_certificate
     }
@@ -298,7 +295,7 @@ resource "google_container_cluster" "primary" {
         for_each = local.cluster_node_metadata_config
 
         content {
-          node_metadata = workload_metadata_config.value.node_metadata
+          mode = workload_metadata_config.value.mode
         }
       }
 
@@ -377,7 +374,7 @@ resource "google_container_cluster" "primary" {
     for_each = local.cluster_workload_identity_config
 
     content {
-      identity_namespace = workload_identity_config.value.identity_namespace
+      workload_pool = workload_identity_config.value.workload_pool
     }
   }
 
@@ -634,9 +631,10 @@ resource "google_container_node_pool" "pools" {
       for_each = local.cluster_node_metadata_config
 
       content {
-        node_metadata = lookup(each.value, "node_metadata", workload_metadata_config.value.node_metadata)
+        mode = lookup(each.value, "node_metadata", workload_metadata_config.value.mode)
       }
     }
+
     {% if beta_cluster %}
     dynamic "sandbox_config" {
       for_each = tobool((lookup(each.value, "sandbox_enabled", var.sandbox_enabled))) ? ["gvisor"] : []

autogen/main/dns.tf.tmpl

@@ -20,8 +20,9 @@
   Delete default kube-dns configmap
  *****************************************/
 module "gcloud_delete_default_kube_dns_configmap" {
-  source                      = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper"
-  version                     = "~> 2.1.0"
+  source  = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper"
+  version = "~> 3.1"
+
   enabled                     = (local.custom_kube_dns_config || local.upstream_nameservers_config) && !var.skip_provisioners
   cluster_name                = google_container_cluster.primary.name
   cluster_location            = google_container_cluster.primary.location

autogen/main/firewall.tf.tmpl

@@ -112,6 +112,7 @@ resource "google_compute_firewall" "master_webhooks" {
   direction   = "INGRESS"
 
   source_ranges = [local.cluster_endpoint_for_nodes]
+  source_tags   = []
   target_tags   = [local.cluster_network_tag]
 
   allow {

autogen/main/main.tf.tmpl

@@ -111,8 +111,11 @@ locals {
     security_group = var.authenticator_security_group
   }]
 
+  // legacy mappings https://github.com/hashicorp/terraform-provider-google/pull/10238
+  old_node_metadata_config_mapping = { GKE_METADATA_SERVER = "GKE_METADATA", GCE_METADATA = "EXPOSE" }
+
   cluster_node_metadata_config = var.node_metadata == "UNSPECIFIED" ? [] : [{
-    node_metadata = var.node_metadata
+    mode = lookup(local.old_node_metadata_config_mapping, var.node_metadata, var.node_metadata)
   }]
 
   cluster_output_name           = google_container_cluster.primary.name
@@ -153,7 +156,7 @@ locals {
   }]
 
   cluster_output_node_pools_names    = concat([for np in google_container_node_pool.pools : np.name], [""])
-  cluster_output_node_pools_versions = concat([for np in google_container_node_pool.pools : np.version], [""])
+  cluster_output_node_pools_versions = { for np in google_container_node_pool.pools : np.name => np.version }
 
   cluster_master_auth_list_layer1 = local.cluster_output_master_auth
   cluster_master_auth_list_layer2 = local.cluster_master_auth_list_layer1[0]
@@ -177,7 +180,7 @@ locals {
   cluster_horizontal_pod_autoscaling_enabled = !local.cluster_output_horizontal_pod_autoscaling_enabled
   workload_identity_enabled                  = !(var.identity_namespace == null || var.identity_namespace == "null")
   cluster_workload_identity_config = ! local.workload_identity_enabled ? [] : var.identity_namespace == "enabled" ? [{
-    identity_namespace = "${var.project_id}.svc.id.goog" }] : [{ identity_namespace = var.identity_namespace
+    workload_pool = "${var.project_id}.svc.id.goog" }] : [{ workload_pool = var.identity_namespace
   }]
 {% if beta_cluster %}
   # BETA features

autogen/main/outputs.tf.tmpl

@@ -114,7 +114,7 @@ output "node_pools_names" {
 }
 
 output "node_pools_versions" {
-  description = "List of node pools versions"
+  description = "Node pool versions by node pool name"
   value       = local.cluster_node_pools_versions
 }
 
@@ -123,23 +123,23 @@ output "service_account" {
   value       = local.service_account
 }
 
+output "instance_group_urls" {
+  description = "List of GKE generated instance groups"
+  value       = distinct(flatten([for np in google_container_node_pool.pools : np.managed_instance_group_urls]))
+}
+
 output "release_channel" {
   description = "The release channel of this cluster"
   value       = var.release_channel
 }
 
 output "identity_namespace" {
-  description = "Workload Identity namespace"
-  value       = length(local.cluster_workload_identity_config) > 0 ? local.cluster_workload_identity_config[0].identity_namespace : null
+  description = "Workload Identity pool"
+  value       = length(local.cluster_workload_identity_config) > 0 ? local.cluster_workload_identity_config[0].workload_pool : null
   depends_on = [
     google_container_cluster.primary
   ]
 }
-
-output "instance_group_urls" {
-  description = "List of GKE generated instance groups"
-  value       = google_container_cluster.primary.instance_group_urls
-}
 {% if private_cluster %}
 
 output "master_ipv4_cidr_block" {

autogen/main/variables.tf.tmpl

@@ -394,18 +394,6 @@ variable "service_account" {
   default     = ""
 }
 
-variable "basic_auth_username" {
-  type        = string
-  description = "The username to be used with Basic Authentication. An empty value will disable Basic Authentication, which is the recommended configuration."
-  default     = ""
-}
-
-variable "basic_auth_password" {
-  type        = string
-  description = "The password to be used with Basic Authentication."
-  default     = ""
-}
-
 variable "issue_client_certificate" {
   type        = bool
   description = "Issues a client certificate to authenticate to the cluster endpoint. To maximize the security of your cluster, leave this option disabled. Client certificates don't automatically rotate and aren't easily revocable. WARNING: changing this after cluster creation is destructive!"
@@ -549,8 +537,13 @@ variable "authenticator_security_group" {
 
 variable "node_metadata" {
   description = "Specifies how node metadata is exposed to the workload running on the node"
-  default     = "GKE_METADATA_SERVER"
+  default     = "GKE_METADATA"
   type        = string
+
+  validation {
+    condition     = contains(["GKE_METADATA", "GCE_METADATA", "UNSPECIFIED", "GKE_METADATA_SERVER", "EXPOSE"], var.node_metadata)
+    error_message = "The node_metadata value must be one of GKE_METADATA, GCE_METADATA or UNSPECIFIED."
+  }
 }
 
 variable "database_encryption" {
@@ -564,7 +557,7 @@ variable "database_encryption" {
 }
 
 variable "identity_namespace" {
-  description = "Workload Identity namespace. (Default value of `enabled` automatically sets project based namespace `[project_id].svc.id.goog`)"
+  description = "The workload pool to attach all Kubernetes service accounts to. (Default value of `enabled` automatically sets project-based pool `[project_id].svc.id.goog`)"
   type        = string
   default     = "enabled"
 }

autogen/main/versions.tf.tmpl

@@ -24,7 +24,7 @@ terraform {
   required_providers {
     google-beta = {
       source  = "hashicorp/google-beta"
-      version = ">= 3.87.0, <4.0.0"
+      version = ">= 4.0.0, < 5.0"
     }
     kubernetes = {
       source  = "hashicorp/kubernetes"
@@ -38,7 +38,7 @@ terraform {
   required_providers {
     google = {
       source  = "hashicorp/google"
-      version = ">= 3.55.0, <4.0.0"
+      version = ">= 4.0.0, < 5.0"
     }
     kubernetes = {
       source  = "hashicorp/kubernetes"

autogen/safer-cluster/main.tf.tmpl

@@ -111,10 +111,6 @@ module "gke" {
   registry_project_ids   = var.registry_project_ids
   grant_registry_access  = var.grant_registry_access
 
-  // Basic Auth disabled
-  basic_auth_username = ""
-  basic_auth_password = ""
-
   issue_client_certificate = false
 
   cluster_resource_labels = var.cluster_resource_labels
@@ -164,7 +160,7 @@ module "gke" {
 
   enable_vertical_pod_autoscaling = var.enable_vertical_pod_autoscaling
 
-  // We enable identity namespace by default.
+  // We enable Workload Identity by default.
   identity_namespace = "${var.project_id}.svc.id.goog"
 
   authenticator_security_group = var.authenticator_security_group

autogen/safer-cluster/outputs.tf.tmpl

@@ -104,7 +104,7 @@ output "node_pools_names" {
 }
 
 output "node_pools_versions" {
-  description = "List of node pools versions"
+  description = "Node pool versions by node pool name"
   value       = module.gke.node_pools_versions
 }
 

cluster.tf

@@ -98,9 +98,6 @@ resource "google_container_cluster" "primary" {
   }
 
   master_auth {
-    username = var.basic_auth_username
-    password = var.basic_auth_password
-
     client_certificate_config {
       issue_client_certificate = var.issue_client_certificate
     }
@@ -165,7 +162,7 @@ resource "google_container_cluster" "primary" {
         for_each = local.cluster_node_metadata_config
 
         content {
-          node_metadata = workload_metadata_config.value.node_metadata
+          mode = workload_metadata_config.value.mode
         }
       }
 
@@ -211,7 +208,7 @@ resource "google_container_cluster" "primary" {
     for_each = local.cluster_workload_identity_config
 
     content {
-      identity_namespace = workload_identity_config.value.identity_namespace
+      workload_pool = workload_identity_config.value.workload_pool
     }
   }
 
@@ -339,10 +336,11 @@ resource "google_container_node_pool" "pools" {
       for_each = local.cluster_node_metadata_config
 
       content {
-        node_metadata = lookup(each.value, "node_metadata", workload_metadata_config.value.node_metadata)
+        mode = lookup(each.value, "node_metadata", workload_metadata_config.value.mode)
       }
     }
 
+
     shielded_instance_config {
       enable_secure_boot          = lookup(each.value, "enable_secure_boot", false)
       enable_integrity_monitoring = lookup(each.value, "enable_integrity_monitoring", true)

dns.tf

@@ -20,8 +20,9 @@
   Delete default kube-dns configmap
  *****************************************/
 module "gcloud_delete_default_kube_dns_configmap" {
-  source                      = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper"
-  version                     = "~> 2.1.0"
+  source  = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper"
+  version = "~> 3.1"
+
   enabled                     = (local.custom_kube_dns_config || local.upstream_nameservers_config) && !var.skip_provisioners
   cluster_name                = google_container_cluster.primary.name
   cluster_location            = google_container_cluster.primary.location

docs/upgrading_to_v18.0.md

@@ -0,0 +1,66 @@
+# Upgrading to v18.0
+
+The v18.0 release of *kubernetes-engine* is a backwards incompatible release.
+
+### Google Cloud Platform Provider upgrade
+The Terraform Kubernetes Engine Module now requires version 4.0 or higher of
+the Google Cloud Platform Provider.
+
+```diff
+terraform {
+  required_providers {
+    google = {
+      source  = "hashicorp/google"
+-      version = "~> 3.0"
++      version = "~> 4.0"
+    }
+    google-beta = {
+      source  = "hashicorp/google-beta"
+-      version = "~> 3.0"
++      version = "~> 4.0"
+    }
+
+  }
+}
+```
+
+### Kubernetes Basic Authentication removed
+Basic authentication is deprecated and has been removed in GKE 1.19 and later.
+Owing to this, the `basic_auth_username` and `basic_auth_password` variables
+have been eliminated.
+
+```diff
+ module "gke" {
+   source  = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
+-  version = "~> 17.0"
++  version = "~> 18.0"
+
+-  basic_auth_username = "admin"
+-  basic_auth_password = "s3crets!"
+}
+```
+
+### Acceptable values for node_metadata modified
+It is recommended to update `node_metadata` variable to one of `GKE_METADATA`,
+`GCE_METADATA` or `UNSPECIFIED`. `GKE_METADATA` replaces the previous
+`GKE_METADATA_SERVER` value, `GCE_METADATA` should be used in place of
+`EXPOSE`, however old values continue to be supported for backwards compatibility.
+The `SECURE` option, previously deprecated, has now been removed.
+
+```diff
+module "gke" {
+  source = "../../modules/safer-cluster"
+
+  node_pools = [
+    {
+
+-     node_metadata = "GKE_METADATA_SERVER"
++     node_metadata = "GKE_METADATA"
+    }
+  ]
+}
+```
+
+### node_pools_versions is now keyed by node-pool name
+The `node_pools_versions` output is now an object keyed by node pool name,
+rather than a list as previously.

examples/acm-terraform-blog-part1/terraform/gke.tf

@@ -16,7 +16,7 @@
 
 module "enabled_google_apis" {
   source  = "terraform-google-modules/project-factory/google//modules/project_services"
-  version = "~> 10.0"
+  version = "~> 11.3"
 
   project_id                  = var.project
   disable_services_on_destroy = false