Skip to content

Commit 6380d5b

Browse files
IIBenIIbberriot
IIBenII
authored and
bberriot
committed
Rebase
2 parents 26a479d + bcd5e03 commit 6380d5b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

75 files changed

+1266
-134
lines changed

‎.github/renovate.json

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,24 +15,24 @@
1515
],
1616
"stabilityDays":0
1717
},
18+
"separateMajorMinor":false,
1819
"packageRules": [
1920
{
2021
"matchPaths": ["examples/**", "test/**", ".github/**"],
2122
"extends": [":semanticCommitTypeAll(chore)"]
2223
},
24+
{
25+
"matchPaths": ["*", "modules/**"],
26+
"extends": [":semanticCommitTypeAll(fix)"]
27+
},
2328
{
2429
"matchDepTypes": ["module"],
25-
"groupName": "TF modules",
26-
"separateMajorMinor":false,
27-
"major": {
28-
"semanticCommitType": "feat!"
29-
}
30+
"groupName": "TF modules"
3031
},
3132
{
3233
"matchDepTypes": ["require"],
3334
"groupName": "GO modules",
34-
"postUpdateOptions": ["gomodTidy"],
35-
"separateMajorMinor":false
35+
"postUpdateOptions": ["gomodTidy"]
3636
},
3737
{
3838
"matchPackageNames": ["go"],
@@ -41,8 +41,7 @@
4141
},
4242
{
4343
"matchPackageNames": ["google", "google-beta"],
44-
"groupName": "terraform googles",
45-
"separateMajorMinor": false
44+
"groupName": "terraform googles"
4645
}
4746
]
4847
}

‎README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,7 @@ Then perform the following commands on the root folder:
188188
| node\_pools\_labels | Map of maps containing node labels by node-pool name | `map(map(string))` | <pre>{<br> "all": {},<br> "default-node-pool": {}<br>}</pre> | no |
189189
| node\_pools\_metadata | Map of maps containing node metadata by node-pool name | `map(map(string))` | <pre>{<br> "all": {},<br> "default-node-pool": {}<br>}</pre> | no |
190190
| node\_pools\_oauth\_scopes | Map of lists containing node oauth scopes by node-pool name | `map(list(string))` | <pre>{<br> "all": [<br> "https://www.googleapis.com/auth/cloud-platform"<br> ],<br> "default-node-pool": []<br>}</pre> | no |
191+
| node\_pools\_resource\_labels | Map of maps containing resource labels by node-pool name | `map(map(string))` | <pre>{<br> "all": {},<br> "default-node-pool": {}<br>}</pre> | no |
191192
| node\_pools\_tags | Map of lists containing node network tags by node-pool name | `map(list(string))` | <pre>{<br> "all": [],<br> "default-node-pool": []<br>}</pre> | no |
192193
| node\_pools\_taints | Map of lists containing node taints by node-pool name | `map(list(object({ key = string, value = string, effect = string })))` | <pre>{<br> "all": [],<br> "default-node-pool": []<br>}</pre> | no |
193194
| non\_masquerade\_cidrs | List of strings in CIDR notation that specify the IP address ranges that do not use IP masquerading. | `list(string)` | <pre>[<br> "10.0.0.0/8",<br> "172.16.0.0/12",<br> "192.168.0.0/16"<br>]</pre> | no |
@@ -201,6 +202,7 @@ Then perform the following commands on the root folder:
201202
| resource\_usage\_export\_dataset\_id | The ID of a BigQuery Dataset for using BigQuery as the destination of resource usage export. | `string` | `""` | no |
202203
| service\_account | The service account to run nodes as if not overridden in `node_pools`. The create\_service\_account variable default value (true) will cause a cluster-specific service account to be created. | `string` | `""` | no |
203204
| service\_external\_ips | Whether external ips specified by a service will be allowed in this cluster | `bool` | `false` | no |
205+
| shadow\_firewall\_rules\_log\_config | The log\_config for shadow firewall rules. You can set this variable to `null` to disable logging. | <pre>object({<br> metadata = string<br> })</pre> | <pre>{<br> "metadata": "INCLUDE_ALL_METADATA"<br>}</pre> | no |
204206
| shadow\_firewall\_rules\_priority | The firewall priority of GKE shadow firewall rules. The priority should be less than default firewall, which is 1000. | `number` | `999` | no |
205207
| skip\_provisioners | Flag to skip all local-exec provisioners. It breaks `stub_domains` and `upstream_nameservers` variables functionality. | `bool` | `false` | no |
206208
| stub\_domains | Map of stub domains and their resolvers to forward DNS queries for a certain domain to an external DNS server | `map(list(string))` | `{}` | no |

‎autogen/main/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,7 @@ The node_pools variable takes the following parameters:
218218
| name | The name of the node pool | | Required |
219219
{% if beta_cluster %}
220220
| placement_policy | Placement type to set for nodes in a node pool. Can be set as [COMPACT](https://cloud.google.com/kubernetes-engine/docs/how-to/compact-placement#overview) if desired | Optional |
221-
| pod_range | The ID of the secondary range for pod IPs. | | Optional |
221+
| pod_range | The name of the secondary range for pod IPs. | | Optional |
222222
{% endif %}
223223
| node_count | The number of nodes in the nodepool when autoscaling is false. Otherwise defaults to 1. Only valid for non-autoscaling clusters | | Required |
224224
| node_locations | The list of zones in which the cluster's nodes are located. Nodes must be in the region of their regional cluster or in the same region as their cluster's zone for zonal clusters. Defaults to cluster level node locations if nothing is specified | " " | Optional |

‎autogen/main/cluster.tf.tmpl

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -682,6 +682,10 @@ resource "google_container_node_pool" "windows_pools" {
682682
local.node_pools_labels["all"],
683683
local.node_pools_labels[each.value["name"]],
684684
)
685+
resource_labels = merge(
686+
local.node_pools_resource_labels["all"],
687+
local.node_pools_resource_labels[each.value["name"]],
688+
)
685689
metadata = merge(
686690
lookup(lookup(local.node_pools_metadata, "default_values", {}), "cluster_name", true) ? { "cluster_name" = var.name } : {},
687691
lookup(lookup(local.node_pools_metadata, "default_values", {}), "node_pool", true) ? { "node_pool" = each.value["name"] } : {},

‎autogen/main/firewall.tf.tmpl

Lines changed: 73 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -34,11 +34,12 @@ resource "google_compute_firewall" "intra_egress" {
3434
direction = "EGRESS"
3535

3636
target_tags = [local.cluster_network_tag]
37-
destination_ranges = [
37+
destination_ranges = concat([
3838
local.cluster_endpoint_for_nodes,
3939
local.cluster_subnet_cidr,
40-
local.cluster_alias_ranges_cidr[var.ip_range_pods],
41-
]
40+
],
41+
local.pod_all_ip_ranges
42+
)
4243

4344
# Allow all possible protocols
4445
allow { protocol = "tcp" }
@@ -143,7 +144,7 @@ resource "google_compute_firewall" "shadow_allow_pods" {
143144
priority = var.shadow_firewall_rules_priority
144145
direction = "INGRESS"
145146

146-
source_ranges = [local.cluster_alias_ranges_cidr[var.ip_range_pods]]
147+
source_ranges = local.pod_all_ip_ranges
147148
target_tags = [local.cluster_network_tag]
148149

149150
# Allow all possible protocols
@@ -154,8 +155,11 @@ resource "google_compute_firewall" "shadow_allow_pods" {
154155
allow { protocol = "esp" }
155156
allow { protocol = "ah" }
156157

157-
log_config {
158-
metadata = "INCLUDE_ALL_METADATA"
158+
dynamic "log_config" {
159+
for_each = var.shadow_firewall_rules_log_config == null ? [] : [var.shadow_firewall_rules_log_config]
160+
content {
161+
metadata = log_config.value.metadata
162+
}
159163
}
160164
}
161165

@@ -177,8 +181,11 @@ resource "google_compute_firewall" "shadow_allow_master" {
177181
ports = ["10250", "443"]
178182
}
179183

180-
log_config {
181-
metadata = "INCLUDE_ALL_METADATA"
184+
dynamic "log_config" {
185+
for_each = var.shadow_firewall_rules_log_config == null ? [] : [var.shadow_firewall_rules_log_config]
186+
content {
187+
metadata = log_config.value.metadata
188+
}
182189
}
183190
}
184191

@@ -209,7 +216,63 @@ resource "google_compute_firewall" "shadow_allow_nodes" {
209216
ports = ["1-65535"]
210217
}
211218

212-
log_config {
213-
metadata = "INCLUDE_ALL_METADATA"
219+
dynamic "log_config" {
220+
for_each = var.shadow_firewall_rules_log_config == null ? [] : [var.shadow_firewall_rules_log_config]
221+
content {
222+
metadata = log_config.value.metadata
223+
}
224+
}
225+
}
226+
227+
resource "google_compute_firewall" "shadow_allow_inkubelet" {
228+
count = var.add_shadow_firewall_rules ? 1 : 0
229+
230+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-inkubelet"
231+
description = "Managed by terraform GKE module: A shadow firewall rule to match the default rule allowing worker nodes & pods communication to kubelet."
232+
project = local.network_project_id
233+
network = var.network
234+
priority = var.shadow_firewall_rules_priority - 1 # rule created by GKE robot have prio 999
235+
direction = "INGRESS"
236+
237+
source_ranges = local.pod_all_ip_ranges
238+
source_tags = [local.cluster_network_tag]
239+
target_tags = [local.cluster_network_tag]
240+
241+
allow {
242+
protocol = "tcp"
243+
ports = ["10255"]
244+
}
245+
246+
dynamic "log_config" {
247+
for_each = var.shadow_firewall_rules_log_config == null ? [] : [var.shadow_firewall_rules_log_config]
248+
content {
249+
metadata = log_config.value.metadata
250+
}
251+
}
252+
}
253+
254+
resource "google_compute_firewall" "shadow_deny_exkubelet" {
255+
count = var.add_shadow_firewall_rules ? 1 : 0
256+
257+
name = "gke-shadow-${substr(var.name, 0, min(25, length(var.name)))}-exkubelet"
258+
description = "Managed by terraform GKE module: A shadow firewall rule to match the default deny rule to kubelet."
259+
project = local.network_project_id
260+
network = var.network
261+
priority = var.shadow_firewall_rules_priority # rule created by GKE robot have prio 1000
262+
direction = "INGRESS"
263+
264+
source_ranges = ["0.0.0.0/0"]
265+
target_tags = [local.cluster_network_tag]
266+
267+
deny {
268+
protocol = "tcp"
269+
ports = ["10255"]
270+
}
271+
272+
dynamic "log_config" {
273+
for_each = var.shadow_firewall_rules_log_config == null ? [] : [var.shadow_firewall_rules_log_config]
274+
content {
275+
metadata = log_config.value.metadata
276+
}
214277
}
215278
}

‎autogen/main/main.tf.tmpl

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,11 @@ locals {
8686

8787
cluster_subnet_cidr = var.add_cluster_firewall_rules ? data.google_compute_subnetwork.gke_subnetwork[0].ip_cidr_range : null
8888
cluster_alias_ranges_cidr = var.add_cluster_firewall_rules ? { for range in toset(data.google_compute_subnetwork.gke_subnetwork[0].secondary_ip_range) : range.range_name => range.ip_cidr_range } : {}
89+
{% if autopilot_cluster != true %}
90+
pod_all_ip_ranges = var.add_cluster_firewall_rules ? compact(concat([local.cluster_alias_ranges_cidr[var.ip_range_pods]], [for k, v in merge(local.node_pools, local.windows_node_pools): local.cluster_alias_ranges_cidr[v.pod_range] if length(lookup(v, "pod_range", "")) > 0] )) : []
91+
{% else %}
92+
pod_all_ip_ranges = var.add_cluster_firewall_rules ? [local.cluster_alias_ranges_cidr[var.ip_range_pods]] : []
93+
{% endif %}
8994

9095
{% if autopilot_cluster != true %}
9196
cluster_network_policy = var.network_policy ? [{

‎autogen/main/variables.tf.tmpl

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,16 @@ variable "node_pools_labels" {
171171
}
172172
}
173173

174+
variable "node_pools_resource_labels" {
175+
type = map(map(string))
176+
description = "Map of maps containing resource labels by node-pool name"
177+
178+
default = {
179+
all = {}
180+
default-node-pool = {}
181+
}
182+
}
183+
174184
variable "node_pools_metadata" {
175185
type = map(map(string))
176186
description = "Map of maps containing node metadata by node-pool name"
@@ -473,9 +483,23 @@ variable "add_shadow_firewall_rules" {
473483
}
474484

475485
variable "shadow_firewall_rules_priority" {
476-
type = number
486+
type = number
477487
description = "The firewall priority of GKE shadow firewall rules. The priority should be less than default firewall, which is 1000."
478-
default = 999
488+
default = 999
489+
validation {
490+
condition = var.shadow_firewall_rules_priority < 1000
491+
error_message = "The shadow firewall rule priority must be lower than auto-created one(1000)."
492+
}
493+
}
494+
495+
variable "shadow_firewall_rules_log_config" {
496+
type = object({
497+
metadata = string
498+
})
499+
description = "The log_config for shadow firewall rules. You can set this variable to `null` to disable logging."
500+
default = {
501+
metadata = "INCLUDE_ALL_METADATA"
502+
}
479503
}
480504

481505
{% if beta_cluster %}

‎autogen/main/variables_defaults.tf.tmpl

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,20 @@ locals {
3535
var.node_pools_labels
3636
)
3737

38+
node_pools_resource_labels = merge(
39+
{ all = {} },
40+
{ default-node-pool = {} },
41+
zipmap(
42+
[for node_pool in var.node_pools : node_pool["name"]],
43+
[for node_pool in var.node_pools : {}]
44+
),
45+
zipmap(
46+
[for node_pool in var.windows_node_pools : node_pool["name"]],
47+
[for node_pool in var.windows_node_pools : {}]
48+
),
49+
var.node_pools_resource_labels
50+
)
51+
3852
node_pools_metadata = merge(
3953
{ all = {} },
4054
{ default-node-pool = {} },

‎autogen/main/versions.tf.tmpl

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,11 @@ terraform {
3838
required_providers {
3939
google = {
4040
source = "hashicorp/google"
41+
<<<<<<< HEAD
4142
version = ">= 4.46.0, < 5.0"
43+
=======
44+
version = ">= 4.45.0, < 5.0"
45+
>>>>>>> master
4246
}
4347
kubernetes = {
4448
source = "hashicorp/kubernetes"

‎autogen/safer-cluster/main.tf.tmpl

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -86,12 +86,13 @@ module "gke" {
8686
// If removing the default node pool, initial_node_count should be at least 1.
8787
initial_node_count = (var.initial_node_count == 0) ? 1 : var.initial_node_count
8888

89-
node_pools = var.node_pools
90-
windows_node_pools = var.windows_node_pools
91-
node_pools_labels = var.node_pools_labels
92-
node_pools_metadata = var.node_pools_metadata
93-
node_pools_taints = var.node_pools_taints
94-
node_pools_tags = var.node_pools_tags
89+
node_pools = var.node_pools
90+
windows_node_pools = var.windows_node_pools
91+
node_pools_labels = var.node_pools_labels
92+
node_pools_resource_labels = var.node_pools_resource_labels
93+
node_pools_metadata = var.node_pools_metadata
94+
node_pools_taints = var.node_pools_taints
95+
node_pools_tags = var.node_pools_tags
9596

9697
node_pools_oauth_scopes = var.node_pools_oauth_scopes
9798

‎autogen/safer-cluster/variables.tf.tmpl

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,16 @@ variable "node_pools_labels" {
168168
}
169169
}
170170

171+
variable "node_pools_resource_labels" {
172+
type = map(map(string))
173+
description = "Map of maps containing resource labels by node-pool name"
174+
175+
default = {
176+
all = {}
177+
default-node-pool = {}
178+
}
179+
}
180+
171181
variable "node_pools_metadata" {
172182
type = map(map(string))
173183
description = "Map of maps containing node metadata by node-pool name"

‎cluster.tf

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -407,6 +407,10 @@ resource "google_container_node_pool" "pools" {
407407
local.node_pools_labels["all"],
408408
local.node_pools_labels[each.value["name"]],
409409
)
410+
resource_labels = merge(
411+
local.node_pools_resource_labels["all"],
412+
local.node_pools_resource_labels[each.value["name"]],
413+
)
410414
metadata = merge(
411415
lookup(lookup(local.node_pools_metadata, "default_values", {}), "cluster_name", true) ? { "cluster_name" = var.name } : {},
412416
lookup(lookup(local.node_pools_metadata, "default_values", {}), "node_pool", true) ? { "node_pool" = each.value["name"] } : {},
@@ -561,6 +565,10 @@ resource "google_container_node_pool" "windows_pools" {
561565
local.node_pools_labels["all"],
562566
local.node_pools_labels[each.value["name"]],
563567
)
568+
resource_labels = merge(
569+
local.node_pools_resource_labels["all"],
570+
local.node_pools_resource_labels[each.value["name"]],
571+
)
564572
metadata = merge(
565573
lookup(lookup(local.node_pools_metadata, "default_values", {}), "cluster_name", true) ? { "cluster_name" = var.name } : {},
566574
lookup(lookup(local.node_pools_metadata, "default_values", {}), "node_pool", true) ? { "node_pool" = each.value["name"] } : {},

‎docs/private_clusters.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,16 @@ If you are using these features with a private cluster, you will need to either:
2020

2121
If you are going to isolate your GKE private clusters from internet access you could check [this guide](https://medium.com/google-cloud/completely-private-gke-clusters-with-no-internet-connectivity-945fffae1ccd) and the associated [repo](https://github.com/andreyk-code/no-inet-gke-cluster).
2222

23+
## Discontiguous multi-Pod CIDR
24+
If you are going to use [discontiguous multi-Pod CIDR](https://cloud.google.com/kubernetes-engine/docs/how-to/multi-pod-cidr) it can happen that GKE robot will not update `gke-[cluster-name]-[cluster-hash]-all` and other firewall rules automatically when you add a new node pool (as stated in [documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/multi-pod-cidr#modified_firewall_rule)). You can prevent this from happening, by using a workaround with shadow firewall rules:
25+
```
26+
module "gke" {
27+
...
28+
add_shadow_firewall_rules = true
29+
shadow_firewall_rules_log_config = null # to save some $ on logs
30+
}
31+
```
32+
2333
## Troubleshooting
2434

2535
### Master Authorized Network

0 commit comments

Comments
 (0)