Skip to content

Commit 27ce085

Browse files
nvnmandadhinvnmandadhi
committed
feat: add island-cluster example
1 parent 19e9c0d commit 27ce085

File tree

7 files changed

+440
-0
lines changed

7 files changed

+440
-0
lines changed

‎examples/island_cluster_with_vm_router/README.md

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# GKE island cluster using VM as router
2+
3+
This example provisions a cluster in an island VPC allowing reuse of the IP address space for multiple clusters in the same project.
4+
5+
1. An appliance(VM as router) with multiple NICs is used to establish connectivity between the island VPC and the existing network.
6+
2. Outbound connections will go through the router.
7+
3. For inbound connections, use Private Service Connect.
8+
9+
## Deploy
10+
11+
1. Use the following example to deploy a cluster using this module.
12+
2. Update `project_id` and `primary_subnet` values in `terraform.tfvars`, and update other variables as needed.
13+
3. Value for `random_suffix` variable needs to be provided for a successful terraform plan.
14+
15+
```
16+
module "gke-island-cluster" {
17+
source = "./"
18+
19+
project_id = var.project_id
20+
region = var.region
21+
random_suffix = "<random_suffix>>"
22+
node_locations = var.node_locations
23+
subnet_cidr = var.subnet_cidr
24+
secondary_ranges = var.secondary_ranges
25+
master_authorized_networks = var.master_authorized_networks
26+
router_machine_type = var.router_machine_type
27+
primary_subnet = var.primary_subnet
28+
primary_net_cidrs = var.primary_net_cidrs
29+
psc_subnet_cidr = var.psc_subnet_cidr
30+
proxy_subnet_cidr = var.proxy_subnet_cidr
31+
}
32+
```
33+
34+
## Requirements
35+
36+
| Name | Version |
37+
|------|---------|
38+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | ~> 1.6 |
39+
40+
## Providers
41+
42+
| Name | Version |
43+
|------|---------|
44+
| <a name="provider_google"></a> [google](#provider\_google) | n/a |
45+
| <a name="provider_google-beta"></a> [google-beta](#provider\_google-beta) | n/a |
46+
47+
## Modules
48+
49+
| Name | Source | Version |
50+
|------|--------|---------|
51+
| <a name="module_gke"></a> [gke](#module\_gke) | terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster | ~> 30.0 |
52+
| <a name="module_net"></a> [net](#module\_net) | terraform-google-modules/network/google | ~> 9.0 |
53+
54+
## Resources
55+
56+
| Name | Type |
57+
|------|------|
58+
| [google-beta_google_gke_hub_membership.primary](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/google_gke_hub_membership) | resource |
59+
| [google_compute_firewall.proxy_snet](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource |
60+
| [google_compute_firewall.psc_snet](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource |
61+
| [google_compute_instance.vm](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance) | resource |
62+
| [google_project_iam_member.gke-dev](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_member) | resource |
63+
| [google_service_account.gke-access-sa](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |
64+
| [google_service_account.gke-sa](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |
65+
66+
## Inputs
67+
68+
| Name | Description | Type | Default | Required |
69+
|------|-------------|------|---------|:--------:|
70+
| <a name="input_master_authorized_networks"></a> [master\_authorized\_networks](#input\_master\_authorized\_networks) | List of master authorized networks. If none are provided, disallow external access (except the cluster node IPs, which GKE automatically whitelists). | `list(object({ cidr_block = string, display_name = string }))` | n/a | yes |
71+
| <a name="input_node_locations"></a> [node\_locations](#input\_node\_locations) | n/a | `list(string)` | n/a | yes |
72+
| <a name="input_primary_net_cidrs"></a> [primary\_net\_cidrs](#input\_primary\_net\_cidrs) | n/a | `list(string)` | n/a | yes |
73+
| <a name="input_primary_subnet"></a> [primary\_subnet](#input\_primary\_subnet) | n/a | `string` | n/a | yes |
74+
| <a name="input_project_id"></a> [project\_id](#input\_project\_id) | n/a | `string` | n/a | yes |
75+
| <a name="input_proxy_subnet_cidr"></a> [proxy\_subnet\_cidr](#input\_proxy\_subnet\_cidr) | n/a | `string` | n/a | yes |
76+
| <a name="input_psc_subnet_cidr"></a> [psc\_subnet\_cidr](#input\_psc\_subnet\_cidr) | n/a | `string` | n/a | yes |
77+
| <a name="input_random_suffix"></a> [random\_suffix](#input\_random\_suffix) | n/a | `string` | n/a | yes |
78+
| <a name="input_region"></a> [region](#input\_region) | n/a | `string` | n/a | yes |
79+
| <a name="input_router_machine_type"></a> [router\_machine\_type](#input\_router\_machine\_type) | n/a | `string` | n/a | yes |
80+
| <a name="input_secondary_ranges"></a> [secondary\_ranges](#input\_secondary\_ranges) | n/a | `map(string)` | n/a | yes |
81+
| <a name="input_subnet_cidr"></a> [subnet\_cidr](#input\_subnet\_cidr) | n/a | `string` | n/a | yes |
82+
83+
## Outputs
84+
85+
| Name | Description |
86+
|------|-------------|
87+
| <a name="output_cluster_id"></a> [cluster\_id](#output\_cluster\_id) | n/a |

‎examples/island_cluster_with_vm_router/main.tf

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
resource "google_service_account" "gke-sa" {
2+
account_id = "gke-sa-${var.random_suffix}"
3+
}
4+
5+
resource "google_service_account" "gke-access-sa" {
6+
account_id = "gke-access-sa-${var.random_suffix}"
7+
}
8+
9+
resource "google_project_iam_member" "gke-dev" {
10+
project = var.project_id
11+
member = "serviceAccount:${google_service_account.gke-access-sa.email}"
12+
role = "roles/container.developer"
13+
}
14+
15+
module "net" {
16+
source = "terraform-google-modules/network/google"
17+
version = "~> 9.0"
18+
19+
network_name = "gke-net-${var.random_suffix}"
20+
routing_mode = "GLOBAL"
21+
project_id = var.project_id
22+
delete_default_internet_gateway_routes = true
23+
24+
subnets = [
25+
{
26+
subnet_name = "${var.region}-snet-${var.random_suffix}"
27+
subnet_ip = var.subnet_cidr
28+
subnet_region = var.region
29+
subnet_private_access = "true"
30+
},
31+
{
32+
subnet_name = "${var.region}-proxy-snet-${var.random_suffix}"
33+
subnet_ip = var.proxy_subnet_cidr
34+
subnet_region = var.region
35+
purpose = "REGIONAL_MANAGED_PROXY"
36+
role = "ACTIVE"
37+
},
38+
{
39+
subnet_name = "${var.region}-psc-snet-${var.random_suffix}"
40+
subnet_ip = var.psc_subnet_cidr
41+
subnet_region = var.region
42+
subnet_private_access = "true"
43+
purpose = "PRIVATE_SERVICE_CONNECT"
44+
}
45+
]
46+
47+
secondary_ranges = {
48+
"${var.region}-snet-${var.random_suffix}" = [
49+
{
50+
range_name = "${var.region}-snet-pods-${var.random_suffix}"
51+
ip_cidr_range = var.secondary_ranges["pods"]
52+
},
53+
{
54+
range_name = "${var.region}-snet-services-${var.random_suffix}"
55+
ip_cidr_range = var.secondary_ranges["services"]
56+
},
57+
]
58+
}
59+
60+
routes = flatten([
61+
[for k, v in var.primary_net_cidrs :
62+
{
63+
name = "egress-gke-${k}-${var.random_suffix}"
64+
description = "egress through the router for range ${v}"
65+
destination_range = v
66+
tags = "gke-${var.random_suffix}"
67+
next_hop_instance = google_compute_instance.vm.self_link
68+
priority = 100
69+
}
70+
],
71+
[
72+
{
73+
name = "default-igw-${var.random_suffix}"
74+
description = "internet through the router"
75+
destination_range = "0.0.0.0/0"
76+
tags = "gke-${var.random_suffix}"
77+
next_hop_instance = google_compute_instance.vm.self_link
78+
priority = 100
79+
}
80+
]
81+
])
82+
83+
firewall_rules = [
84+
{
85+
name = "iap-fw-${var.random_suffix}"
86+
direction = "INGRESS"
87+
allow = [
88+
{
89+
protocol = "TCP"
90+
ports = ["22"]
91+
}
92+
]
93+
ranges = ["35.235.240.0/20"]
94+
},
95+
{
96+
name = "tcp-primary-fw-${var.random_suffix}"
97+
direction = "INGRESS"
98+
allow = [
99+
{
100+
protocol = "TCP"
101+
}
102+
]
103+
ranges = [
104+
var.subnet_cidr,
105+
var.secondary_ranges["pods"]
106+
]
107+
},
108+
]
109+
}
110+
111+
module "gke" {
112+
source = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster"
113+
version = "~> 30.0"
114+
115+
depends_on = [google_compute_instance.vm]
116+
117+
name = "gke-test-${var.random_suffix}"
118+
project_id = var.project_id
119+
region = var.region
120+
release_channel = "RAPID"
121+
zones = var.node_locations
122+
network = module.net.network_name
123+
subnetwork = "${var.region}-snet-${var.random_suffix}"
124+
ip_range_pods = "${var.region}-snet-pods-${var.random_suffix}"
125+
ip_range_services = "${var.region}-snet-services-${var.random_suffix}"
126+
http_load_balancing = true
127+
horizontal_pod_autoscaling = true
128+
enable_private_endpoint = true
129+
enable_private_nodes = true
130+
datapath_provider = "ADVANCED_DATAPATH"
131+
monitoring_enable_managed_prometheus = false
132+
enable_shielded_nodes = true
133+
master_global_access_enabled = false
134+
master_ipv4_cidr_block = var.secondary_ranges["master_cidr"]
135+
master_authorized_networks = var.master_authorized_networks
136+
deletion_protection = false
137+
remove_default_node_pool = true
138+
disable_default_snat = true
139+
gateway_api_channel = "CHANNEL_STANDARD"
140+
141+
node_pools = [
142+
{
143+
name = "default-${var.random_suffix}"
144+
machine_type = "e2-highcpu-2"
145+
min_count = 1
146+
max_count = 100
147+
local_ssd_count = 0
148+
spot = true
149+
local_ssd_ephemeral_count = 0
150+
disk_size_gb = 100
151+
disk_type = "pd-standard"
152+
image_type = "COS_CONTAINERD"
153+
logging_variant = "DEFAULT"
154+
auto_repair = true
155+
auto_upgrade = true
156+
service_account = google_service_account.gke-sa.email
157+
initial_node_count = 1
158+
enable_secure_boot = true
159+
},
160+
]
161+
162+
node_pools_tags = {
163+
all = ["gke-${var.random_suffix}"]
164+
}
165+
166+
node_pools_oauth_scopes = {
167+
all = [
168+
"https://www.googleapis.com/auth/logging.write",
169+
"https://www.googleapis.com/auth/monitoring",
170+
]
171+
}
172+
173+
timeouts = {
174+
create = "10m"
175+
update = "10m"
176+
delete = "10m"
177+
}
178+
}
179+
180+
resource "google_gke_hub_membership" "primary" {
181+
provider = google-beta
182+
183+
project = var.project_id
184+
membership_id = "${var.project_id}-${module.gke.name}"
185+
location = var.region
186+
187+
endpoint {
188+
gke_cluster {
189+
resource_link = "//container.googleapis.com/${module.gke.cluster_id}"
190+
}
191+
}
192+
authority {
193+
issuer = "https://container.googleapis.com/v1/${module.gke.cluster_id}"
194+
}
195+
}
196+
197+
resource "google_compute_firewall" "psc_snet" {
198+
name = "allow-psc-${var.random_suffix}"
199+
direction = "INGRESS"
200+
allow {
201+
protocol = "tcp"
202+
}
203+
source_ranges = [var.psc_subnet_cidr]
204+
target_service_accounts = [google_service_account.gke-sa.email]
205+
network = module.net.network_id
206+
project = var.project_id
207+
}
208+
209+
resource "google_compute_firewall" "proxy_snet" {
210+
name = "allow-proxy-${var.random_suffix}"
211+
direction = "INGRESS"
212+
allow {
213+
protocol = "tcp"
214+
}
215+
source_ranges = [var.proxy_subnet_cidr]
216+
target_service_accounts = [google_service_account.gke-sa.email]
217+
network = module.net.network_id
218+
project = var.project_id
219+
}

‎examples/island_cluster_with_vm_router/outputs.tf

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
output "cluster_id" {
2+
value = module.gke.cluster_id
3+
}

‎examples/island_cluster_with_vm_router/providers.tf

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
terraform {
2+
required_version = "~> 1.6"
3+
4+
required_providers {
5+
google = {
6+
source = "hashicorp/google"
7+
}
8+
google-beta = {
9+
source = "hashicorp/google-beta"
10+
}
11+
}
12+
}
13+
14+
provider "google" {
15+
project = var.project_id
16+
}
17+
18+
provider "google-beta" {
19+
project = var.project_id
20+
}

‎examples/island_cluster_with_vm_router/router.tf

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
resource "google_compute_instance" "vm" {
2+
project = var.project_id
3+
zone = var.node_locations[0]
4+
name = "router-${var.random_suffix}"
5+
machine_type = var.router_machine_type
6+
allow_stopping_for_update = true
7+
boot_disk {
8+
initialize_params {
9+
image = "debian-cloud/debian-12"
10+
}
11+
}
12+
can_ip_forward = true
13+
shielded_instance_config {
14+
enable_secure_boot = true
15+
}
16+
network_interface {
17+
subnetwork = var.primary_subnet
18+
}
19+
network_interface {
20+
subnetwork = module.net.subnets["${var.region}/${var.region}-snet-${var.random_suffix}"]["self_link"]
21+
}
22+
metadata_startup_script = <<-EOT
23+
#!/bin/bash
24+
set -ex
25+
sudo apt-get update
26+
echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.conf
27+
sudo sysctl -p
28+
sudo iptables -A FORWARD -i ens5 -o ens4 -j ACCEPT
29+
sudo iptables -A FORWARD -i ens4 -o ens5 -m state --state ESTABLISHED,RELATED -j ACCEPT
30+
GWY_URL="http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/1/gateway"
31+
GWY_IP=$(curl $${GWY_URL} -H "Metadata-Flavor: Google")
32+
sudo ip route add ${var.secondary_ranges["pods"]} via $${GWY_IP} dev ens5
33+
sudo iptables -t nat -A POSTROUTING -o ens4 -s 0.0.0.0/0 -j MASQUERADE
34+
EOT
35+
}

0 commit comments

Comments
 (0)