|
1 | | -# terraform-google-kubernetes-engine |
2 | | -A Cloud Foundation Toolkit Module: Opinionated Google Cloud Platform project creation and configuration with Shared VPC, IAM, APIs, etc. |
| 1 | +# Terraform Kubernetes Engine Module |
| 2 | + |
| 3 | +This module handles opinionated Google Cloud Platform Kubernetes Engine cluster creation and configuration with Node Pools, IP MASQ, Network Policy, etc. |
| 4 | + |
| 5 | +## Requirements |
| 6 | +### Kubectl |
| 7 | +- [kubectl](https://github.com/kubernetes/kubernetes/releases) 1.9.x |
| 8 | +### Terraform plugins |
| 9 | +- [Terraform](https://www.terraform.io/downloads.html) 0.10.x |
| 10 | +- [terraform-provider-google](https://github.com/terraform-providers/terraform-provider-google) plugin v1.8.0 |
| 11 | + |
| 12 | +### Configure a Service Account |
| 13 | +In order to execute this module you must have a Service Account with the following: |
| 14 | + |
| 15 | +#### Roles |
| 16 | +The service account with the following roles: |
| 17 | +- roles/compute.viewer on the project |
| 18 | +- roles/container.clusterAdmin on the project |
| 19 | + |
| 20 | +### Enable API's |
| 21 | +In order to operate with the Service Account you must activate the following APIs on the project where the Service Account was created: |
| 22 | + |
| 23 | +- Compute Engine API - compute.googleapis.com |
| 24 | +- Kubernetes Engine API - container.googleapis.com |
| 25 | + |
| 26 | +## Install |
| 27 | + |
| 28 | +### Terraform |
| 29 | +Be sure you have the correct Terraform version (0.10.x), you can choose the binary here: |
| 30 | +- https://releases.hashicorp.com/terraform/ |
| 31 | + |
| 32 | +## Usage |
| 33 | +There are multiple examples included in the [examples](./examples/) folder but simple usage is as follows: |
| 34 | + |
| 35 | +```hcl |
| 36 | +module "gke" { |
| 37 | + source = "github.com/terraform-google-modules/terraform-google-kubernetes-engine" |
| 38 | + project_id = "<PROJECT ID>" |
| 39 | + name = "gke-test-1" |
| 40 | + region = "us-central1" |
| 41 | + zones = ["us-central1-a", "us-central1-b", "us-central1-f"] |
| 42 | + network = "vpc-01" |
| 43 | + subnetwork = "us-central1-01" |
| 44 | + ip_range_pods = "us-central1-01-gke-01-pods" |
| 45 | + ip_range_services = "us-central1-01-gke-01-services" |
| 46 | + http_load_balancing = false |
| 47 | + horizontal_pod_autoscaling = true |
| 48 | + kubernetes_dashboard = true |
| 49 | + network_policy = true |
| 50 | +
|
| 51 | + node_pools = [ |
| 52 | + { |
| 53 | + name = "default-node-pool" |
| 54 | + machine_type = "n1-standard-2" |
| 55 | + min_count = 1 |
| 56 | + max_count = 100 |
| 57 | + disk_size_gb = 100 |
| 58 | + disk_type = "pd-standard" |
| 59 | + image_type = "COS" |
| 60 | + auto_repair = true |
| 61 | + auto_upgrade = true |
| 62 | + service_account = "project-service-account@<PROJECT ID>.iam.gserviceaccount.com" |
| 63 | + }, |
| 64 | + ] |
| 65 | +
|
| 66 | + node_pools_labels = { |
| 67 | + all = {} |
| 68 | +
|
| 69 | + default-node-pool = { |
| 70 | + default-node-pool = "true" |
| 71 | + } |
| 72 | + } |
| 73 | +
|
| 74 | + node_pools_taints = { |
| 75 | + all = [] |
| 76 | +
|
| 77 | + default-node-pool = [ |
| 78 | + { |
| 79 | + key = "default-node-pool" |
| 80 | + value = "true" |
| 81 | + effect = "PREFER_NO_SCHEDULE" |
| 82 | + }, |
| 83 | + ] |
| 84 | + } |
| 85 | +
|
| 86 | + node_pools_tags = { |
| 87 | + all = [] |
| 88 | +
|
| 89 | + default-node-pool = [ |
| 90 | + "default-node-pool", |
| 91 | + ] |
| 92 | + } |
| 93 | +} |
| 94 | +``` |
| 95 | + |
| 96 | +Then perform the following commands on the root folder: |
| 97 | + |
| 98 | +- `terraform init` to get the plugins |
| 99 | +- `terraform plan` to see the infrastructure plan |
| 100 | +- `terraform apply` to apply the infrastructure build |
| 101 | +- `terraform destroy` to destroy the built infrastructure |
| 102 | +[^]: (autogen_docs_start) |
| 103 | + |
| 104 | + |
| 105 | +## Inputs |
| 106 | + |
| 107 | +| Name | Description | Type | Default | Required | |
| 108 | +|------|-------------|:----:|:-----:|:-----:| |
| 109 | +| description | The description of the cluster | string | `` | no | |
| 110 | +| horizontal_pod_autoscaling | Enable horizontal pod autoscaling addon | string | `false` | no | |
| 111 | +| http_load_balancing | Enable httpload balancer addon | string | `true` | no | |
| 112 | +| ip_masq_link_local | Whether to masquerade traffic to the link-local prefix (169.254.0.0/16). | string | `false` | no | |
| 113 | +| ip_masq_resync_interval | The interval at which the agent attempts to sync its ConfigMap file from the disk. | string | `60s` | no | |
| 114 | +| ip_range_pods | The secondary ip range to use for pods | string | - | yes | |
| 115 | +| ip_range_services | The secondary ip range to use for pods | string | - | yes | |
| 116 | +| kubernetes_dashboard | Enable kubernetes dashboard addon | string | `false` | no | |
| 117 | +| kubernetes_version | The Kubernetes version of the masters. If set to 'latest' it will pull latest available version in the selected region. | string | `1.10.6-gke.2` | no | |
| 118 | +| maintenance_start_time | Time window specified for daily maintenance operations in RFC3339 format | string | `05:00` | no | |
| 119 | +| name | The name of the cluster (required) | string | - | yes | |
| 120 | +| network | The VPC network to host the cluster in (required) | string | - | yes | |
| 121 | +| network_policy | Enable network policy addon | string | `false` | no | |
| 122 | +| network_project_id | The project ID of the shared VPC's host (for shared vpc support) | string | `` | no | |
| 123 | +| node_pools | List of maps containing node pools | list | `<list>` | no | |
| 124 | +| node_pools_labels | Map of maps containing node labels by node-pool name | map | `<map>` | no | |
| 125 | +| node_pools_tags | Map of lists containing node network tags by node-pool name | map | `<map>` | no | |
| 126 | +| node_pools_taints | Map of lists containing node taints by node-pool name | map | `<map>` | no | |
| 127 | +| node_version | The Kubernetes version of the node pools. Defaults kubernetes_version (master) variable and can be overridden for individual node pools by setting the `version` key on them. Must be empyty or set the same as master at cluster creation. | string | `` | no | |
| 128 | +| non_masquerade_cidrs | List of strings in CIDR notation that specify the IP address ranges that do not use IP masquerading. | list | `<list>` | no | |
| 129 | +| project_id | The project ID to host the cluster in (required) | string | - | yes | |
| 130 | +| region | The region to host the cluster in (required) | string | - | yes | |
| 131 | +| regional | Whether is a regional cluster (zonal cluster if set false. WARNING: changing this after cluster creation is destructive!) | string | `true` | no | |
| 132 | +| stub_domains | Map of stub domains and their resolvers to forward DNS queries for a certain domain to an external DNS server | map | `<map>` | no | |
| 133 | +| subnetwork | The subnetwork to host the cluster in (required) | string | - | yes | |
| 134 | +| zones | The zones to host the cluster in (optional if regional cluster / required if zonal) | list | `<list>` | no | |
| 135 | + |
| 136 | +## Outputs |
| 137 | + |
| 138 | +| Name | Description | |
| 139 | +|------|-------------| |
| 140 | +| ca_certificate | Cluster ca certificate (base64 encoded) | |
| 141 | +| endpoint | Cluster endpoint | |
| 142 | +| horizontal_pod_autoscaling_enabled | Whether horizontal pod autoscaling enabled | |
| 143 | +| http_load_balancing_enabled | Whether http load balancing enabled | |
| 144 | +| kubernetes_dashboard_enabled | Whether kubernetes dashboard enabled | |
| 145 | +| location | Cluster location (region if regional cluster, zone if zonal cluster) | |
| 146 | +| master_version | Current master kubernetes version | |
| 147 | +| min_master_version | Minimum master kubernetes version | |
| 148 | +| name | Cluster name | |
| 149 | +| network_policy_enabled | Whether network policy enabled | |
| 150 | +| node_pools_names | List of node pools names | |
| 151 | +| node_pools_versions | List of node pools versions | |
| 152 | +| region | Cluster region | |
| 153 | +| type | Cluster type (regional / zonal) | |
| 154 | +| zones | List of zones in which the cluster resides | |
| 155 | + |
| 156 | +[^]: (autogen_docs_end) |
| 157 | + |
| 158 | +## Infrastructure |
| 159 | +The resources/services/activations/deletions that this module will create/trigger are: |
| 160 | +- Create a GKE cluster with the provided addons |
| 161 | +- Create GKE Node Pool(s) with provided configuration and attach to cluster |
| 162 | +- Replace the default kube-dns configmap if `stub_domains` are provided |
| 163 | +- Activate network policy if `network_policy` is true |
| 164 | +- Add `ip-masq-agent` configmap with provided `non_masquerade_cidrs` if `network_policy` is true |
| 165 | + |
| 166 | +## File structure |
| 167 | +The project has the following folders and files: |
| 168 | + |
| 169 | +- /: root folder |
| 170 | +- /examples: examples for using this module |
| 171 | +- /scripts: Scripts for specific tasks on module (see Infrastructure section on this file) |
| 172 | +- /test: Folders with files for testing the module (see Testing section on this file) |
| 173 | +- /main.tf: main file for this module, contains all the resources to create |
| 174 | +- /variables.tf: all the variables for the module |
| 175 | +- /output.tf: the outputs of the module |
| 176 | +- /readme.MD: this file |
| 177 | + |
| 178 | +## Testing |
| 179 | + |
| 180 | +### Requirements |
| 181 | +- [bundler](https://github.com/bundler/bundler) |
| 182 | +- [gcloud](https://cloud.google.com/sdk/install) |
| 183 | +- [jq](https://stedolan.github.io/jq/) 1.5 |
| 184 | +- [terraform-docs](https://github.com/segmentio/terraform-docs/releases) 0.3.0 |
| 185 | + |
| 186 | +### Autogeneration of documentation from .tf files |
| 187 | +Run |
| 188 | +``` |
| 189 | +make generate_docs |
| 190 | +``` |
| 191 | + |
| 192 | +### Integration test |
| 193 | +#### Terraform integration tests |
| 194 | +The integration tests for this module leverage [kitchen-terraform](https://github.com/newcontext-oss/kitchen-terraform) and [kitchen-inspec](https://github.com/inspec/kitchen-inspec). |
| 195 | + |
| 196 | +The tests will do the following: |
| 197 | +- Perform `bundle install` command |
| 198 | + - Installs `kitchen-terraform` and `kitchen-inspec` gems |
| 199 | +- Perform `kitchen create` command |
| 200 | + - Performs a `terraform init` |
| 201 | +- Perform `kitchen converge` command |
| 202 | + - Performs a `terraform apply -auto-approve` |
| 203 | +- Perform `kitchen validate` command |
| 204 | + - Performs inspec tests. |
| 205 | + - Shell out to `gcloud` to validate expected resources in GCP. |
| 206 | + - Shell out to `kubectl` to validate expected resource in Kubernetes. |
| 207 | + - Shell out to `terraform` to validate outputs. |
| 208 | +- Permos `kitchen destroy` command |
| 209 | + - Performs a `terraform destroy -force` |
| 210 | + |
| 211 | +You can use the following command to run the integration test in the root folder |
| 212 | + |
| 213 | + `make test_integration` |
| 214 | + |
| 215 | +### Linting |
| 216 | +The makefile in this project will lint or sometimes just format any shell, |
| 217 | +Python, golang, Terraform, or Dockerfiles. The linters will only be run if |
| 218 | +the makefile finds files with the appropriate file extension. |
| 219 | + |
| 220 | +All of the linter checks are in the default make target, so you just have to |
| 221 | +run |
| 222 | + |
| 223 | +``` |
| 224 | +make -s |
| 225 | +``` |
| 226 | + |
| 227 | +The -s is for 'silent'. Successful output looks like this |
| 228 | + |
| 229 | +``` |
| 230 | +Running shellcheck |
| 231 | +Running flake8 |
| 232 | +Running go fmt and go vet |
| 233 | +Running terraform validate |
| 234 | +Running hadolint on Dockerfiles |
| 235 | +Checking for required files |
| 236 | +Testing the validity of the header check |
| 237 | +.. |
| 238 | +---------------------------------------------------------------------- |
| 239 | +Ran 2 tests in 0.026s |
| 240 | +
|
| 241 | +OK |
| 242 | +Checking file headers |
| 243 | +The following lines have trailing whitespace |
| 244 | +``` |
| 245 | + |
| 246 | +The linters |
| 247 | +are as follows: |
| 248 | +* Shell - shellcheck. Can be found in homebrew |
| 249 | +* Python - flake8. Can be installed with 'pip install flake8' |
| 250 | +* Golang - gofmt. gofmt comes with the standard golang installation. golang |
| 251 | +is a compiled language so there is no standard linter. |
| 252 | +* Terraform - terraform has a built-in linter in the 'terraform validate' |
| 253 | +command. |
| 254 | +* Dockerfiles - hadolint. Can be found in homebrew |
0 commit comments