Skip to content

Commit 027fffb

Browse files
sjbermansjberman
committed
CP/DP split: Secure connection (#3244)
Problem: We want to ensure that the connection between the control plane and data plane is authenticated and secure. Solution: 1. Configure agent to send the kubernetes service token in the request. The control plane validates this token using the TokenReview API to ensure the agent is authenticated. 2. Configure TLS certificates for both the control and data planes. By default, a Job will run when installing NGF that creates self-signed certificates in the nginx-gateway namespace. The server Secret is mounted to the control plane, and the control plane copies the client Secret when deploying nginx resources. This Secret is mounted to the agent. The control plane will reset the agent connection if it detects that its own certs have changed. For production environments, we'll recommend a user configures TLS using cert-manager instead, for better security and certificate rotation.
1 parent 4bbd22b commit 027fffb

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

50 files changed

+3082
-120
lines changed

‎charts/nginx-gateway-fabric/README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,12 @@ The following table lists the configurable parameters of the NGINX Gateway Fabri
258258

259259
| Key | Description | Type | Default |
260260
|-----|-------------|------|---------|
261+
| `certGenerator` | The certGenerator section contains the configuration for the cert-generator Job. | object | `{"agentTLSSecretName":"agent-tls","annotations":{},"overwrite":false,"serverTLSSecretName":"server-tls"}` |
262+
| `certGenerator.agentTLSSecretName` | The name of the base Secret containing TLS CA, certificate, and key for the NGINX Agent to securely communicate with the NGINX Gateway Fabric control plane. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). | string | `"agent-tls"` |
263+
| `certGenerator.annotations` | The annotations of the cert-generator Job. | object | `{}` |
264+
| `certGenerator.overwrite` | Overwrite existing TLS Secrets on startup. | bool | `false` |
265+
| `certGenerator.serverTLSSecretName` | The name of the Secret containing TLS CA, certificate, and key for the NGINX Gateway Fabric control plane to securely communicate with the NGINX Agent. Must exist in the same namespace that the NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway). | string | `"server-tls"` |
266+
| `clusterDomain` | The DNS cluster domain of your Kubernetes cluster. | string | `"cluster.local"` |
261267
| `nginx` | The nginx section contains the configuration for all NGINX data plane deployments installed by the NGINX Gateway Fabric control plane. | object | `{"config":{},"container":{},"debug":false,"image":{"pullPolicy":"Always","repository":"ghcr.io/nginx/nginx-gateway-fabric/nginx","tag":"edge"},"imagePullSecret":"","imagePullSecrets":[],"kind":"deployment","plus":false,"pod":{},"replicas":1,"service":{"externalTrafficPolicy":"Local","type":"LoadBalancer"},"usage":{"caSecretName":"","clientSSLSecretName":"","endpoint":"","resolver":"","secretName":"nplus-license","skipVerify":false}}` |
262268
| `nginx.config` | The configuration for the data plane that is contained in the NginxProxy resource. | object | `{}` |
263269
| `nginx.container` | The container configuration for the NGINX container. | object | `{}` |

‎charts/nginx-gateway-fabric/templates/certs-job.yaml

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
apiVersion: v1
2+
kind: ServiceAccount
3+
metadata:
4+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
5+
namespace: {{ .Release.Namespace }}
6+
labels:
7+
{{- include "nginx-gateway.labels" . | nindent 4 }}
8+
annotations:
9+
"helm.sh/hook": pre-install
10+
{{- if or .Values.nginxGateway.serviceAccount.imagePullSecret .Values.nginxGateway.serviceAccount.imagePullSecrets }}
11+
imagePullSecrets:
12+
{{- if .Values.nginxGateway.serviceAccount.imagePullSecret }}
13+
- name: {{ .Values.nginxGateway.serviceAccount.imagePullSecret }}
14+
{{- end }}
15+
{{- if .Values.nginxGateway.serviceAccount.imagePullSecrets }}
16+
{{- range .Values.nginxGateway.serviceAccount.imagePullSecrets }}
17+
- name: {{ . }}
18+
{{- end }}
19+
{{- end }}
20+
{{- end }}
21+
---
22+
apiVersion: rbac.authorization.k8s.io/v1
23+
kind: Role
24+
metadata:
25+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
26+
namespace: {{ .Release.Namespace }}
27+
labels:
28+
{{- include "nginx-gateway.labels" . | nindent 4 }}
29+
annotations:
30+
"helm.sh/hook": pre-install
31+
rules:
32+
- apiGroups:
33+
- ""
34+
resources:
35+
- secrets
36+
verbs:
37+
- create
38+
- update
39+
- get
40+
---
41+
apiVersion: rbac.authorization.k8s.io/v1
42+
kind: RoleBinding
43+
metadata:
44+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
45+
namespace: {{ .Release.Namespace }}
46+
labels:
47+
{{- include "nginx-gateway.labels" . | nindent 4 }}
48+
annotations:
49+
"helm.sh/hook": pre-install
50+
roleRef:
51+
apiGroup: rbac.authorization.k8s.io
52+
kind: Role
53+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
54+
subjects:
55+
- kind: ServiceAccount
56+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
57+
namespace: {{ .Release.Namespace }}
58+
---
59+
apiVersion: batch/v1
60+
kind: Job
61+
metadata:
62+
name: {{ include "nginx-gateway.fullname" . }}-cert-generator
63+
namespace: {{ .Release.Namespace }}
64+
labels:
65+
{{- include "nginx-gateway.labels" . | nindent 4 }}
66+
annotations:
67+
{{- with .Values.certGenerator.annotations -}}
68+
{{ toYaml . | nindent 4 }}
69+
{{- end }}
70+
"helm.sh/hook": pre-install, pre-upgrade
71+
spec:
72+
template:
73+
metadata:
74+
annotations:
75+
{{- with .Values.certGenerator.annotations -}}
76+
{{ toYaml . | nindent 8 }}
77+
{{- end }}
78+
spec:
79+
containers:
80+
- args:
81+
- generate-certs
82+
- --service={{ include "nginx-gateway.fullname" . }}
83+
- --cluster-domain={{ .Values.clusterDomain }}
84+
- --server-tls-secret={{ .Values.certGenerator.serverTLSSecretName }}
85+
- --agent-tls-secret={{ .Values.certGenerator.agentTLSSecretName }}
86+
{{- if .Values.certGenerator.overwrite }}
87+
- --overwrite
88+
{{- end }}
89+
env:
90+
- name: POD_NAMESPACE
91+
valueFrom:
92+
fieldRef:
93+
fieldPath: metadata.namespace
94+
image: {{ .Values.nginxGateway.image.repository }}:{{ default .Chart.AppVersion .Values.nginxGateway.image.tag }}
95+
imagePullPolicy: {{ .Values.nginxGateway.image.pullPolicy }}
96+
name: cert-generator
97+
securityContext:
98+
seccompProfile:
99+
type: RuntimeDefault
100+
capabilities:
101+
drop:
102+
- ALL
103+
allowPrivilegeEscalation: false
104+
readOnlyRootFilesystem: true
105+
runAsUser: 101
106+
runAsGroup: 1001
107+
restartPolicy: Never
108+
serviceAccountName: {{ include "nginx-gateway.fullname" . }}-cert-generator
109+
securityContext:
110+
fsGroup: 1001
111+
runAsNonRoot: true
112+
ttlSecondsAfterFinished: 0

‎charts/nginx-gateway-fabric/templates/clusterrole.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,12 @@ rules:
5959
verbs:
6060
- list
6161
- watch
62+
- apiGroups:
63+
- authentication.k8s.io
64+
resources:
65+
- tokenreviews
66+
verbs:
67+
- create
6268
- apiGroups:
6369
- gateway.networking.k8s.io
6470
resources:

‎charts/nginx-gateway-fabric/templates/deployment.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ spec:
4242
- --gatewayclass={{ .Values.nginxGateway.gatewayClassName }}
4343
- --config={{ include "nginx-gateway.config-name" . }}
4444
- --service={{ include "nginx-gateway.fullname" . }}
45+
- --agent-tls-secret={{ .Values.certGenerator.agentTLSSecretName }}
4546
{{- if .Values.nginx.imagePullSecret }}
4647
- --nginx-docker-secret={{ .Values.nginx.imagePullSecret }}
4748
{{- end }}
@@ -155,6 +156,9 @@ spec:
155156
readOnlyRootFilesystem: true
156157
runAsUser: 101
157158
runAsGroup: 1001
159+
volumeMounts:
160+
- name: nginx-agent-tls
161+
mountPath: /var/run/secrets/ngf
158162
{{- with .Values.nginxGateway.extraVolumeMounts -}}
159163
{{ toYaml . | nindent 8 }}
160164
{{- end }}
@@ -179,6 +183,10 @@ spec:
179183
nodeSelector:
180184
{{- toYaml .Values.nginxGateway.nodeSelector | nindent 8 }}
181185
{{- end }}
186+
volumes:
187+
- name: nginx-agent-tls
188+
secret:
189+
secretName: {{ .Values.certGenerator.serverTLSSecretName }}
182190
{{- with .Values.nginxGateway.extraVolumes -}}
183191
{{ toYaml . | nindent 6 }}
184192
{{- end }}

‎charts/nginx-gateway-fabric/templates/scc.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,4 +34,6 @@ users:
3434
- {{ printf "system:serviceaccount:%s:%s" .Release.Namespace (include "nginx-gateway.serviceAccountName" .) }}
3535
requiredDropCapabilities:
3636
- ALL
37+
volumes:
38+
- secret
3739
{{- end }}

‎charts/nginx-gateway-fabric/values.schema.json

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,48 @@
11
{
22
"$schema": "http://json-schema.org/draft-07/schema#",
33
"properties": {
4+
"certGenerator": {
5+
"description": "The certGenerator section contains the configuration for the cert-generator Job.",
6+
"properties": {
7+
"agentTLSSecretName": {
8+
"default": "agent-tls",
9+
"description": "The name of the base Secret containing TLS CA, certificate, and key for the NGINX Agent to securely\ncommunicate with the NGINX Gateway Fabric control plane. Must exist in the same namespace that the\nNGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway).",
10+
"required": [],
11+
"title": "agentTLSSecretName",
12+
"type": "string"
13+
},
14+
"annotations": {
15+
"description": "The annotations of the cert-generator Job.",
16+
"required": [],
17+
"title": "annotations",
18+
"type": "object"
19+
},
20+
"overwrite": {
21+
"default": false,
22+
"description": "Overwrite existing TLS Secrets on startup.",
23+
"required": [],
24+
"title": "overwrite",
25+
"type": "boolean"
26+
},
27+
"serverTLSSecretName": {
28+
"default": "server-tls",
29+
"description": "The name of the Secret containing TLS CA, certificate, and key for the NGINX Gateway Fabric control plane\nto securely communicate with the NGINX Agent. Must exist in the same namespace that the NGINX Gateway Fabric\ncontrol plane is running in (default namespace: nginx-gateway).",
30+
"required": [],
31+
"title": "serverTLSSecretName",
32+
"type": "string"
33+
}
34+
},
35+
"required": [],
36+
"title": "certGenerator",
37+
"type": "object"
38+
},
39+
"clusterDomain": {
40+
"default": "cluster.local",
41+
"description": "The DNS cluster domain of your Kubernetes cluster.",
42+
"required": [],
43+
"title": "clusterDomain",
44+
"type": "string"
45+
},
446
"global": {
547
"description": "Global values are values that can be accessed from any chart or subchart by exactly the same name.",
648
"required": [],

‎charts/nginx-gateway-fabric/values.yaml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
# yaml-language-server: $schema=values.schema.json
22

3+
# -- The DNS cluster domain of your Kubernetes cluster.
4+
clusterDomain: cluster.local
5+
36
# -- The nginxGateway section contains configuration for the NGINX Gateway Fabric control plane deployment.
47
nginxGateway:
58
# FIXME(lucacome): https://github.com/nginx/nginx-gateway-fabric/issues/2490
@@ -429,3 +432,21 @@ nginx:
429432

430433
# -- Enable debugging for NGINX. Uses the nginx-debug binary. The NGINX error log level should be set to debug in the NginxProxy resource.
431434
debug: false
435+
436+
# -- The certGenerator section contains the configuration for the cert-generator Job.
437+
certGenerator:
438+
# -- The annotations of the cert-generator Job.
439+
annotations: {}
440+
441+
# -- The name of the Secret containing TLS CA, certificate, and key for the NGINX Gateway Fabric control plane
442+
# to securely communicate with the NGINX Agent. Must exist in the same namespace that the NGINX Gateway Fabric
443+
# control plane is running in (default namespace: nginx-gateway).
444+
serverTLSSecretName: server-tls
445+
446+
# -- The name of the base Secret containing TLS CA, certificate, and key for the NGINX Agent to securely
447+
# communicate with the NGINX Gateway Fabric control plane. Must exist in the same namespace that the
448+
# NGINX Gateway Fabric control plane is running in (default namespace: nginx-gateway).
449+
agentTLSSecretName: agent-tls
450+
451+
# -- Overwrite existing TLS Secrets on startup.
452+
overwrite: false

0 commit comments

Comments
 (0)