Skip to content

Commit 7d34a0d

Browse files
yo-gayo-ga
authored
feat: Specify Managed Prefix List for access control (#3584)
* feat: new annotation * feat: new ec2model * feat: prefix list implement * chore: annotation docs * feat: add ingress unit test * feat: add service unit test * chore: update example * chore: multiple pl test
1 parent 1a83911 commit 7d34a0d

File tree

11 files changed

+870
-6
lines changed

11 files changed

+870
-6
lines changed

‎docs/guide/ingress/annotations.md

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ You can add annotations to kubernetes Ingress and Service objects to customize t
3333
| [alb.ingress.kubernetes.io/listen-ports](#listen-ports) | json |'[{"HTTP": 80}]' \| '[{"HTTPS": 443}]'|Ingress|Merge|
3434
| [alb.ingress.kubernetes.io/ssl-redirect](#ssl-redirect) | integer |N/A|Ingress|Exclusive|
3535
| [alb.ingress.kubernetes.io/inbound-cidrs](#inbound-cidrs) | stringList |0.0.0.0/0, ::/0|Ingress|Exclusive|
36+
| [alb.ingress.kubernetes.io/security-group-prefix-lists](#security-group-prefix-lists) | stringList |pl-00000000, pl-1111111|Ingress|Exclusive|
3637
| [alb.ingress.kubernetes.io/certificate-arn](#certificate-arn) | stringList |N/A|Ingress|Merge|
3738
| [alb.ingress.kubernetes.io/ssl-policy](#ssl-policy) | string |ELBSecurityPolicy-2016-08|Ingress|Exclusive|
3839
| [alb.ingress.kubernetes.io/target-type](#target-type) | instance \| ip |instance|Ingress,Service|N/A|
@@ -515,7 +516,7 @@ Access control for LoadBalancer can be controlled with following annotations:
515516
`inbound-cidrs` is merged across all Ingresses in IngressGroup, but is exclusive per listen-port.
516517

517518
- the `inbound-cidrs` will only impact the ports defined for that Ingress.
518-
- if same listen-port is defined by multiple Ingress within IngressGroup, inbound-cidrs should only be defined on one of the Ingress.
519+
- if same listen-port is defined by multiple Ingress within IngressGroup, `inbound-cidrs` should only be defined on one of the Ingress.
519520

520521
!!!note "Default"
521522

@@ -530,10 +531,32 @@ Access control for LoadBalancer can be controlled with following annotations:
530531
alb.ingress.kubernetes.io/inbound-cidrs: 10.0.0.0/24
531532
```
532533

534+
- <a name="security-group-prefix-lists">`alb.ingress.kubernetes.io/security-group-prefix-lists`</a> specifies the managed prefix lists that are allowed to access LoadBalancer.
535+
536+
!!!note "Merge Behavior"
537+
`security-group-prefix-lists` is merged across all Ingresses in IngressGroup, but is exclusive per listen-port.
538+
539+
- the `security-group-prefix-lists` will only impact the ports defined for that Ingress.
540+
- if same listen-port is defined by multiple Ingress within IngressGroup, `security-group-prefix-lists` should only be defined on one of the Ingress.
541+
542+
!!!warning ""
543+
This annotation will be ignored if `alb.ingress.kubernetes.io/security-groups` is specified.
544+
545+
!!!warning ""
546+
If you'd like to use this annotation, make sure your security group rule quota is enough. If you'd like to know how the managed prefix list affects your quota, see the [reference](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-aws-managed-prefix-lists.html#aws-managed-prefix-list-weights) in the AWS documentation for more details.
547+
548+
!!!tip ""
549+
If you only use this annotation without `inbound-cidrs`, the controller managed security group would ignore the `inbound-cidrs` default settings.
550+
551+
!!!example
552+
```
553+
alb.ingress.kubernetes.io/security-group-prefix-lists: pl-000000, pl-111111
554+
```
555+
533556
- <a name="security-groups">`alb.ingress.kubernetes.io/security-groups`</a> specifies the securityGroups you want to attach to LoadBalancer.
534557

535558
!!!note ""
536-
When this annotation is not present, the controller will automatically create one security group, the security group will be attached to the LoadBalancer and allow access from [`inbound-cidrs`](#inbound-cidrs) to the [`listen-ports`](#listen-ports).
559+
When this annotation is not present, the controller will automatically create one security group, the security group will be attached to the LoadBalancer and allow access from [`inbound-cidrs`](#inbound-cidrs) and [`security-group-prefix-lists`](#security-group-prefix-lists) to the [`listen-ports`](#listen-ports).
537560
Also, the securityGroups for Node/Pod will be modified to allow inbound traffic from this securityGroup.
538561

539562
!!!note ""

‎docs/guide/service/annotations.md

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
| Name | Type | Default | Notes |
1717
|--------------------------------------------------------------------------------------------------|-------------------------|---------------------------|--------------------------------------------------------|
1818
| [service.beta.kubernetes.io/load-balancer-source-ranges](#lb-source-ranges) | stringList | | |
19+
| [service.beta.kubernetes.io/aws-load-balancer-security-group-prefix-lists](#lb-security-group-prefix-lists) | stringList | | |
1920
| [service.beta.kubernetes.io/aws-load-balancer-type](#lb-type) | string | | |
2021
| [service.beta.kubernetes.io/aws-load-balancer-nlb-target-type](#nlb-target-type) | string | | default `instance` in case of LoadBalancerClass |
2122
| [service.beta.kubernetes.io/aws-load-balancer-name](#load-balancer-name) | string | | |
@@ -446,6 +447,22 @@ Load balancer access can be controlled via following annotations:
446447
service.beta.kubernetes.io/load-balancer-source-ranges: 10.0.0.0/24
447448
```
448449

450+
- <a name="lb-security-group-prefix-lists">`service.beta.kubernetes.io/aws-load-balancer-security-group-prefix-lists`</a> specifies the managed prefix lists that are allowed to access the NLB.
451+
452+
!!!warning ""
453+
this annotation will be ignored if `service.beta.kubernetes.io/aws-load-balancer-security-groups` is specified.
454+
455+
!!!warning ""
456+
If you'd like to use this annotation, make sure your security group rule quota is enough. If you'd like to know how the managed prefix list affects your quota, see the [reference](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-aws-managed-prefix-lists.html#aws-managed-prefix-list-weights) in the AWS documentation for more details.
457+
458+
!!!tip ""
459+
If you only use this annotation without `load-balancer-source-ranges`, the controller managed security group would ignore the `load-balancer-source-ranges` default settings.
460+
461+
!!!example
462+
```
463+
service.beta.kubernetes.io/aws-load-balancer-security-group-prefix-lists: pl-00000000, pl-1111111
464+
```
465+
449466
- <a name="lb-scheme">`service.beta.kubernetes.io/aws-load-balancer-scheme`</a> specifies whether the NLB will be internet-facing or internal. Valid values are `internal`, `internet-facing`. If not specified, default is `internal`.
450467

451468
!!!example
@@ -465,7 +482,7 @@ Load balancer access can be controlled via following annotations:
465482
- <a name="security-groups">`service.beta.kubernetes.io/aws-load-balancer-security-groups`</a> specifies the frontend securityGroups you want to attach to an NLB.
466483

467484
!!!note ""
468-
When this annotation is not present, the controller will automatically create one security group. The security group will be attached to the LoadBalancer and allow access from `inbound-cidrs` to the `listen-ports`.
485+
When this annotation is not present, the controller will automatically create one security group. The security group will be attached to the LoadBalancer and allow access from `load-balancer-source-ranges` and `aws-load-balancer-security-group-prefix-lists` to the `listen-ports`.
469486
Also, the securityGroups for target instances/ENIs will be modified to allow inbound traffic from this securityGroup.
470487

471488
!!!note ""

‎pkg/annotations/constants.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ const (
4747
IngressSuffixTargetNodeLabels = "target-node-labels"
4848
IngressSuffixManageSecurityGroupRules = "manage-backend-security-group-rules"
4949
IngressSuffixMutualAuthentication = "mutual-authentication"
50+
IngressSuffixSecurityGroupPrefixLists = "security-group-prefix-lists"
5051

5152
// NLB annotation suffixes
5253
// prefixes service.beta.kubernetes.io, service.kubernetes.io
@@ -86,4 +87,5 @@ const (
8687
SvcLBSuffixLoadBalancerSecurityGroups = "aws-load-balancer-security-groups"
8788
SvcLBSuffixManageSGRules = "aws-load-balancer-manage-backend-security-group-rules"
8889
SvcLBSuffixEnforceSGInboundRulesOnPrivateLinkTraffic = "aws-load-balancer-inbound-sg-rules-on-private-link-traffic"
90+
SvcLBSuffixSecurityGroupPrefixLists = "aws-load-balancer-security-group-prefix-lists"
8991
)

‎pkg/deploy/ec2/security_group_manager.go

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,10 @@ func buildIPPermissionInfo(permission ec2model.IPPermission) (networking.IPPermi
168168
labels := networking.NewIPPermissionLabelsForRawDescription(permission.UserIDGroupPairs[0].Description)
169169
return networking.NewGroupIDIPPermission(protocol, permission.FromPort, permission.ToPort, permission.UserIDGroupPairs[0].GroupID, labels), nil
170170
}
171+
if len(permission.PrefixLists) == 1 {
172+
labels := networking.NewIPPermissionLabelsForRawDescription(permission.PrefixLists[0].Description)
173+
return networking.NewPrefixListIDPermission(protocol, permission.FromPort, permission.ToPort, permission.PrefixLists[0].ListID, labels), nil
174+
}
171175
return networking.IPPermissionInfo{}, errors.New("invalid ipPermission")
172176
}
173177

‎pkg/ingress/model_build_listener.go

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,7 @@ type listenPortConfig struct {
103103
protocol elbv2model.Protocol
104104
inboundCIDRv4s []string
105105
inboundCIDRv6s []string
106+
prefixLists []string
106107
sslPolicy *string
107108
tlsCerts []string
108109
mutualAuthentication *elbv2model.MutualAuthenticationAttributes
@@ -111,6 +112,8 @@ type listenPortConfig struct {
111112
func (t *defaultModelBuildTask) computeIngressListenPortConfigByPort(ctx context.Context, ing *ClassifiedIngress) (map[int64]listenPortConfig, error) {
112113
explicitTLSCertARNs := t.computeIngressExplicitTLSCertARNs(ctx, ing.Ing)
113114
explicitSSLPolicy := t.computeIngressExplicitSSLPolicy(ctx, ing)
115+
var prefixListIDs []string
116+
t.annotationParser.ParseStringSliceAnnotation(annotations.IngressSuffixSecurityGroupPrefixLists, &prefixListIDs, ing.Ing.Annotations)
114117
inboundCIDRv4s, inboundCIDRV6s, err := t.computeIngressExplicitInboundCIDRs(ctx, ing)
115118
if err != nil {
116119
return nil, err
@@ -146,6 +149,7 @@ func (t *defaultModelBuildTask) computeIngressListenPortConfigByPort(ctx context
146149
protocol: protocol,
147150
inboundCIDRv4s: inboundCIDRv4s,
148151
inboundCIDRv6s: inboundCIDRV6s,
152+
prefixLists: prefixListIDs,
149153
}
150154
if protocol == elbv2model.ProtocolHTTPS {
151155
if len(explicitTLSCertARNs) == 0 {

‎pkg/ingress/model_build_managed_sg.go

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,18 @@ func (t *defaultModelBuildTask) buildManagedSecurityGroupIngressPermissions(_ co
9797
})
9898
}
9999
}
100+
for _, prefixID := range cfg.prefixLists {
101+
permissions = append(permissions, ec2model.IPPermission{
102+
IPProtocol: "tcp",
103+
FromPort: awssdk.Int64(port),
104+
ToPort: awssdk.Int64(port),
105+
PrefixLists: []ec2model.PrefixList{
106+
{
107+
ListID: prefixID,
108+
},
109+
},
110+
})
111+
}
100112
}
101113
return permissions
102114
}

‎pkg/ingress/model_builder.go

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -298,6 +298,9 @@ func (t *defaultModelBuildTask) mergeListenPortConfigs(_ context.Context, listen
298298
mergedInboundCIDRv6s := sets.NewString()
299299
mergedInboundCIDRv4s := sets.NewString()
300300

301+
var mergedInboundPrefixListsProvider *types.NamespacedName
302+
mergedInboundPrefixLists := sets.NewString()
303+
301304
var mergedSSLPolicyProvider *types.NamespacedName
302305
var mergedSSLPolicy *string
303306

@@ -329,6 +332,17 @@ func (t *defaultModelBuildTask) mergeListenPortConfigs(_ context.Context, listen
329332
}
330333
}
331334

335+
if len(cfg.listenPortConfig.prefixLists) != 0 {
336+
cfgInboundPrefixLists := sets.NewString(cfg.listenPortConfig.prefixLists...)
337+
if mergedInboundPrefixListsProvider == nil {
338+
mergedInboundPrefixListsProvider = &cfg.ingKey
339+
mergedInboundPrefixLists = cfgInboundPrefixLists
340+
} else if !mergedInboundPrefixLists.Equal(cfgInboundPrefixLists) {
341+
return listenPortConfig{}, errors.Errorf("conflicting inbound-prefix-lists, %v: %v | %v: %v",
342+
*mergedInboundPrefixListsProvider, mergedInboundPrefixLists.List(), cfg.ingKey, cfgInboundPrefixLists.List())
343+
}
344+
}
345+
332346
if cfg.listenPortConfig.sslPolicy != nil {
333347
if mergedSSLPolicyProvider == nil {
334348
mergedSSLPolicyProvider = &cfg.ingKey
@@ -359,7 +373,7 @@ func (t *defaultModelBuildTask) mergeListenPortConfigs(_ context.Context, listen
359373

360374
}
361375

362-
if len(mergedInboundCIDRv4s) == 0 && len(mergedInboundCIDRv6s) == 0 {
376+
if len(mergedInboundCIDRv4s) == 0 && len(mergedInboundCIDRv6s) == 0 && len(mergedInboundPrefixLists) == 0 {
363377
mergedInboundCIDRv4s.Insert("0.0.0.0/0")
364378
mergedInboundCIDRv6s.Insert("::/0")
365379
}
@@ -377,6 +391,7 @@ func (t *defaultModelBuildTask) mergeListenPortConfigs(_ context.Context, listen
377391
protocol: mergedProtocol,
378392
inboundCIDRv4s: mergedInboundCIDRv4s.List(),
379393
inboundCIDRv6s: mergedInboundCIDRv6s.List(),
394+
prefixLists: mergedInboundPrefixLists.List(),
380395
sslPolicy: mergedSSLPolicy,
381396
tlsCerts: mergedTLSCerts,
382397
mutualAuthentication: mergedMtlsAttributes,

0 commit comments

Comments
 (0)