Skip to content

add information on load balancer lifecycle #2703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

add information on load balancer lifecycle #2703

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
77f8781
add lifecycle info
geoffcline Jun 24, 2022
dd99c0d
add lifecycle info
geoffcline Jun 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/guide/ingress/annotations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ You can add annotations to kubernetes Ingress and Service objects to customize t
- Annotations that configures LoadBalancer / Listener behaviors have different merge behavior when IngressGroup feature is been used. `MergeBehavior` column below indicates how such annotation will be merged.
- Exclusive: such annotation should only be specified on a single Ingress within IngressGroup or specified with same value across all Ingresses within IngressGroup.
- Merge: such annotation can be specified on all Ingresses within IngressGroup, and will be merged together.
- The controller may not update the configuration of an *existing* load balancer if an annotation is added/updated. New load balancers will be have an appropriate configuration.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This controller should update the existing load balancer. In certain case, for example when the load balancer scheme gets modified from internal to internet-facing or vice-versa, controller deletes existing lb and recreates a new one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I commented my issue on the other PR (#2680 (comment)) -- let's discuss there


## Annotations
|Name | Type |Default|Location|MergeBehavior|
Expand Down
1 change: 1 addition & 0 deletions docs/guide/service/annotations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- stringList: `"s1,s2,s3"`
- stringMap: `"k1=v1,k2=v2"`
- json: `"{ \"key\": \"value\" }"`
- The controller may not update the configuration of an *existing* load balancer if an annotation is added/updated. New load balancers will be have an appropriate configuration.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Barring some limitations, controller does update the existing load balancer. There are some issues with the kube-controller-manager provisioned load balancer, but this controller should be able to update most of the configuration.

Limitations:

  • modifying/adding the annotation service.beta.kubernetes.io/aws-load-balancer-type after service creation is not recommended as there is a potential for leaking the AWS LB resource in case both kcm and this controller act on the service
  • if user modifies the lb scheme, controller deletes and creates a new lb with the new scheme
  • updating subnets may not work - this is due to any temporary limitation put in by the AWS


## Annotations
!!!warning
Expand Down
18 changes: 18 additions & 0 deletions docs/how-it-works.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,21 @@ Ingress traffic starts at the ALB and reaches the Kubernetes nodes through each
#### IP mode
Ingress traffic starts at the ALB and reaches the Kubernetes pods directly. CNIs must support directly accessible POD ip via [secondary IP addresses on ENI](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html).

## Lifecycle of AWS Load Balancers

!!! warning
The controller, generally, does not update load balancers after creation.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

load balancer does get updated, this warning is not accurate.



The controller updates the targets of load balancers in response to cluster workload changes. However, the controller may not update the configuration of load balancers in response to annotations added to cluster resources after the load balancer is created.

For example, if you add the annotation `aws-load-balancer-ssl-cert` to a service after the load balancer is created, the controller will *not* update the configuration of the AWS load balancer. However, if you delete the AWS load balancer, the controller will *recreate* the load balancer with the desired configuration.

Recommendations:
1. Assume AWS load balancers may need to be replaced.
1. Do not manually update the configuration of load balancers provisioned by the controller. This will cause the in-cluster resource definitions and the AWS configuration to get out of sync.
1. If durability of a specific load balancer resource is important, consider using IAM permissions to block the controller from creating or deleting new load balancers. Use the controller only to update the targets of self-managed load balancers.
1. Use the [ExternalDNS Controller](https://github.com/kubernetes-sigs/external-dns) to automatically update DNS records instead of relying on a specific load balancer IP address or hostname.