Skip to content

fix: health check timeout for services/NLB #2899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Dec 2, 2022
Merged

fix: health check timeout for services/NLB #2899

merged 3 commits into from
Dec 2, 2022

Conversation

project0
Copy link
Contributor

@project0 project0 commented Nov 28, 2022 •
edited
Loading

Issue

fix #2898

and it solves this error when the interval is set lower:

{"level":"error","ts":1669717296.2819514,"logger":"controller.service","msg":"Reconciler error","name":nlb","namespace":"app","error":"ValidationError: Health check interval must be greater than the timeout.\n\tstatus code: 400, request id: b2ddcc99-6828-4768-a7c3-ca19ea018291"}

Description

The annotation for health check timeout was not consumed at all for services, the test have been also wrong.

Checklist

  • Added tests that cover your change (if possible)
  • Added/modified documentation as required (such as the README.md, or the docs directory)
  • Manually tested
  • Made sure the title of the PR is a good description that can go into the release notes

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

  • Backfilled missing tests for code in same general area 🎉
  • Refactored something and made the world a better place 🌟

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Nov 28, 2022 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: project0 / name: Richard Hillmann (99b7922f7d720674bcc6f17815ea495945f6d516)

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Nov 28, 2022
@k8s-ci-robot
Copy link
Contributor

Welcome @project0!

It looks like this is your first PR to kubernetes-sigs/aws-load-balancer-controller 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/aws-load-balancer-controller has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 28, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @project0. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 28, 2022
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Nov 28, 2022
@project0 project0 force-pushed the fix/alb-health-timeout-annotation branch from 99b7922 to 45f0554 Compare November 28, 2022 17:16
@project0
Copy link
Contributor Author

I am a bit of afraid to mark it only as "bug fix", as it clearly is changing now the default settings as supposed, so it will maybe set unwanted default values. Maybe worth to mention in the change log.

@project0
fix: health check timeout for services/NLB
b3b374f 
The annotation for health check timeout was not consumed at all for
services
@project0 project0 force-pushed the fix/alb-health-timeout-annotation branch from 45f0554 to b3b374f Compare November 28, 2022 18:58
@project0
Copy link
Contributor Author

sorry, did not fixed all tests, should be green now :-)

@kishorj kishorj added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Nov 29, 2022
@kishorj
Copy link
Collaborator

kishorj commented Nov 29, 2022

@project0, thanks for your contribution. We ignored the health check timeout earlier due to some limitations on the NLB side. The limitations have now been removed, so we should be able to configure the timeouts as expected. There is still a possibility some regions might not support the new configuration, so I'd recommend to add a feature gate flag set to true by default so we can disable the timeout if support is not available in a certain region.

kishorj
kishorj reviewed Nov 29, 2022
View reviewed changes
@project0
Copy link
Contributor Author

project0 commented Nov 30, 2022 •
edited
Loading

@project0, thanks for your contribution. We ignored the health check timeout earlier due to some limitations on the NLB side. The limitations have now been removed, so we should be able to configure the timeouts as expected. There is still a possibility some regions might not support the new configuration, so I'd recommend to add a feature gate flag set to true by default so we can disable the timeout if support is not available in a certain region.

@kishorj thanks for the explanation. It would have been nice if the documentation had reflected the not implemented feature by intention. It would be good to either not document them or add a warning next time.

@project0 project0 force-pushed the fix/alb-health-timeout-annotation branch from d809bdc to abf2252 Compare November 30, 2022 15:18
@project0 project0 requested review from kishorj and removed request for M00nF1sh November 30, 2022 15:18
@project0
add feature gate ServiceHealthCheckTimeout
5edc83f 
allows people to set old behavior
@project0 project0 force-pushed the fix/alb-health-timeout-annotation branch from abf2252 to 5edc83f Compare November 30, 2022 15:28
kishorj
kishorj reviewed Nov 30, 2022
View reviewed changes
pkg/service/model_build_target_group.go Outdated
@@ -309,6 +320,9 @@ func (t *defaultModelBuildTask) buildTargetGroupHealthCheckIntervalSeconds(_ con
}

func (t *defaultModelBuildTask) buildTargetGroupHealthCheckTimeoutSeconds(_ context.Context, defaultHealthCheckTimeout int64) (int64, error) {
if !t.featureGates.Enabled(config.ServiceHealthCheckTimeout) {
return 0, nil
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

timeout value 0 will not work. Two issues:

  • 0 is not valid timeout, the current valid range is [2, 120]
  • in case of the prior behavior, the elbv2 api didn't accept the timeout values other than 10 or 30 if the field was set

The feature gate check needs to be done at the calling side

Copy link
Contributor Author

@project0 project0 Dec 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The feature gate check needs to be done at the calling side

I am not sure what you mean here, you mean i should duplicate code at the method caller? why, the behavior is triggered for both defaults.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with the *int64 return type, the feature gate check can be done on the function as well. I take back my earlier comments regards to the feature gate check.

pkg/service/model_build_target_group.go Outdated
@@ -161,6 +171,7 @@ func (t *defaultModelBuildTask) buildTargetGroupHealthCheckConfigForInstanceMode
Protocol: &healthCheckProtocol,
Path: healthCheckPathPtr,
IntervalSeconds: &intervalSeconds,
TimeoutSeconds: &healthCheckTimeoutSeconds,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nil ptr if not enabled

@kishorj
Copy link
Collaborator

kishorj commented Dec 1, 2022

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 1, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kishorj, project0

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 1, 2022
@kishorj
Copy link
Collaborator

kishorj commented Dec 1, 2022

@project0, thanks again for your contribution. I will merge the changes once the e2e tests pass.

@kishorj
Copy link
Collaborator

kishorj commented Dec 2, 2022

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 2, 2022
@k8s-ci-robot k8s-ci-robot merged commit 2e740a7 into kubernetes-sigs:main Dec 2, 2022
This was referenced Dec 30, 2022
Closed
Closed
@kishorj kishorj mentioned this pull request Jan 12, 2023
Merged
Timothy-Dougherty pushed a commit to adammw/aws-load-balancer-controller that referenced this pull request Nov 9, 2023
@project0 @Timothy-Dougherty
fix: health check timeout for services/NLB (kubernetes-sigs#2899)
b4ee441 
* fix: health check timeout for services/NLB

The annotation for health check timeout was not consumed at all for
services

* add feature gate ServiceHealthCheckTimeout

allows people to set old behavior

* feat: rename featuregate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kishorj kishorj kishorj approved these changes

Assignees

@kishorj kishorj

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Service NLB health check timeout annotation is ignored.
3 participants
@project0 @k8s-ci-robot @kishorj