v3: prod worker graceful shutdown

[nicktrn]

• Give workers 10 minutes to finish their current attempt when receiving SIGTERM
• Fail run with timeout error if it doesn't exit in that timeframe
• Fix an issue where large timeout delays could exceed 32 bit signed integer limits

[changeset-bot]

🦋 Changeset detected

Latest commit: 44a1a92

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[matt-aitken]

Some questions on this:

• In what situations would workers get a SIGTERM?
• What's the worst outcome for a customer running a task and they received a SIGTERM?

[nicktrn]

Should probably have prefaced this by stating the previous default, which was to quit immediately. This is more of a precaution to limit situations that are difficult to debug. With these changes we'll know when this happens, and we have control over the shutdown process.

In what situations would workers get a SIGTERM?

Manually or automatically terminating containers / pods.

• Manual: May be required when runs appear "stuck" or otherwise unresponsive.
• Automatic: Could be for a number of reasons. OOM conditions or other critical issues, and when scaling down with tasks still on the affected node. I'm trying to reduce the chances of any of this happening.

What's the worst outcome for a customer running a task and they received a SIGTERM?

Worst case would be the entire run fails and has to be replayed.

Could up the grace period to an hour for now?

[matt-aitken]

I had one question