Improve resume reliability of restored runs #1458
Conversation
🦋 Changeset detectedLatest commit: 5811339 The changes in this PR will be included in the next version bump. This PR includes changesets to release 10 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
WalkthroughThe pull request introduces enhancements to the "trigger.dev" component, specifically improving the reliability of worker resume functionality. A new mechanism is implemented to replay the ready signal for restored workers, ensuring they can effectively resume operations. Additionally, modifications to the Changes
Possibly related PRs
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (1)
.changeset/sixty-donuts-fail.md (1)
5-5: Consider enhancing the changeset description.While the description is clear, it would be helpful to include:
- The specific reliability issues being addressed
- The impact of these improvements on worker behavior
- Any notable changes in the resume process
Example enhancement:
-Improve resume reliability by replaying ready signal of restored workers +Improve resume reliability by replaying ready signal of restored workers. This fixes potential deadlocks where restored workers would fail to resume operations due to missed ready signals. Workers now properly re-establish their ready state after restoration, ensuring consistent task processing.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
.changeset/sixty-donuts-fail.md(1 hunks)packages/cli-v3/src/entryPoints/deploy-run-controller.ts(8 hunks)
🔇 Additional comments (7)
.changeset/sixty-donuts-fail.md (1)
1-3: LGTM! Appropriate version bump.
The patch version bump is correct as this change improves existing functionality without breaking changes.
packages/cli-v3/src/entryPoints/deploy-run-controller.ts (6)
100-105: Addition of 'readyForResumeReplay' property is properly implemented
The new private property readyForResumeReplay is appropriately defined and typed, facilitating the management of resume replay state.
374-385: Ensuring proper state reset in #prepareForRetry()
Including this.readyForResumeReplay = undefined; in the #prepareForRetry() method ensures that the replay state is correctly cleared during retries, preventing unintended behavior from stale state.
422-422: Resetting 'readyForResumeReplay' in #resumeAfterDuration()
Clearing this.readyForResumeReplay when resuming after duration maintains consistent state management and prevents potential replay issues.
473-558: Implementation of #readyForResume() method is sound
The new method #readyForResume() properly handles the logic for resuming operations, including validation checks for necessary identifiers and emitting the READY_FOR_RESUME event. The retry mechanism and idempotency key management appear correctly implemented.
737-737: Clearing 'readyForResumeReplay' in RESUME_AFTER_DEPENDENCY handler
Resetting this.readyForResumeReplay on resuming after dependency ensures that the replay state does not persist incorrectly.
1362-1364: Including 'readyForResumeReplay' in #status provides better visibility
Adding readyForResumeReplay to the #status getter ensures that the current replay state is exposed, aiding in debugging and monitoring.
Previously, it was possible for a restored worker to come up and to signal it's ready to resume when nobody was listening. We will replay the readiness signal with exponential backoff now.
Summary by CodeRabbit
New Features
Bug Fixes