[ws-manager-mk2] Prevent partial workspace backups #16997
Conversation
Furisto
added
do-not-merge/hold
team: workspace
Furisto
force-pushed
the
fo/mk2-restart-backup
branch
from
93eb0eb to
4aff3e3
Compare
| wsc.emitEvent(ws, "Backup", disposeErr) | ||
| wsc.emitEvent(ws, "Backup", fmt.Errorf("failed to backup workspace: %w", disposeErr)) | ||
|
|
||
| err = wsc.operations.DeleteWorkspace(ctx, ws.Name) |
There was a problem hiding this comment.
how do we ensure that the workspace controller doesn't in the meantime remove the finalizer and delete the workspace resource (because the BackupComplete condition is added)? Is it OK if we don't complete the DeleteWorkspace call in rare occasions, e.g. if daemon restarts after Backup and before Delete, and when daemon comes back up the workspace resource is gone
There was a problem hiding this comment.
We have housekeeping in place that deletes abandoned workspaces, so even if the deletion of a workspace is interrupted it will be cleaned up after 5 minutes. It currently only takes care of mk1 workspaces but I will add one for mk2 as well. Just for getting started with testing mk2 in Dedicated the current approach is good enough in my opinion. We have to fix it for prod though.
roboquat
added
deployed: workspace
Description
Depends on #16823
After we have taken the backup we delete the workspace content and only then we update the backup condition on the workspace CR. If the delete operation is interrupted due to ws-daemon restart we will retry the backup due to the missing backup condition and will create only a partial backup. Instead we now first update the backup condition and only then we delete the workspace content.
Related Issue(s)
n.a.
How to test
Release Notes
Documentation
Build Options:
Run the build with werft instead of GHA
Run Leeway with
--dont-testPublish Options
Installer Options
Add desired feature flags to the end of the line above, space separated
Preview Environment Options:
If enabled this will build
install/previewIf enabled this will create the environment on GCE infra
Valid options are
all,workspace,webapp,ide,jetbrains,vscode,ssh