Skip to content

[ws-manager-mk2] Prevent partial workspace backups #16997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 24, 2023
Merged

[ws-manager-mk2] Prevent partial workspace backups #16997

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions components/ws-daemon/pkg/controller/workspace_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,7 +269,7 @@ func (wsc *WorkspaceController) handleWorkspaceStop(ctx context.Context, ws *wor
}
}

gitStatus, disposeErr := wsc.operations.DisposeWorkspace(ctx, DisposeOptions{
gitStatus, disposeErr := wsc.operations.BackupWorkspace(ctx, DisposeOptions{
Meta: WorkspaceMeta{
Owner: ws.Spec.Ownership.Owner,
WorkspaceId: ws.Spec.Ownership.WorkspaceID,
Expand All @@ -289,7 +289,7 @@ func (wsc *WorkspaceController) handleWorkspaceStop(ctx context.Context, ws *wor
ws.Status.GitStatus = toWorkspaceGitStatus(gitStatus)

if disposeErr != nil {
log.Error(disposeErr, "failed to dispose workspace", "name", ws.Name)
log.Error(disposeErr, "failed to backup workspace", "name", ws.Name)
ws.Status.SetCondition(workspacev1.NewWorkspaceConditionBackupFailure(disposeErr.Error()))
} else {
ws.Status.SetCondition(workspacev1.NewWorkspaceConditionBackupComplete())
Expand All @@ -302,7 +302,14 @@ func (wsc *WorkspaceController) handleWorkspaceStop(ctx context.Context, ws *wor
wsc.metrics.recordFinalizeTime(time.Since(disposeStart).Seconds(), ws)
}

wsc.emitEvent(ws, "Backup", disposeErr)
wsc.emitEvent(ws, "Backup", fmt.Errorf("failed to backup workspace: %w", disposeErr))

err = wsc.operations.DeleteWorkspace(ctx, ws.Name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we ensure that the workspace controller doesn't in the meantime remove the finalizer and delete the workspace resource (because the BackupComplete condition is added)? Is it OK if we don't complete the DeleteWorkspace call in rare occasions, e.g. if daemon restarts after Backup and before Delete, and when daemon comes back up the workspace resource is gone

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have housekeeping in place that deletes abandoned workspaces, so even if the deletion of a workspace is interrupted it will be cleaned up after 5 minutes. It currently only takes care of mk1 workspaces but I will add one for mk2 as well. Just for getting started with testing mk2 in Dedicated the current approach is good enough in my opinion. We have to fix it for prod though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok, that works

if err != nil {
wsc.emitEvent(ws, "Backup", fmt.Errorf("failed to clean up workspace: %w", err))
return ctrl.Result{}, err
}

return ctrl.Result{}, err
}

Expand Down
15 changes: 13 additions & 2 deletions components/ws-daemon/pkg/controller/workspace_operations.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ func (wso *WorkspaceOperations) creator(owner, workspaceId, instanceId string, i
}
}

func (wso *WorkspaceOperations) DisposeWorkspace(ctx context.Context, opts DisposeOptions) (*csapi.GitStatus, error) {
func (wso *WorkspaceOperations) BackupWorkspace(ctx context.Context, opts DisposeOptions) (*csapi.GitStatus, error) {
ws, err := wso.provider.Get(ctx, opts.Meta.InstanceId)
if err != nil {
return nil, fmt.Errorf("cannot find workspace %s during DisposeWorkspace: %w", opts.Meta.InstanceId, err)
Expand Down Expand Up @@ -207,16 +207,27 @@ func (wso *WorkspaceOperations) DisposeWorkspace(ctx context.Context, opts Dispo
}
}

return repo, nil
}

func (wso *WorkspaceOperations) DeleteWorkspace(ctx context.Context, instanceID string) error {
ws, err := wso.provider.Get(ctx, instanceID)
if err != nil {
return fmt.Errorf("cannot find workspace %s during DisposeWorkspace: %w", instanceID, err)
}

if err = ws.Dispose(ctx, wso.provider.hooks[session.WorkspaceDisposed]); err != nil {
glog.WithError(err).Error("cannot dispose session")
return err
}

// remove workspace daemon directory in the node
if err := os.RemoveAll(ws.ServiceLocDaemon); err != nil {
glog.WithError(err).Error("cannot delete workspace daemon directory")
return err
}

return repo, nil
return nil
}

func (wso *WorkspaceOperations) SnapshotIDs(ctx context.Context, workspaceID string) (snapshotUrl, snapshotName string, err error) {
Expand Down