[ws-manager-mk2] Refactor metrics with EverReady condition #17114
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WVerlaek
commented
Mar 31, 2023
Comment on lines
+163
to
+179
| if c := wsk8s.GetCondition(ws.Status.Conditions, string(workspacev1.WorkspaceConditionFailed)); c != nil { | ||
| reason = StopReasonFailed | ||
| if !wsk8s.ConditionPresentAndTrue(ws.Status.Conditions, string(workspacev1.WorkspaceConditionEverReady)) { | ||
| // Don't record 'failed' if there was a start failure. | ||
| reason = StopReasonStartFailure | ||
| } else if strings.Contains(c.Message, "Pod ephemeral local storage usage exceeds the total limit of containers") { | ||
| reason = StopReasonOutOfSpace | ||
| } | ||
| } else if wsk8s.ConditionPresentAndTrue(ws.Status.Conditions, string(workspacev1.WorkspaceConditionAborted)) { | ||
| reason = StopReasonAborted | ||
| } else if wsk8s.ConditionPresentAndTrue(ws.Status.Conditions, string(workspacev1.WorkspaceConditionTimeout)) { | ||
| reason = StopReasonTimeout | ||
| } else if wsk8s.ConditionPresentAndTrue(ws.Status.Conditions, string(workspacev1.WorkspaceConditionClosed)) { | ||
| reason = StopReasonTabClosed | ||
| } else { | ||
| reason = StopReasonRegular | ||
| } |
There was a problem hiding this comment.
re-ordered these checks compared to mk1, to e.g. check for Failure first. If there's both a failure and closed condition, we'd want the failure reason to be reported instead of closed
WVerlaek
commented
Mar 31, 2023
| reason = StopReasonFailed | ||
| if !wsk8s.ConditionPresentAndTrue(ws.Status.Conditions, string(workspacev1.WorkspaceConditionEverReady)) { | ||
| // Don't record 'failed' if there was a start failure. | ||
| reason = StopReasonStartFailure |
There was a problem hiding this comment.
different from MK1: instead of not incrementing the stop metric when there was a start failure, we increment it now but with its own unique reason.
WVerlaek
added a commit
that referenced
this pull request
Mar 31, 2023
roboquat
pushed a commit
that referenced
this pull request
Apr 3, 2023
aledbf
approved these changes
Apr 8, 2023
roboquat
added
deployed: workspace
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Introduce an
EverReadycondition:Ready, i.e. when content init succeeded and the supervisor readiness check passes.This condition then allows us to:
Stoppedphase without ever becoming readyRunningtoInitializingif its container becomes unready. Once a workspace becomes ready it should not move backwards in phaseTo track workspace failures that happen after startup, a new
workspace_failure_totalmetric is added which gets incremented whenever a workspaces receives theFailedcondition. This includes content init and disposal failures, but also all other possible failures.The
workspace_stops_totalmetric is also updated to match MK1 behaviour by including the stopreasonlabel.Related Issue(s)
Fixes WKS-29, WKS-23
How to test
Run unit tests
Or in a preview env, check a workspace starts and stops as expected, and the right metrics are incremented.
E.g. the following stop metric reasons are reported for a workspace with an image build:
Release Notes
Documentation
Build Options:
Run the build with werft instead of GHA
Run Leeway with
--dont-testPublish Options
Installer Options
Add desired feature flags to the end of the line above, space separated
Preview Environment Options:
If enabled this will build
install/previewIf enabled this will create the environment on GCE infra
Valid options are
all,workspace,webapp,ide,jetbrains,vscode,ssh