[obs] re-enable regular not active alerts #18341
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kylos101
changed the title
workspace_activity_total
kylos101
force-pushed
the
kylos101/fix-active-rule
branch
4 times, most recently
from
68fe33a to
d93cad6
Compare
kylos101
changed the title
workspace_activity_total
easyCZ
approved these changes
Jul 26, 2023
kylos101
force-pushed
the
kylos101/fix-active-rule
branch
2 times, most recently
from
d208291 to
b552177
Compare
|
Hey @easyCZ , could I ask for a follow-up review? I made a few changes. Yesterday's incident gave us a nice opportunity test if these underlying alerts would have triggered. 🙂 |
easyCZ
approved these changes
Jul 28, 2023
Related to ENG-15 Now that we have related data, we should resume triggering alerts if the data condition occurs.
Was getting 404
kylos101
force-pushed
the
kylos101/fix-active-rule
branch
from
750321c to
d40f2c3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Now that
gitpod_workspace_regular_not_active_percentage_mk2is fixed, re-enable regular not active alerts.Additionally, given yesterday's incident, we had data to test and fix the related alert expressions.
Depends on https://github.com/gitpod-io/runbooks/pull/418
Changes summary and walkthrough generated by Copilot
Related Issue(s)
Fixes ENG-20
How to test
Here you can see the recording rule is working again (it no longer has a value of 1), after I ran the enable alerts job on both workspace clusters.
Here you can see how the updated
GitpodWorkspaceTooManyRegularNotActiveMk2alert (f608648) would have caught the incident (this is me forwardingus102's prometheus to confirm the expression):Similar corrections to
GitpodWorkspacesNotStartingMk2would have also helped catch the incident from July 26, too.Documentation
Preview status
Gitpod was successfully deployed to your preview environment.
Build Options
Build
Run the build with werft instead of GHA
Run Leeway with
--dont-testPublish
Installer
Add desired feature flags to the end of the line above, space separated
Preview Environment
If enabled this will build
install/previewIf enabled this will create the environment on GCE infra
Valid options are
all,workspace,webapp,ide,jetbrains,vscode,ssh/hold