Warn when IPFS is running out of storage (#18386)

* Warn when IPFS is running out of storage

* Add critical alert

Author: WVerlaek

operations/observability/mixins/workspace/rules/central/ipfs.yaml

@@ -0,0 +1,35 @@
+# Copyright (c) 2023 Gitpod GmbH. All rights reserved.
+# Licensed under the GNU Affero General Public License (AGPL).
+# See License.AGPL.txt in the project root for license information.
+
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  labels:
+    prometheus: k8s
+    role: alert-rules
+  name: ipfs-monitoring-rules
+spec:
+  groups:
+    - name: ipfs-alerts
+      rules:
+        - alert: IPFSStorageLow
+          labels:
+            severity: warning
+            team: workspace
+          for: 10m
+          annotations:
+            runbook_url: https://github.com/gitpod-io/runbooks/blob/main/runbooks/IPFSAlmostOutOfStorage.md
+            summary: IPFS is running out of storage within a few days in a workspace cluster. Create and shift to a new workspace cluster with a fresh IPFS.
+            description: IPFS in cluster {{ $labels.cluster }} is running out of storage. This happens naturally for workspace clusters over time, and is prevented by recreating the clusters regularly. Once out of storage, IPFS will stop working, and we fall back to the GCP registry for workspace image pulls, resulting in slower workspace startup times.
+          expr: min(node_filesystem_avail_bytes{device="/dev/mapper/lvm--disk-ipfs", node=~"services-.*", cluster!~"ephemeral.*"}) by (cluster) < 600*1024*1024*1024
+        - alert: IPFSStorageCritical
+          labels:
+            severity: critical
+            team: workspace
+          for: 10m
+          annotations:
+            runbook_url: https://github.com/gitpod-io/runbooks/blob/main/runbooks/IPFSAlmostOutOfStorage.md
+            summary: IPFS is now very close to running out of storage in a workspace cluster. Create and shift to a new workspace cluster with a fresh IPFS.
+            description: IPFS in cluster {{ $labels.cluster }} has almost ran out of storage. This happens naturally for workspace clusters over time, and is prevented by recreating the clusters regularly. Once out of storage, IPFS will stop working, and we fall back to the GCP registry for workspace image pulls, resulting in slower workspace startup times.
+          expr: min(node_filesystem_avail_bytes{device="/dev/mapper/lvm--disk-ipfs", node=~"services-.*", cluster!~"ephemeral.*"}) by (cluster) < 200*1024*1024*1024