Skip to content

feat(module): add DVCR capacity alert #1461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(module): add DVCR capacity alert #1461

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 29 additions & 4 deletions monitoring/prometheus-rules/dvcr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
plk_protocol_version: "1"
plk_markup_format: "markdown"
plk_labels_as_annotations: "pod"
plk_create_group_if_not_exists__d8_virtualization_health: "D8VirtualizationHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
plk_grouped_by__d8_virtualization_health: "D8VirtualizationHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
plk_create_group_if_not_exists__d8_virtualization_dvcr_health: "D8VirtualizationDVCRHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
plk_grouped_by__d8_virtualization_dvcr_health: "D8VirtualizationDVCRHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
summary: The dvcr Pod is NOT Ready.
description: |
The recommended course of action:
Expand All @@ -27,10 +27,35 @@
annotations:
plk_protocol_version: "1"
plk_markup_format: "markdown"
plk_create_group_if_not_exists__d8_virtualization_health: "D8VirtualizationHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
plk_grouped_by__d8_virtualization_health: "D8VirtualizationHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
plk_create_group_if_not_exists__d8_virtualization_dvcr_health: "D8VirtualizationDVCRHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
plk_grouped_by__d8_virtualization_dvcr_health: "D8VirtualizationDVCRHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
summary: The dvcr Pod is NOT Running.
description: |
The recommended course of action:
1. Retrieve details of the Deployment: `kubectl -n d8-virtualization describe deploy dvcr`
2. View the status of the Pod and try to figure out why it is not running: `kubectl -n d8-virtualization describe pod -l app=dvcr`

- alert: D8VirtualizationDVCRInsufficientCapacityRisk
expr: |
min by (persistentvolumeclaim, namespace) (
(kubelet_volume_stats_available_bytes{namespace="d8-virtualization", persistentvolumeclaim="dvcr"} < 5 * 1024 * 1024 * 1024)
or
(kubelet_volume_stats_available_bytes{namespace="d8-virtualization", persistentvolumeclaim="dvcr"} / (kubelet_volume_stats_capacity_bytes / 100) < 20)
)
for: 10m
labels:
severity_level: "6"
tier: cluster
annotations:
plk_protocol_version: "1"
plk_markup_format: "markdown"
plk_create_group_if_not_exists__d8_virtualization_dvcr_health: "D8VirtualizationDVCRHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
plk_grouped_by__d8_virtualization_dvcr_health: "D8VirtualizationDVCRHealth,tier=~tier,prometheus=deckhouse,kubernetes=~kubernetes"
summary: Risk of insufficient DVCR storage capacity.
description: |
The DVCR storage (PVC {{ $labels.persistentvolumeclaim }} in namespace {{ $labels.namespace }}) has less than 5GB or 20% free space, risking failure to create new images or disks. Available: {{ $value | humanize }} bytes.

The recommended course of action:

- Check PVC details: `d8 k -n d8-virtualization describe pvc dvcr`
- Extend DVCR storage size in the virtualization module settings or remove unnecessary images (ClusterVirtualImage\VirtualImage).
Loading