Project Overview

SelfHeal is a Kubernetes self-healing demo. It receives Prometheus Alertmanager webhooks, diagnoses simulated infrastructure failures, and executes bounded corrective actions such as restarting pods, scaling deployments, or rolling back unhealthy releases.

The platform validates recovery after each action, writes an append-only JSONL audit log, exposes Prometheus metrics at /metrics, and can send optional Slack notifications. It also runs locally in dry-run mode before touching a cluster.

What It Does

• Detects Kubernetes failures from Alertmanager payloads.
• Restarts crash-looping, pending, or unready pods.
• Scales overloaded deployments up to a configurable replica cap.
• Rolls back unhealthy deployments.
• Validates recovery after each action.
• Records every alert result in an append-only JSONL audit log.
• Exposes operational metrics at /metrics.
• Sends optional Slack notifications.

Architecture

Kubernetes workloads -> Prometheus -> Alertmanager -> SelfHeal /alerts
                                                    |-> Kubernetes API
                                                    |-> Audit log
                                                    |-> /metrics
                                                    |-> Slack

Technology Stack

• Python for the webhook service, diagnosis, action execution, and validation.
• Kubernetes API for bounded pod, deployment, and rollout operations.
• Prometheus and Alertmanager for monitoring and alert delivery.
• Docker and Kubernetes manifests for deployment.
• Slack webhooks for optional recovery notifications.

Safety Model

SelfHeal keeps automation bounded by design:

• Dry-run mode is the local default.
• Namespace filters control where mutations are allowed.
• Repeated actions are throttled by a cooldown.
• Restart attempts are capped.
• Replica scaling has a configurable maximum.
• Every alert result is audited and exported as metrics.

Local Dry Run

$env:PYTHONPATH = "src"
$env:SELFHEAL_DRY_RUN = "true"
python -m selfheal dry-run-alert examples/alertmanager-payload.json

The project also includes a webhook mode, Kubernetes deployment manifests, monitoring resources, a Grafana starter dashboard, demo workloads, and guided real-world demo scripts.

Demo Results

The demo includes a sample website and a guided test menu for simulating a broken Kubernetes pod. In the demonstrated flow, Prometheus and Alertmanager deliver the failure alert, SelfHeal restarts the simulated broken pod, and Kubernetes creates a healthy replacement.

Configuration

SelfHeal is configured through environment variables including SELFHEAL_DRY_RUN, SELFHEAL_NAMESPACE_ALLOWLIST, SELFHEAL_IGNORED_NAMESPACES, SELFHEAL_MAX_RESTART_ATTEMPTS, SELFHEAL_MAX_REPLICAS, and SELFHEAL_AUDIT_LOG_PATH.

Project Layout

src/selfheal/          Python package
k8s/                   Kubernetes deployment, RBAC, rules, and routing
helm-values/           kube-prometheus-stack values
grafana/               Starter dashboard
examples/              Alertmanager sample payloads
docs/                  Architecture and operations notes
tests/                 Unit tests