Custom Notifications with Alert Manager’s Webhook Receiver in Kubernetes

Prometheus’s AlertManager receives the alerts send from Prometheus’ alerting rules, and then manages them accordingly. One of the action is to send out external notifications such as Email, SMS, Chats. Out of box, AlertManager provides a rich set of integrations for the notifications. However, in real life projects, these lists are always not enough. For my case, I need to send out email through Gmail and SMS through Twilo.

Quoted from AlertManager’s documentation

We’re not actively adding new receivers, we recommend implementing custom notification integrations via the webhook receiver

Fair enough, I will create my own webhook receiver for Gmail and Twilo.

The Kubernetes here used is the one with IBM Cloud Private V 2.1.

1. Configure Receiver in Prometheus

First let’s define the alertmanager’s configuration as a configmap in Kubernetes, and apply it through the kubectl command line.

apiVersion: v1
kind: ConfigMap
metadata:
  name: monitoring-prometheus-alertmanager
  namespace: kube-system
data:
  alertmanager.yml: |-
    global:
    receivers:
      - name: default-receiver
      - name: ycap-webhook
        webhook_configs:
          - url…