Custom Notifications with Alert Manager’s Webhook Receiver in Kubernetes

Prometheus’s AlertManager receives the alerts send from Prometheus’ alerting rules, and then manages them accordingly. One of the action is to send out external notifications such as Email, SMS, Chats. Out of box, AlertManager provides a rich set of integrations for the notifications. However, in real life projects, these lists are always not enough. For my case, I need to send out email through Gmail and SMS through Twilo.

Quoted from AlertManager’s documentation

We’re not actively adding new receivers, we recommend implementing custom notification integrations via the webhook receiver

Fair enough, I will create my own webhook receiver for Gmail and Twilo.

The Kubernetes here used is the one with IBM Cloud Private V 2.1.

1. Configure Receiver in Prometheus

First let’s define the alertmanager’s configuration as a configmap in Kubernetes, and apply it through the kubectl command line.

apiVersion: v1
kind: ConfigMap
metadata:
  name: monitoring-prometheus-alertmanager
  namespace: kube-system
data:
  alertmanager.yml: |-
    global:
    receivers:
      - name: default-receiver
      - name: ycap-webhook
        webhook_configs:
          - url: "http://am-webhook.ycap/webhook"
    route:
      group_wait: 10s
      group_interval: 5m
      receiver: default-receiver
      repeat_interval: 3h
      
      routes: 
        - receiver: ycap-webhook
          match_re:
            app: ycap

Other than the default-receiver, I defined my ycap-webhook receiver, which is using webhook-configs and the URL is pointing to http://am-webhook.ycap/webhook The hostname am-webhook.ycap is the Kubernetes’ service name that I am going to create later together with the custom webhook Deployment in the namespace of ycap .

If the alert regex matches the label of app with the value of ycap , it will be routed to the ycap-webhook receiver to handle.

2. Alert Rule

Secondly I define the alert rules to generate alert. Create the following configmap and apply it with kubectl similarly.