Merge pull request #41 from robusta-dev/more-alert-enrichers-and-docs

More alert enrichers and docs

Author: arikalon1

docs/images/graph-enricher.png

@@ -4,7 +4,7 @@ Prometheus Alert Enrichment
 ##################################
 
 Introduction
-^^^^^^^^^^^^^^^
+--------------
 Robusta has special features for handling Prometheus alerts in Kubernetes clusters including:
 
 1. **Enrichers:** playbooks that enrich alerts with extra information based on the alert type
@@ -17,38 +17,34 @@ These features are still in beta and therefore have been implemented differently
 of operation, you configure a root ``alerts_integration`` playbook in ``active_playbooks.yaml`` and then add special enrichment
 and silencer playbooks underneath that playbook. In the future, this functionality will likely be merged into regular playbooks.
 
-Setup and configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Configure Robusta
+---------------------------------
 
-Configure Prometheus AlertManager
-----------------------------------
-Before you can enrich prometheus alerts, you must forward Prometheus alerts to Robusta by adding a webhook receiver to AlertsManager.
-See :ref:`Setting up the webhook` for details.
+.. admonition:: Configure Prometheus AlertManager
 
-Configure Robusta
-------------------------------
-Lets look at the simplest possible ``active_playbooks.yaml`` which instructs Robusta to forward Prometheus alerts to Slack without any enrichment:
+    Before you can enrich prometheus alerts, you must forward Prometheus alerts to Robusta by adding a webhook receiver to AlertsManager.
 
-| **Enabling it:**
+    See :ref:`Setting up the webhook` for details.
+
+
+Lets look at the simplest possible ``active_playbooks.yaml`` which instructs Robusta to forward Prometheus alerts to Slack without any enrichment:
 
 .. code-block:: yaml
 
    active_playbooks:
   - name: "alerts_integration"
 
 The above configuration isn't very useful because we haven't enriched any alerts yet.
-However, we do get a minor aesthetic benefit because Robusta adds pretty formatting to alerts as you can see below:
+However, Robusta still sends default information for every alert as you can see below.
 
 .. image:: /images/default-slack-enrichment.png
   :width: 30 %
   :align: center
 
 Adding an Enricher
--------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Now lets add an enricher to ``active_playbooks.yaml`` which enriches the ``HostHighCPULoad`` alert:
 
-| **Enabling it:**
-
 .. code-block:: yaml
 
    active_playbooks:
@@ -78,7 +74,7 @@ Therefore, in the above example, we explicitly added back the ``AlertDefaults``
 Make sure to check out the full list of enrichers to see what you can add.
 
 Setting the default enricher
-------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 You can change the default enricher(s) for all alerts using the ``default_enrichers`` parameter.
 
@@ -91,10 +87,8 @@ You can change the default enricher(s) for all alerts using the ``default_enrich
         - name: "AlertDefaults"
 
 Adding a Silencer
------------------
-Now lets look at an example ``active_playbooks.yaml`` which silences KubePodCrashLooping alerts in the first ten minutes after a node (re)starts:
-
-| **Enabling it:**
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Lets silence `KubePodCrashLooping` alerts in the first ten minutes after a node (re)starts:
 
 .. code-block:: yaml
 
@@ -109,8 +103,8 @@ Now lets look at an example ``active_playbooks.yaml`` which silences KubePodCras
             post_restart_silence: 600 # seconds
 
 Full example
-----------------
-Here is an example which shows all the features discussed above working together:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Here are all the above features working together:
 
 .. code-block:: yaml
 
@@ -133,108 +127,179 @@ Here is an example which shows all the features discussed above working together
           params:
             post_restart_silence: 600 # seconds
 
-Available enrichers
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Available Enrichers
+-----------------------
 
-**AlertDefaults:** send the alert message and labels to Slack
+AlertDefaults
+^^^^^^^^^^^^^^^^
+Send the alert message and labels to Slack
 
-**NodeCPUAnalysis:** provide deep analysis of node cpu usage
-This enricher use ``prometheus``. The ``prometheus`` url can be overriden in the ``global_config`` section.
-For example - ``prometheus_url: "http://prometheus-k8s.monitoring.svc.cluster.local:9090"``
+NodeCPUAnalysis
+^^^^^^^^^^^^^^^^^^^^^
+Provide analysis of node cpu usage.
 
-**OOMKillerEnricher:** shows which pods were recently OOM Killed on a node
+.. note::
+    This enricher use ``prometheus``. The ``prometheus`` url can be overriden in the ``global_config`` section.
 
-**GraphEnricher:** display a graph of the Prometheus query which triggered the alert
-This enricher use ``prometheus``. The ``prometheus`` url can be overriden in the ``global_config`` section.
-For example - ``prometheus_url: "http://prometheus-k8s.monitoring.svc.cluster.local:9090"``
+    For example - ``prometheus_url: "http://prometheus-k8s.monitoring.svc.cluster.local:9090"``
 
-**StackOverflowEnricher:** add a button in Slack to search for the alert name on StackOverflow
+GraphEnricher
+^^^^^^^^^^^^^^^^^^^^^
+Display a graph of the Prometheus query which triggered the alert.
 
-**NodeRunningPodsEnricher:** add a list of the pods running on the node, with the pod Ready status
+`See note above regarding the prometheus_url parameter.`
 
-.. image:: /images/node-running-pods.png
-  :width: 80 %
-  :align: center
+.. admonition:: Example
 
-**NodeAllocatableResourcesEnricher:** add the allocatable resources available on the node
+    .. image:: /images/graph-enricher.png
+      :width: 50 %
+      :align: center
 
-.. image:: /images/node-allocatable-resources.png
-  :width: 80 %
-  :align: center
+TemplateEnricher
+^^^^^^^^^^^^^^^^^^^^^
+Add a paragraph to the alert's description containing templated markdown. You can inject any of the alert's Prometheus labels into the markdown.
 
-**DaemonsetEnricher:** for daemonset related alerts, adds details about the daemonset status
+A variable like ``$foo`` will be replaced by the value of the Prometheus label ``foo``. If a label isn't present then the text "<missing>" will be used instead.
 
-.. image:: /images/daemonset-enricher.png
-  :width: 80 %
-  :align: center
+Common variables to use are ``$alertname``, ``$deployment``, ``$namespace``, and ``$node``
 
-**DaemonsetMisscheduledAnalysis:** analyze the known Prometheus alert ``KubernetesDaemonsetMisscheduled`` and provide
-actionable advice on how to fix it. This enricher **only** displays output when it can verify that the alert is a false
-positive.
+The template can include all markdown directives supported by Slack. Note that Slack markdown links use a different format than GitHub.
 
-.. image:: /images/daemonset-misscheduled.png
+.. admonition:: Example
 
-**PodBashEnricher:** runs the specified bash command, on the **pod** associated with the alert
+    .. code-block:: yaml
 
-| **Note:** The bash command must be installed on the target pod
+       active_playbooks:
+       (...)
+          - alert_name: "ContainerVolumeUsage"
+            enrichers:
+            - name: "TemplateEnricher"
+              params:
+                template: "The alertname is $alertname and the pod is $pod"
 
-| **Example Usage:**
+LogsEnricher
+^^^^^^^^^^^^^^^^^^^^^
+Fetch logs related to the alert and attach them to the alert as a file.
 
-.. code-block:: yaml
+The pod to fetch logs for is determined by the alert's ``pod`` label from Prometheus.
 
-   active_playbooks:
-   (...)
-      - alert_name: "ContainerVolumeUsage"
-        enrichers:
-        - name: "PodBashEnricher"
-          params:
-            bash_command: "df -h"
+By default, if the alert has no label named ``pod`` then this enricher will silently do nothing. To show an explicit error, set the ``warn_on_missing_label`` parameter to ``true``
 
-| **The results:**
+OOMKillerEnricher
+^^^^^^^^^^^^^^^^^^^^^
+Shows which pods were recently OOM Killed on a node
 
-.. image:: /images/disk-usage.png
-  :width: 80 %
-  :align: center
+StackOverflowEnricher
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Add a button in Slack to search for the alert name on StackOverflow
 
-**NodeBashEnricher:** runs the specified bash command, on the **node** associated with the alert
+NodeRunningPodsEnricher
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Add a list of the pods running on the node, with the pod Ready status
 
-| **Example Usage:**
+.. admonition:: Example
 
-.. code-block:: yaml
+    .. image:: /images/node-running-pods.png
+      :width: 80 %
+      :align: center
 
-   active_playbooks:
-   (...)
-      - alert_name: "HostOutOfDiskSpace"
-        enrichers:
-        - name: "NodeBashEnricher"
-          params:
-            bash_command: "df -h"
+NodeAllocatableResourcesEnricher
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Add the allocatable resources available on the node
 
-**DeploymentStatusEnricher:** adds deployment condition statuses
+.. admonition:: Example
 
-| **Example Usage:**
+    .. image:: /images/node-allocatable-resources.png
+      :width: 80 %
+      :align: center
 
-.. code-block:: yaml
+DaemonsetEnricher
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For daemonset related alerts, adds details about the daemonset status
 
-   active_playbooks:
-   (...)
-      - alert_name: "KubernetesDeploymentReplicasMismatch"
-        enrichers:
-        - name: "DeploymentStatusEnricher"
+.. admonition:: Example
 
-| **The results:**
+    .. image:: /images/daemonset-enricher.png
+      :width: 80 %
+      :align: center
 
-.. image:: /images/deployment-status-details.png
-  :width: 100 %
-  :align: center
+DaemonsetMisscheduledAnalysis
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Analyze the known Prometheus alert ``KubernetesDaemonsetMisscheduled`` and provide actionable advice on how to fix it.
+This enricher **only** displays output when it can verify that the alert is a false positive.
+
+.. admonition:: Example
+
+    .. image:: /images/daemonset-misscheduled.png
+
+PodBashEnricher
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Runs the specified bash command, on the **pod** associated with the alert. The bash command must already be installed in the target pod.
+
+.. admonition:: Example
+
+    .. code-block:: yaml
+
+       active_playbooks:
+       (...)
+          - alert_name: "ContainerVolumeUsage"
+            enrichers:
+            - name: "PodBashEnricher"
+              params:
+                bash_command: "df -h"
+
+    .. image:: /images/disk-usage.png
+      :width: 80 %
+      :align: center
+
+NodeBashEnricher
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Runs the specified bash command, on the **node** associated with the alert
+
+.. admonition:: Example
+
+    .. code-block:: yaml
+
+       active_playbooks:
+       (...)
+          - alert_name: "HostOutOfDiskSpace"
+            enrichers:
+            - name: "NodeBashEnricher"
+              params:
+                bash_command: "df -h"
+
+
+DeploymentStatusEnricher
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Adds deployment condition statuses
+
+.. admonition:: Example
+
+    .. code-block:: yaml
+
+       active_playbooks:
+       (...)
+          - alert_name: "KubernetesDeploymentReplicasMismatch"
+            enrichers:
+            - name: "DeploymentStatusEnricher"
+
+    .. image:: /images/deployment-status-details.png
+      :width: 100 %
+      :align: center
 
 Available Silencers
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+-----------------------
+
+NodeRestartSilencer
+^^^^^^^^^^^^^^^^^^^^^^^^^
+After a node is restarted, silence alerts for pods running on it.
+
+.. admonition:: Parameters
 
-**NodeRestartSilencer:** After a node is restarted, silence alerts for pods running on it.
-| params: post_restart_silence, (seconds), default to 300
+    **post_restart_silence**: length of the silencing period in seconds; defaults to 300
 
 
-**DaemonsetMisscheduledSmartSilencer:** Silence the Prometheus alert ``KubernetesDaemonsetMisscheduled`` under
-conditions matching a known false alarm
+DaemonsetMisscheduledSmartSilencer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Silence the Prometheus alert ``KubernetesDaemonsetMisscheduled`` under conditions matching a known false alarm