Skip to content

Expose docker API events to prometheus

License

Notifications You must be signed in to change notification settings

sbadia/docker-events-exporter

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Docker events exporter

Docker events exporter expose docker API events (oom, start, …) to prometheus metrics.

Summary

node-problem-detector doesn't expose the pod name, but only show events using logs files. In case of OOM is not very helpful, Docker events exporter use docker API to retrieve pod, container and namespace.

Run on kubernetes as daemonsets

You can apply this ds on our cluster, in order to expose API events as prometheus metrics.

{
  "apiVersion": "extensions/v1beta1",
  "kind": "DaemonSet",
  "metadata": {
    "name": "events-notifier",
    "namespace": "inf"
  },
  "spec": {
    "updateStrategy": {
      "type": "RollingUpdate"
    },
    "template": {
      "metadata": {
        "labels": {
          "name": "events-notifier"
        },
        "annotations": {
          "prometheus.io/scrape": "true",
          "prometheus.io/port": "9000"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "events-notifier",
            "image": "clustree:docker-events-exporter",
            "imagePullPolicy": "Always",
            "resources": {
              "requests": {
                "cpu": "0.1"
              },
              "limits": {
                "memory": "20Mi",
                "cpu": "0.3"
             }
            },
            "volumeMounts": [
              {
                "name": "dockersock",
                "mountPath": "/var/run/docker.sock"
              }
            ],
            "ports": [
              {
                "name": "prometheus",
                "containerPort": 9000
              }
            ]
          }
        ],
        "volumes": [
          {
            "name": "dockersock",
            "hostPath": {
              "path": "/var/run/docker.sock"
            }
          }
        ]
      }
    }
  }
}

And then fetch events on port 9000

❯ curl 10.252.8.147:9000
# HELP docker_events Docker events
# TYPE docker_events counter
docker_events{env="int2",event="exec_start: bash",pod="ldap"} 1.0
docker_events{env="int2",event="oom",pod="ldap-2246486038-30nc1"} 1.0

Prometheus alerts ?

Then you can imagine to configure prometheus alerts based on thoses metrics, for example about OOM events…

groups:
- name: host_health
  - alert: oom
    expr: rate(docker_events{event="oom",kubernetes_namespace="inf"}[1m]) > 0
    labels:
      routing: slackonly
    annotations:
      link: '{{ $labels.event }} - {{ $labels.env }} - {{ $labels.pod }}'

Releases

No releases published

Packages

No packages published

Languages