Monitoring your Azure Container Services (AKS) cluster with Grafana and InfluxDB

With any compute platform, whether it’s on premise, or cloud based, it is essential that we have a capability to monitor how our application is performing on that platform. The same applies with Kubernetes.

With Azure Container Services (AKS), Microsoft’s go to monitoring solution is Log Analytics, which is part of OMS. With this approach, you install a monitoring agent via a Kubernetes Daemonset, so that the monitoring agent runs on every node within the cluster. Now this implementation works well, and will probably satisfy most users requirements, but it’s always nice to have choice in life.

Another approach is to use Heapster. Heapster aggregates monitoring data from each node in the cluster and then writes the monitoring data to a configurable backend store. Heapster runs as a single pod in the cluster, and queries the Kubelet of each node within the cluster for the node’s monitoring data. You’ll find that the Heapster pod and its associated service are already running within the AKS cluster. To find It you can run a Kubelet get pods, but providing the kube-system namespace in the query so:

get pods --namespace=kube-system

One of Heapster’s supported backend stores is InfluxDB. InfluxDB with Grafana as a front end provides us with a rich visualizer for the captured monitoring data

Now setting up InfluxDB and Grafana is straight forward, we just run them as pods on our cluster. Installation instructions can be found here, and the deployment files can be found here

If you decide not to remove the existing Heapster deployment within your cluster, then you’ll at least need to update it to include a sink to InfluxDB, otherwise the InfluxDB k8s database won’t be created, and the monitoring data won’t be persisted. The fragment below shows the relevant entry for the InfluxDB sink.

- --source=kubernetes:https://kubernetes.default

- --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086

You can now go to the IP address of your Grafana service and see how your cluster is performing.