Enhance your Kubernetes Monitoring with Grafana

Created with Sketch.

Enhance your Kubernetes Monitoring with Grafana

So, why would you want to leverage Grafana to visualise your kubernetes metrics?

Grafana is the world’s most popular way to create dashboards. This enables us to query, visualise, alert on and understand our metrics no matter where they’re stored. Grafana can connect to a plethora of data sources; Prometheus, Graphite, and AWS CloudWatch are just a few examples. You can create, explore, and share highly customised dashboards for almost any use case. It’s also open-source, meaning considerable cost saving when compared to proprietary alternatives.

This post is a whistle stop tour of Grafana. We cover the Installation all the way through to some best practices. We’ve not designed this to be a complete guide to teach you all things Grafana. However, we’ve designed it to give you all the information you need to get immediate value.

Installation

Installing Grafana on Kubernetes could be accomplished by defining the resources yourself using YAML, you could also use a Third Party provider to create and manage your Grafana resources, or do this via Helm. As with most applications running on Kubernetes the quickest and simplest approach is to leverage Helm. The Grafana Helm chart allows us to fully install and configure Grafana with just a couple of commands.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install grafana grafana/grafana

Keep in mind that the default Helm installation doesn’t persist any data. Any data will be lost when the Grafana pods are terminated.

To enable persistent storage, first create the volume ready for the persistent volume claim that the Helm chart will create.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  hostPath:
    path: /mnt/grafana
  accessModes:
    - ReadWriteOnce

Then install Grafana to Kubernetes with the following command:

helm install grafana grafana/grafana --set persistence.enabled=true

If you have a storage class already setup you may want to set the persistence.storageClassName to your storage class value.

You can also change the size of the claim with the persistence.size parameter. You can find all available parameters here.

Dashboard Access

There are many different ways to gain access to Grafana. You can securely expose Grafana using your preferred Kubernetes Ingress Controller. This will allow secure access to multiple teams within your organisation. We are going to look at how to leverage kubectl port-forward to gain access from outside of the Kubernetes cluster to Grafana, this will allow us to quickly validate our installation.

Get your ‘admin’ user password by running:

kubectl get secret grafana -o jsonpath="{.data.admin-password}" | base64 --decode

Get the Grafana pod name and setup port forwarding by running these commands in the same shell:

export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward $POD_NAME 3000

You should now be able to access Grafana on http://localhost:3000 and login with the admin username and password.

Data Sources

Grafana supports many different data sources, each data source has a specific Query Editor. Out of the box you get the Core Data Sources and a few bundled enterprise plugins, however you can extend these by installing data source plugins.

To configure a new data source, navigate to Configuration -> Data Sources. From here you’ll be able to configure all of your available data sources. Once done and tested, we can start creating some useful Kubernetes dashboards!

Dashboards

Dashboards are panels that’s organized and arranged into one or more rows. Grafana ships with a variety of Panels. You can also add more by installing them as plugins.

Lets get started by creating a simple panel displaying the number of running pods in our Kubernetes cluster.

  • In the side bar, hover your cursor over the Create (+) icon and then select Dashboard.
  • Click Add new panel.
  • In the Query editor below the graph, enter the below query, then press Shift + Enter:
sum(kube_pod_status_phase{namespace=~".*", phase="Running"})
  • In the Panel section, set the Panel title to Running Pods, then select Stat under Visualization.
kubernetes running pods grafana

This will give you a simple panel which clearly displays the number of running pods within Kubernetes. Feel free to explore your available Kubernetes metrics within Grafana and start adding relevant panels. Once you’re happy, save and share your dashboard!

You can import various useful Kubernetes dashboards from grafana.com

Visualisation Types

Below are a few visualisation types with a brief description and use case, these change how your panels look and feel within your dashboards.

  • Single Stat: Displays a single stat metric, such as the above example displaying how many pods are currently running.
  • Gauge: Gauge is a single value panel. You can set limits to display if it’s in an acceptable range.
  • Graph: Used to graph metrics over time. You can use Graph for most scenarios.
  • Single Stat with status: Similar functionality to Gauge, used to display numeric metric and whether it’s in an acceptable range.
  • Worldmap: Displays a world map with plot points. This can be useful for displaying the location of clients.

Variables

Variables in Grafana allow you to build interactive dashboards. You can use variables in queries and in panel titles. Variables are displayed as drop down lists, making it extremely efficient to change the data being displayed.

Before queries are sent to your data source the query is interpolated, the formatting of the variable interpolation depends on the data source.

Advanced variable format options can be found in the Grafana Documentation.

Alerting

Alerts allow you to identify problems in your systems. This allows you to respond to indications of a potential incident before an outage occurs.

Grafana Alerts consists of two parts.

  • Rules: Alert rules are defined by one or more conditions.
  • Notification channel: When the conditions of an alert rule are met, the configured channels are notified.

When an alert changes state, it sends out notifications. Each alert rule can have multiple notifications. Grafana’s alerting functionality only works for graph panels with time-series output.

Remember, don’t alert too frequently as this creates alert fatigue. This could result in alerts not being treated with the scrutiny they require.

Add new alerts when needed, outage postmortems should show a clear timeline. If alerts aren’t fired quick enough, you may need to find a better metric to alert on. This gives you those invaluable moments to react before an outage.

Some Best Practices

Below are a few best practices when creating Grafana Dashboards.

  • Make your dashboards easy to interpret, this will help other users as well as your future self!
  • Keep your graphs simple and focused on displaying helpful data. Try not to make them convoluted with unnecessary panels.
  • Avoid excessive dashboard refreshing, this will help reduce backend load.
  • Try to avoid stacking graph data as this can hide important data.

Conclusion

Grafana is most commonly used for a good reason. It’s open source, has fantastic compatibility, it’s highly customisable and enables teams to visually share poignant metrics immediately.

Grafana is perfect for visualising Kubernetes metrics. As Prometheus is natively supported by Kubernetes, it makes perfect sense to leverage both Prometheus and Grafana to monitor and visualise your Kubernetes workloads.

 

No Comments

Add your comment