A closer look at two popular data visualization tools and tips for choosing the best option for you
Despite similar sounding names, Kibana and Grafana take very different approaches to the data they visualize. In this post, I’ll walk you through their main differences and show which tool is best suited for your specific needs.
— OverOps (@overopshq) November 15, 2017
I found it slightly complicated to install and setup Kibana, but if you are already using any components in the ELK (Elasticsearch, Logstash, Kibana) stack then you have already conquered these hurdles. For experimentation purposes, I recommend using Docker. This docker-compose.yml file should get you started, start the service with docker-compose up.
Then follow Elastic’s official data import guide to load data for experimentation.
Grafana offers a cloud-hosted option, or you can run it yourself, with Grafana available in all Linux package managers, Homebrew for Mac, and as a Docker image. I used the Docker option, by running:
Open your browser to http://localhost:3000/login and log in, the default account details are ‘admin’ for both the username and password.
Grafana and Kibana have two well-defined, yet different, directions for visualizing data, and they reflect this in the sources you can pull data from.
Kibana focuses on helping you search, explore and analyze log data stored in Elasticsearch, and that’s the only source it supports, if you’re not using Elasticsearch then there’s no point using it. If you are, then its integration is tight and well developed.
Grafana offers inbuilt methods for connecting to over 30 data sources, including Elasticsearch, and focuses on visualizing time-series metric data, which can be logging data, but is better suited to visualizing data from constant streaming sources, such as sensors and metric reporting.
Before filtering and querying your log data with Kibana, you need to configure indices of your data, in the long term this will speed up your experiences with Kibana.
Once this is complete, Kibana has two options for searching your data, the more standard Lucene query syntax or the Elasticsearch query DSL. They are both powerful and broad but require a steep learning curve if you’ve not used them before. On the positive side, many other applications use the Lucene query syntax.
Grafana offers fewer options for refining the underlying data before you visualize it. However each data source ships with a query editor that suits the source, not a generic language for all.
For example, here’s the query editor for an Elasticsearch data source:
Once you do understand the query languages that Kibana supports, then the charts you can create are complex and detailed and you can save queries to recreate visuals with up-to-date data.
For example, this stacked pie chart groups the quantities of data into the 5 ranges you can see on the top level of the key (right-hand side of picture), and then relates the second ranges on the outer ring to each pie segment. This enables you to see how two sets of data relate to each other.
This chart visualizes log data as geographic locations on a world map, great for identifying where an event occurred.
Grafana tries to make visualizing a chart as straightforward as possible, giving you visuals before you spend a long time creating a dashboard that is not useful to you. For the example below I added a local Elasticsearch data source using the default Shakespeare and log data found in their getting started tutorial and created a ‘heatmap’ and ‘graph’ dashboard. From start to finish this took me less than a minute.
This heatmap shows the amount of log data in each time frame.
You can change aspects of a dashboard by clicking the cog icon in the Grafana top bar, for example, the title, or a description, and add more than one row to a dashboard to display more than one representation of information.
Access and Permissions
Unless you sign up for Elastic’s X-Pack plugin, then all dashboards and visualizations in Kibana are public to anyone with access to the service. Again, it offers a lot of flexibility and configuration options, but takes time to understand what’s possible and set up what you need. If you used the Docker option I mentioned above, then you have X-Pack preinstalled.
You can only configure aspects of X-Pack with the Kibana kibana.yml file and Elastic considers it to still be in development, with somewhat confusing documentation as you try to figure out what’s possible and how. This is possibly due to X-Pack being a consolidation of older plugins with new functionality, and so, is still incohesive. You can restrict types of access to management operations, indices and fields, encrypt data from the top down and maintain an access log.
For example, this role restricts members of the ShakesTeam to read-only access to the shakes* index.
Grafana ships with role-based access, but it’s much simpler than what Kibana offers. You create different ‘organizations’, that you can use to create groups and teams within a company, and add users to these. Each user can have a different role, and these roles offer broad permissions, but you can’t restrict access to individual dashboards or charts, you have to break these out into organizations first.
This same level of access also applies to API access to Grafana.
It’s hard to tell how old each project is, they both have a similar number of releases, with Grafana firmly beating Kibana for most GitHub statistics (never a reliable indication of community, but a start). Grafana also has a lot less StackOverflow questions, but it’s hard to tell if this is due to users having less problems with it, or if each platform’s own forums and support channels handle their efforts differently.
For those of you that are more interested in the business side, then Elastic appears to be better resourced, with much larger community activity in meetups, events, conference appearances and training platforms. I personally have also seen far more conference presentations in aspects of the ELK stack than Grafana, but this could be due to Kibana being a more complex and broad application.
Grafana, Kibana and OverOps
Returning to data sources, one of the interesting new features OverOps has recently introduced is an enhanced StatsD integration which allows users to publish metrics to both platforms and more. It lets you export time series data for any exception, logged warning, or error, through finely grained views that go down up to a specific event, through individual applications, servers, classes or even methods:
And here’s how it looks in Grafana, for example:
For each of these events, OverOps lets you dive deep and get the complete source code and variable state across the entire call stack of each error. It automated the root cause analysis of each error and shows you the exact state that caused it, without having to sift through log files and search for clues:
In this article, I highlighted the main differences between Kibana and Grafana and showed you how and when you might want to use each of their feature set(s). In large applications or companies, you will use them together in conjunction with multiple data sources and visualization needs. In these cases, and no matter the source, OverOps will help you dig beneath the information you visualize and act on it, providing a better experience for all your customers and users.