Monitoring Cassandra with Grafana and Influx DB

Posted in: Cassandra, Technical Track

Hello,

In this post I will explain how to set up Cassandra monitoring with influxDB and Grafana. This can also be used to connect to other monitoring systems (Graphite, Collectd, etc…) but since both influxDB and Grafana are hot topics at the moment I decided to follow the trend! I was asked why I was doing this when a tool like OpsCenter is available, but sometimes you want to have all your systems reporting to a single dashboard. And if your dashboard is Grafana and your Backend is influxDB then you will learn how to connect Cassandra to it!

Assumptions:
– You are running a Linux system (This post is based on CentOS 7)
– You are using Cassandra 1.2+ (I’m using 2.1.3 in this case)

Prerequisites

  • Cassandra Installation
  • Graphite Metrics Jar
  • influxDB – https://influxdb.com/
  • Grafana – https://grafana.org/
  • Apache (Any webserver would do)

Installing and configure influxDB

This one is dead easy, once you have the package install it (rpm -i, dpkg -i). Start the service:

service influxdb start

Once the service is running, go to the configuration (/opt/influxdb/shared/config.toml) and edit the file so that under [input_plugins] it looks like this:

# Configure the graphite api
[input_plugins.graphite]
enabled = true
# address = "0.0.0.0" # If not set, is actually set to bind-address.
port = 2003
database = "cassandra-metrics" # store graphite data in this database
udp_enabled = true

Save the file, reload the service:

service influxdb reload

Now go to your browser localhost:8083, click connect (no credentials should be needed), and after you logged in, enter in a database name (use cassandra-metrics) and click Create (This should be your only option). Now you can click the database, and add an user to it (and make it admin). Now create another database, with name “grafana”, create an admin for that database also.
Now you are done with influxDB.

Installing Grafana

Grafana is a bit more tricky, since it is needed to configure a webserver also. Let’s assume apache is installed, and the home directory for www is /var/www/html.

So get the grafana package and extract it to /var/www/html. So the end result should be something like /var/www/html/grafana.

Now do the following:

cd /var/www/html/grafana
cp config.sample.js config.js

Now let’s configure the connection between influXDB and Grafana. Open for edit the new copied file config.js and edit it so it looks like this:

datasources: {
  influxdb: {
    type: 'influxdb',
    url: "https://localhost:8086/db/cassandra-metrics",
    username: 'admin',
    password: 'admin',
  },
  grafana: {
    type: 'influxdb',
    url: "https://localhost:8086/db/grafana",
    username: 'admin',
    password: 'admin',
    grafanaDB: true
  },
},

Now redirect your browser to localhost/grafana and you will have the Grafana default dashboard.

Preparing Cassandra

Now the final piece of the puzzle. Now we follow more or less the Cassandra guide that exists here, but we need to make some changes to make it more valuable (and allow multiple nodes to provide metrics).

So, first of all, put the metrics-graphite-2.2.0.jar in all the Cassandra nodes /lib directory.
Now create a yaml file with similar to the Datastax example, lets call it influx-reporting.yaml and store it on /conf directory. Now edit the file again so it looks like this:

graphite:
-
  period: 60
  timeunit: 'SECONDS'
  prefix: 'Node1'
  hosts:
  - host: 'localhost'
    port: 2003
  predicate:
    color: "white"
    useQualifiedName: true
    patterns:
    - ".*"

What did we change here, first we added a prefix field, this will allow us to identify the node that is providing the metrics. It must be different for every node, otherwise the metrics will overwrite/mix with each other. Then we decided to allow all patterns (“.*”), this means that Cassandra will push out all the metrics into influxDB. You can decide whether or not this is too much and just enable the metrics you want (find out more about it here).

Now edit the cassandra-env.sh so that it will read the yaml file to provide the metrics. Add the following line to the end of the file:

JVM_OPTS="$JVM_OPTS -Dcassandra.metricsReporterConfigFile=influx-reporting.yaml"

If all is done correctly, you can restart the Cassandra node (or nodes, but don’t do it all at the same time, 2min between each is recommended) and if the log file has the following message:

INFO [main] YYYY-MM-DD HH:MM:SS,SSS CassandraDaemon.java:353 - Trying to load metrics-reporter-config from file: inf
lux-reporting.yaml
INFO [main] YYYY-MM-DD HH:MM:SS,SSS GraphiteReporterConfig.java:68 - Enabling GraphiteReporter to localhost:2003

All is good!

Graphing!

Grafana is not that difficult to use, so once you start exploring a bit (And reading the documentation) you will find out doing nice graphs. This could be a long post only about graphing out, so I’m just go and post some images of the graphs I’m getting out of Grafana so that you can see how it can be powerful and help you on keeping your Cassandra Healthy.

Grafana_cassandra-test3

Grafana_cassandra-test2

Grafana_cassandra-test1

 

Want to Learn more about Pythian’s Cassandra Services?

email

Interested in working with Carlos? Schedule a tech call.

About the Author

Carlos Rolo is a Datastax Certified Cassandra Architect, and has deep expertise with distributed architecture technologies. Carlos is driven by challenge, and enjoys the opportunities to discover new things and new ways of learning that come with working at Pythian. He has become known and trusted by customers and colleagues for his ability to understand complex problems, and to work well under pressure. He prides himself on being a tenacious problem solver, while remaining a calm and positive presence on any team. When Carlos isn’t working he can be found playing water polo or enjoying the his local community. Carlos holds a Bachelor of Electro-technical Engineering, and a Master of Control Systems and Automation.

7 Comments. Leave new

Great post. Hopefully you’ll enjoy Grafana 2.0 when it comes out — it’s a single binary with no dependencies, so the set up is tons easier :)

Reply

Useful post, thanks !

Have you been able to display graphs dynamically in Grafana ? That is instead of having to define a query for each node, you would use a generic one that display the selected metric for all the nodes (or the ones selected by the user) ?

I have been trying to do that unsuccessfully as shown in this screenshot : https://postimg.org/image/duo6p6kox/

I’m using a query like :
SELECT mean(“value”) FROM /$node.org.apache.cassandra.metrics.Compaction.TotalCompactionsCompleted.count/
WHERE $timeFilter
GROUP BY time($interval) fill(null)
ALIAS BY $node
but the ALIAS BY is not what I want when $node matches multiple nodes as in the screenshot.

Cheers

Joel

Reply

I tried the same way you have suggested but the Cassandra is not giving the Inputs to the Influxdb. I was not able to find any graphs or measurements in the influxdb server.

Reply

I believe the issue is that this tutorial seems to suggest that Graphite is not required, but it is. The Cassandra config file shows that something must be listening on port 2003, but this article does not cover mention that Graphite must be installed and listening on this port.

Reply

I had to use metrics-graphite-3.1.2.jar with Cassandra 2.2 if that can help.

Reply

Thanks for reply,
I have tried the same but unable to find the following in the C*ndra log
“INFO [main] YYYY-MM-DD HH:MM:SS,SSS CassandraDaemon.java:353 – Trying to load metrics-reporter-config from file: inf
lux-reporting.yaml”

Reply

I would double check cassandra-env.sh to see if you set -Dcassandra.metricsReporterConfigFile option.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *