Exporting custom metrics to influxdb

Posted in: DevOps, Open Source, Site Reliability Engineering, Technical Track

In the TICK stack from influxdata , Telegraf is used to collect and send data to influxdb.

Metrics in Influxdb can then be viewed using visualization tools like Grafana.

There are several input and output plugins for telegraf, which can be included in the /etc/telegraf/telegraf.conf file to support wide range of metric sources and destinations.

For example, adding swap plugin like below would export swapping information from the host along with a tag name metrics_source.


To demonstrate this, let us use file output plugin with influx as dataformat.

  files = ["stdout", "/tmp/metrics.out"]
  data_format = "influx"

In this post we will focus on exporting metrics from a container host using telegraf container.

Lets create a Dockerfile to include the telegraf.conf file..

FROM telegraf
COPY telegraf.conf /etc/telegraf/telegraf.conf
CMD ["telegraf"]

build the container..

# docker build -t telegrafdemo .</code?

and run it in foreground, so that we can the see the output plugin in work.

# docker run --name telegrafdemo telegrafdemo
[..] swap,metrics_source=telegraf_demo,host=e9e7086036b8 total=8271163392i,used=0i,free=8271163392i,used_percent=0 1514876555000000000
swap,metrics_source=telegraf_demo,host=e9e7086036b8 out=0i,in=0i 1514876555000000000
swap,host=e9e7086036b8,metrics_source=telegraf_demo used_percent=0,total=8271163392i,used=0i,free=8271163392i 1514876560000000000
swap,metrics_source=telegraf_demo,host=e9e7086036b8 in=0i,out=0i 1514876560000000000
swap,metrics_source=telegraf_demo,host=e9e7086036b8 used=0i,free=8271163392i,used_percent=0,total=8271163392i 1514876565000000000
swap,metrics_source=telegraf_demo,host=e9e7086036b8 out=0i,in=0i 1514876565000000000

If existing plugins are not suitable for the job, a new plugin can be written

Another option is to use exec plugin to execute a program to collect metrics in any of the accepted input data formats.

For example, following script parse through the output of uptime command and

uptime=`awk '{print $1}' /proc/uptime`
if uptime |grep -q user ; then
load1=`uptime | grep -ohe 'up .*' | sed 's/,//g' | awk '{ print $7}'`
load5=`uptime | grep -ohe 'up .*' | sed 's/,//g' | awk '{ print $8}'`
load15=`uptime | grep -ohe 'up .*' | sed 's/,//g' | awk '{ print $9}'`
load1=`uptime | grep -ohe 'up .*' | sed 's/,//g' | awk '{ print $5}'`
load5=`uptime | grep -ohe 'up .*' | sed 's/,//g' | awk '{ print $6}'`
load15=`uptime | grep -ohe 'up .*' | sed 's/,//g' | awk '{ print $7}'`
echo "uptime,host=$hostname uptime=$uptime,load1=$load1,load5=$load5,load15=$load15"

The script output value is echoed in InfluxDB’s Line Protocol format.

# sh uptime.sh
uptime,host=localhost.localdomain uptime=25524.65,load1=1.14,load5=1.06,load15=1.04

Configuration looks like following. In this example influxdb output plugin is used.

interval = "5s"
round_interval = true
  commands = ["/etc/telegraf/uptime.sh"]
  data_format = "influx"

  url = "http://influxdemo:8086"
  database = "telegraf"

Script file is included in the Dockerfile.

FROM telegraf
COPY telegraf.conf /etc/telegraf/telegraf.conf
COPY uptime.sh /etc/telegraf/uptime.sh
RUN chmod +x /etc/telegraf/uptime.sh
CMD ["telegraf"]

To ensure connection between the containers lets setup a docker network,

# docker network create --driver=bridge mintonw
# docker network ls
7674c3097332 bridge bridge local
d42af7253e0f host host local
fd11e0df5eb6 mintonw bridge local

Start an influxb container in above network.

# docker run --net=mintonw --name influxdemo influxdb

In the directory with new telegraf.conf and Dockerfile, build the container and then run the container in docker network,

# docker build -t telegrafdemo .
# docker run --net=mintonw --name telegrafdemo telegrafdemo

Connect to influxdb container,

# docker exec -ti 02bb869bf606 /bin/bash
bash-4.3# influx host localhost
Connected to http://localhost:8086 version 1.2.4
InfluxDB shell version: 1.2.4

and verify that the data stored properly in influxdb.

> show databases
name: databases
> use telegraf
Using database telegraf
> show measurements
name: measurements
> select * from uptime
name: uptime
time host load1 load15 load5 metrics_source uptime
---- ---- ----- ------ ----- -------------- ------
1514874715000000000 3595f01eba01 0.26 0.66 0.54 telegraf_demo 13198.45
1514874720000000000 3595f01eba01 0.24 0.65 0.53 telegraf_demo 13203.45
1514874725000000000 3595f01eba01 0.22 0.65 0.52 telegraf_demo 13208.45
1514874730000000000 3595f01eba01 0.2 0.65 0.52 telegraf_demo 13213.45
1514874735000000000 3595f01eba01 0.18 0.64 0.51 telegraf_demo 13218.45
1514874740000000000 3595f01eba01 0.17 0.64 0.5 telegraf_demo 13223.45
1514874745000000000 3595f01eba01 0.15 0.63 0.49 telegraf_demo 13228.45

As demonstrated above, custom metrics can be exported easily using telegraf exec plugins to influxdb.

P.S The functionality explained here is only for demonstration purpose. Metrics from uptime command can be exported using an existing system input plugin, as long as /var/run/utmp file from host is available in the container.


Interested in working with Minto? Schedule a tech call.

About the Author

Minto Joseph is an expert in opensource technologies with a deep understanding of Linux. This allows him to troubleshoot issues from kernel to the application layer. He also has extensive experience in debugging Linux performance issues. Minto is now using his strong opensource skills to explore a new generation tools as well, to architect, implement and debug value-added client environments.

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *