Picasoft metrics bot

This bot aims to collect some metrics from different services and push them to InfluxDB database. It uses Python3 and is designed to be easily extends.

Usage

You can simply run this script after installing requirements (pip3 install -r requirements.txt) and creating a configuration file with python3 main.py.

Configuration

This bot use a JSON file to get some different configuration variables it needs. An example of this file is provided under config/config_example.json and need to be copy (and modified) as config/config.json. This file contains InfluxDB connection information and configuration needed for each module (Etherpad, Mattermost, etc. see Modules section).

InfluxDB

Configuration for InfluxDB is under the influxdb key. It is a simple object with following subkeys :

• url : complete URL to connect to InfluxDB (eg. https://my.influxinstance.tld)
• user : A user with write access to InfluxDB
• password : Password for the user
• database : Database to use

Docker

A simple Docker image is provided in order to run this bot on a regular basis. It is a simple Python 3 (Alpine based) Docker image with all the requirements. The entrypoint is quite simple : a while loop that call the main.py script and sleep for some times. The interval to sleep between each calls can be configured with the environment variable INTERVAL_SECONDS (default to 60).

Also, don't forget to mount your configuration file on /code/config/config.json

Modules

This bot is modular : each module provide an interface allow to collect metrics for a service. The main function use those modules and push all data to InfluxDB.

Etherpad

Etherpad module allow to get the number of pads of your Etherpad instances. It only calls the https://your.instance.tld/stats.json, which is an endpoint that give a small JSON with pads count. This is an Etherpad plugin that can be install easily.

Configuration

To enable Etherpad module, you need to add an etherpad key to the modules object inside configuration JSON file. The value should be like this :

  "etherpad" : [
    {
      "url" : "https://my.etherpad.tld",
      "name" : "instancename"
    }
  ]

Pay attention that it is a list of objects (wich contains URL and name of your instance) in order to collect data from multiple Etherpad instances.

Metrics

Etherpad module exports following metrics :

• etherpad_pads_count : Number of pads on the instance
• etherpad_blank_pads_count : Number of blank pads on the instance
• etherpad_total_users : Number of connected users Each metric have a name tag with the name of the instance.

Mattermost

Mattermost module collects several metrics from the Mattermost API /analytics/old endpoint.

Configuration

To enable Mattermost module, you need to add a mattermost key to the modules object inside configuration JSON file. The value should be like this :

  "mattermost" : [
    {
      "url" : "https://my.mattermost.tld",
      "user" : "admin_username",
      "password" : "admin_password",
      "name" : "instancename"
    }
  ]

Pay attention that it is a list of objects (wich contains URL, credentials and name of your instance) in order to collect data from multiple Mattermost instances.

Metrics

Mattermost module exports following metrics :

• mattermost_public_channels_count : Number of public channels on the instance
• mattermost_private_channels_count : Number of private channels on the instance
• mattermost_posts_count : Number of posts on the instance
• mattermost_users_count : Number of users accounts on the instance
• mattermost_teams_count : Number of teams created on the instance
• mattermost_daily_posts : Number of posts created on a day
• mattermost_daily_users : Number of users that write a post on a day Each metric have a name tag with the name of the instance.

Creating a module

This bot can be easily extends by adding a module to gather metrics from a new service.

Python package

Your module should be a, local, Python package with the name of your service. For example, if you want to get stats from Peertube, create a peertube folder with a __init__.py file and all the Python scripts you need for this. Everything should run with Python 3, and you can add some requirements to the requirements.txt file.

Methods

Your module have to export a single Python class (let's say PeertubeCollector) that contains at least two methods :

• init : The function called during object instanciation. This function will receive the entire configuration for your module from the configuration file.
• collect : A function called by the main script that have to return a list of correct InfluxDB metrics

Except of those two methods, you can do whatever you want inside the module.

Metrics format

The collect methods have to return a list of InfluxDB metric. A InfluxDB metric is a dict like this one :

{
    "measurement": "name_of_your_metric",
    "tags": {
        "tag1key": "tag1value",
        "tag2key": "tag2value"
    },
    "time": 1537193258183,
    "fields": {
        "value": 15
    }
}

The fields.value value is your metric value. The time value must be a timestamp in miliseconds (DO NOT use seconds, always convert).

Add to main script

When your package is ready, you can add it to the main.py script. First, import it at the beginning with from modulename import ModuleCollector (eg. from peertube import PeertubeCollector).

Then add few lines to the main function to call your module only when it is enable in the configuration file. You should check that there is your module on the modules object from configuration file, and give the configuration to your class initializer. For example, you can imagine something like this :

if 'peertube' in config['modules']:
  peertube = PeertubeCollector(config['modules']['peertube'])
  peertube_data = peertube.collect()
  influx_client.write_points(peertube_data, 'ms')

Documentation

It is important to add some documentation in order to use your module. First, add a configuration example inside the config/config_example.json file.

Then write some documentation inside this README file with at least :

• quick description of your module
• an example of configuration with some explanations
• the list of metrics and associated tags that are collect by the module