Metrics Publications

.

Custom Metrics

A custom metric is a single unique combination of metric name, hostname and any number of tags that is not an infrastructure metric. Infrastructure metrics are considered to be metrics which are sent by default when StatsBee/collectD or others are installed and monitor the health of your technology infrastructure. BeeInstant can handle metrics with unlimited cardinality; this means that you can set tags as specific as you can imagine, making root cause analysis more efficient and metrics analysis more accurate.

Custom metrics provide you with the ability to measure the metrics which matter most to you to ensure you are best achieving your business KPI’s. Our free plan allows you to have 50 custom metrics so you can see for yourself the power custom metrics holds without taking any financial risk. Our pro plan contains 100 custom metrics per host. We do not enforce any hard limit on our custom metrics. If your circumstances require additional custom metrics feel free to contact us, and we will suit your needs to the best of our ability.

Using BeeInstant’s SDK

BeeInstant currently has an SDK for Java, Go and .NET(with other languages in the making), so you can easily publish custom metrics from your applications. Each SDK provides multi-dimensional metrics that allow engineers to tailor make their very own metrics that suits unique cases. The metrics are aggregated at global levels across hosts, times and dimensions. For example, engineers can easily visualise percentile 99.99% of their API latencies at service level by aggregating metrics published from all of their hosts.

Each SDK requires you have a BeeInstant public key, secret key and endpoint. This can be found by clicking on the setting icon on the BeeInstant dashboard and then clicking on the account icon. If you do not have an account yet make one now for free on our website

Java SDK

The official BeeInstant Java SDK along with detailed instructions and examples can be found on our GitHub page here. A quick start guide is posted below.

Installation and Setup

To begin, get your BeeInstant credentials, this includes your public key, secret key and the BeeInstant endpoint. Now Launch your service with JVM options using the following line, replacing public key, secret key and endpoint with their respective values:-
-Dbeeinstant.endpoint=<endpoint> -Dbeeinstant.publicKey=<public_key> -Dbeeinstant.secretKey=<secret_key>
The SDK is currently on maven central, you can get it there or simply add the following dependency to your project. Select which build tool you are currently using below to bet the dependency code to add.

Maven

<dependency>
  <groupId>com.beeinstant</groupId>
  <artifactId>metrics</artifactId>
  <version>1.1.7</version>
</dependency>

SBT

libraryDependencies += "com.beeinstant" % "metrics" % "1.1.7"

Gradle

dependencies {
    compile 'com.beeinstant:metrics:1.1.7'
}
Usage
We must first initialise the SDK so we can use it. Afterwards, we create a dedicated metrics logger, giving it a specific tag and at the end of the file ensure to close the SDK by running the following respective commands. Replacing metricsManagernName with a name such as "VideoMeasuring" and tag with "api=Upload".
MetricsManager.init("metricsManagernName");
final MetricsLogger metricsLogger = MetricsManager.getMetricsLogger("tag");
...
MetricsManager.shutdown();
A metric logger is inspired from traditional loggers for example Log4J. But instead of logging text, a metric logger publishes timing metrics, counters or arbitrary metrics in real-time.

Metric Types
Timer
try (TimerMetric timer = metricsLogger.startTimer("ProcessingTime")) {
    // ...
    // Executue some code
    // ...
}
Counter
try (TimerMetric timer = metricsLogger.startTimer("ProcessingTime")) {
    try {
        // ...
        // Executue some code
        // ...
        metricsLogger.incCounter("Success", 1);    
        } catch (Exception e) {
            metricsLogger.incCounter("Failure", 1);
        }
}
Recorder
try (TimerMetric timer = metricsLogger.startTimer("ProcessingTime")) {
    // Read video
    try {
        long numOfReadBytes = 0;
        // Reading ...
        // numOfReadBytes += 100
        // Reading ...
        // numOfReadBytes += 200
        // ...
        metricsLogger.record("ReadBytes", numOfReadBytes, Unit.BYTE); 
        // Publishing metric as bytes
        metricsLogger.incCounter("Success", 1);
    } catch (Exception e) {
        metricsLogger.incCounter("Failure", 1);
    }
}

Additional Metric Manipulation
BeeInstant provides dimension extension feature that allows us to dig deeper into our metrics. By extending the dimension, we make the metric more specific which makes metric analysis more efficient and accurate. For example, a video upload API could be broken down as follows.
 void handleVideoUpload() {
    // ...
    try (TimerMetric timer = metricsLogger.startTimer("ProcessingTime")) {

        final Metrics readMetrics = metricsLogger.extendDimensions("step=Read");
        try (TimerMetric readTimer = readMetrics.startTimer("Time")) {
            // Reading video here...
        }

        final Metrics compressMetrics = metricsLogger.extendDimensions("step=Compress");
        try (TimerMetric compressTimer = compressMetrics.startTimer("Time")) {
            // Compressing video here...
        }

        final Metrics storeMetrics = metricsLogger.extendDimensions("step=Store");
        try (TimerMetric storeTimer = storeMetrics.startTimer("Time")) {
            // Storing video here...
        }
    }
    // ...
}
GoLang SDK

The official BeeInstant GoLang SDK along with detailed instructions and examples can be found on our GitHub page here. A quick start guide is posted below.

Installation and Setup
To begin, get your BeeInstant credentials, this includes your public key, secret key and the BeeInstant endpoint. To install the GoLang SDK run the following command which will install the newest version from our git repository
go get -v github.com/beeinstant-dev/beeinstant-go-sdk
To import the SDK into your application add the following line to your imports.
import (
    ...
    beeinstant "github.com/beeinstant-dev/beeinstant-go-sdk"
)
    

Usage
Now that the SDK is imported we must initialise it using the public key, secret key and endpoint gathered from your account and replacing "myServiceName" and "myEnvironment" with identifiable labels.
beeinstant.InitMetricLogger("MyServiceName",
        "MyEnvironment",
        "PublicKey",
        "SecretKey",
        "Endpoint",
        10) // flush once every 10 seconds

Metric Types
Timer
Time can be measured by first initialising a variable set to the current time, then performing all required operations and lastly publishing the current time from the saved time priot to performing the operations. This can be done with the following code:
startTime := time.Now().UnixNano()
//
// processing works here
//
beeinstant.GetMetricLogger().Record("MyProcessingTime",
    float64((time.Now().UnixNano()-startTime)/1000000),
    beeinstant.MILLI_SECOND)
Counter
beeinstant.GetMetricLogger().IncCounter("metricName", 1)
Recorder
beeinstant.GetMetricLogger().Record("MyPayload", 100, beeinstant.BYTE)

Additional Metric Manipultion
BeeInstant provides dimension extension feature that allows us to dig deeper into our metrics. By extending the dimension, we make the metric more specific which makes metric analysis more efficient and accurate. For example, a video upload API could be broken down as follows.
beeinstant.GetMetricLogger().
    ExtendDimensions("api=PublishMetrics,location=Dublin").
    IncCounter("MyCounter", 1)
.NET SDK

here. A quick start guide is posted below. The official BeeInstant GoLang SDK along with detailed instructions and examples can be found on our GitHub page here. A quick start guide is posted below. Our current SDK targets netstandard.2.0

Installation and Setup
In the root of your applications directory create a JSON configuration file called "beeInstant.config.json". Add the following code to the file, changing public key, secret key and endpoint to your values found in your account
{
    "publicKey": "yourPublicKey",
    "secretKey": "yourSecretKey",
    "endPoint": "yourEndPoint",
    "flushInSeconds": "5", // flush existing metrics to the server every 5 seconds
    "flushStartDelayInSeconds": "10", // due time before automatically sending first metric to the server
    "isManualFlush": false // if true, you will need to push metrics manually, otherwise metrics will be pushed automatically
}

Next install the nuget package which can be viewed here. We provide for installation via PackageManager, .NET CLI and Paket CLI.

Package Manager

Install-Package BeeInstant.NetSDK -Version 1.0.1

.NET CLI

dotnet add package BeeInstant.NetSDK --version 1.0.1

Paket CLI

paket add BeeInstant.NetSDK --version 1.0.1


Usage
We must first initialise the SDK so we can use it. Afterwards, we create a dedicated metrics logger, giving it a specific tag and at the end of the file ensure to close the SDK by running the following respective commands. Replacing metricsManagernName with a name such as "VideoMeasuring" and tag with "api=Upload".
MetricsManager.Initialize("metricsManagernName");
var metricsLogger = MetricsManager.GetMetricsLogger("tag");
...
MetricsManager.Shutdown();
A metric logger is inspired from traditional loggers for example Log4J. But instead of logging text, a metric logger publishes timing metrics, counters or arbitrary metrics in real-time.

Metric types
Timer
using(var timer = metricsLogger.StartTimer("ProcessingTime")){
...
Do Actions
...
}
Counter
metricsLogger.IncrementCounter("metricName", 1); // e.g ("success", 1)
Recorder
metricsLogger.Record("ReadBytes", numOfReadBytes,Unit.Byte);

Additional Metric Maniupulation
BeeInstant provides dimension extension feature that allows us to dig deeper into our metrics. By extending the dimension we make the metric more specific which makes metric analysis more efficient and accurate.
...
    using (var timer = metricsLogger.StartTimer("ProcessingTime")){
var readMetrics = metricsLogger.ExtendDimensions("step=Read");
        using (var readTimer = readMetrics.StartTimer("Time"))
        {
            // Reading video here...
        }
...
}

Sending Custom Metrics using StatsBee and StatsD Client

An alternative to using a SDK to send metrics is using a StatsD client, there are StatsD clients available for nearly every programming language so if we do not have a SDK for your language it is the next best alternative. For the following examples you can launch a docker container using the command in the account setting which will send metrics to BeeInstant

Python

To install the python statsD client run the following command.

pip install statsd

We will now import and configure the statsDClient.

from statsd import TCPStatsClient
statsDClient = TCPStatsClient(‘localhost’, 8094)

Metric Types

Counter
By default when you increment a counter it will only increase it by 1, however, you can increment it by any amount you wish by passing in a counter as the second parameter. To decrement values just change the ‘incr’ to ‘decr’
statsDClient.incr(‘someMetric’) # Increases someMetric by 1
statsDClient.incr(‘otherMetric’, 5) # Increases otherMetric by 5
Timer
Timers are excellent for tracking how long something took to execute. Their operation is simple, start the timer immediately before the code and stop it immediately after
timer = statsDClient.timer(‘someName’)
timer.start()
…
    Some code
…
timer.stop()
Gauges
Gauges are not subject to averaging, and they don’t change unless you change them. That is, once you set a gauge value, it will be a flat line on the graph until you change it again.

statsDClient.gauge('someName', 70)
Ruby
To install the Ruby StatsD client run the following donwload the package which and insert it into your application, the package can be downloaded by clicking here.

To instantiate the statsD object insert the following line into your code.
StatsD.backend = StatsD::Instrument::Backends::UDPBackend.new("127.0.0.1:8125", :statsd)

Metric Types

Counter
By default when you increment a counter it will only increase it by 1, however, you can increment it by any amount you wish by passing in a counter as the second parameter. To decrement values just change the ‘incr’ to ‘decr’. If the specified key doesnt exist it will create it for you
StatsD.increment(‘someName.insert’) # increments metric by 1
StatsD.increment(‘someName.insert’, 10) # you can also specify how much to increment the key by
Measure
Measure uses timers which are excellent for tracking how long something took to execute. Their operation is simple, start the timer immediately before the code and stop it immediately after
StatsD.measure(‘someFunction.insert’) do
  application.insert(product)
end
Gauges
Gauges are not subject to averaging, and they don’t change unless you change them. That is, once you set a gauge value, it will be a flat line on the graph until you change it again.
StatsD.gauge(‘someValue’, 12)
Set
A set keeps track of the number of unique values that have been seen. This is a good fit for keeping track of the number of unique visitors. The value can be a string.
StatsD.set(‘BeeInstant.customers’, “12345”)

PHP

To install the PHP statsD client we use composer by running the following command.

composer require beberlei/metrics

Now that the dependency has been added we can instantiate the client in the file with the following line:

<?php
$collector = \Beberlei\Metrics\Factory::create(‘statsd’);

Metric Types

Counter
To increment or decrement a counter we can use the following line of code. If the provided key doesnt exist it will create it for you.
$collector->increment(‘metricName’); # increments metric by 1
$collector->decrement(‘metricName’);

We can increase a counter by a given value by calling the measure function and passing in the value to increment by as the second parameter

$collector->measure(“someName”, 7) # increments metric by 7
$collector->measure(“someName”, -2) # decrements metric by -4
Timer
Timers which are excellent for tracking how long something took to execute. Their operation is simple, start the timer immediately before the code and stop it immediately after
$start = microtime(true);
// some code
$diff  = microtime(true) - $start;
$collector->timing(‘metricName’, $diff);

Java

The client jar is distributed via maven central, and can be downloaded here. Note this requires It requires Java 8, netty and SLF4J.

We then add the following dependency to our project:

<dependency>
    <groupId>com.timgroup</groupId>
    <artifactId>java-statsd-client</artifactId>
    <version>3.0.1</version>
</dependency>

Once the dependency has been added then import the client into the file using:

import com.timgroup.statsd.StatsDClient;
import com.timgroup.statsd.NonBlockingStatsDClient;

Now that the client has been imported we can instantiate the cliend using:

private static final StatsDClient statsd = new NonBlockingStatsDClient("someName", 8125);

Metric Types

Counter
By default, when you increment a metric it will only increase it by 1. To decrement values just change the ‘increment’ to ‘decrement’. If the specified key doesnt exist it will create it for you
statsd.incrementCounter(“someName”); // Increments the metric by 1
statsd.decrementCounter(“someName”); // Decrements the metric by 1
Timer
Measure uses timers which are excellent for tracking how long something took to execute. Their operation is simple, start the timer immediately before the code and stop it immediately after
long startTime = System.currentTimeMillis();
….
Do Some Code
…..
statsd.recordExecutionTime(“bag”, System.currentTimeMillis() - startTime);
Gauges
Gauges are not subject to averaging, and they don’t change unless you change them. That is, once you set a gauge value, it will be a flat line on the graph until you change it again.
statsd.recordGaugeValue(“someValue”, 100);
Set
A set keeps track of the number of unique values that have been seen. This is a good fit for keeping track of the number of unique visitors. The value can be a string.
statsd.recordSetEvent(“someMetric”, “someValue”);

C#

https://github.com/justeat/JustEat.StatsD NEED TO CONFIRM INSTALLATION PROCESS To install the package first download the repository found here and add to your application.

We will now insantiate the StatsDPublisher Object to enable us to send metrics.

var statsDConfig = new StatsDConfiguration { Host = "someIdentifiableName" };
IStatsDPublisher statsDPublisher = new StatsDPublisher(statsDConfig);

Metric Types
When sending metrics ensure to only send a long or a double.

Counter
By default when you increment a counter it will only increase it by 1, however, you can increment it by any amount you wish by passing in a counter as the first parameter followed by the metric name. To decrement values just change the ‘Increment’ to ‘Decrement’. If the specified key doesnt exist it will create it for you
statsDPublisher.Increment(“someMetric”); // Increments the metric by 1
statsDPublisher.Increment(10, “DoSomething.Attempt”); // Increments the metric by 10
Timer
Measure uses timers which are excellent for tracking how long something took to execute.
using (statsDPublisher.StartTimer(“someName”))
{
    DoSomething();
}
Gauges
Gauges are not subject to averaging, and they don’t change unless you change them. That is, once you set a gauge value, it will be a flat line on the graph until you change it again.
statsDPublisher.Gauge(100, “someValue”);
Set
A set keeps track of the number of unique values that have been seen. This is a good fit for keeping track of the number of unique visitors. The value can be a string.
statsd.recordSetEvent(“someMetric”, “someValue”);

Infrastructure Metrics

Collect infrastructure metrics via Telegraf

To configure the statsD client in our file we will use TCP as this results in no data loss unlike UDP.

Telegraf is a plugin-driven server agent used for collecting metrics. It is wholly developed in Go and compiles into a single binary with no external dependencies, ensuring minimal memory overhead. Plugins can easily be added and remove by editing the configuration file.

Note: The following installations are for Telegraf version 1.7.3

Mac via HomeBrew

brew update
brew install telegraf

Docker

docker pull telegraf

Ubuntu & Debian

wget https://dl.influxdata.com/telegraf/releases/telegraf_1.7.3-1_amd64.deb
sudo dpkg -i telegraf_1.7.3-1_amd64.deb

RedHat & CentOS

wget https://dl.influxdata.com/telegraf/releases/telegraf-1.7.3-1.x86_64.rpm
sudo yum localinstall telegraf-1.7.3-1.x86_64.rpm

Windows Binaries (64-bit)

wget https://dl.influxdata.com/telegraf/releases/telegraf-1.7.3_windows_amd64.zip
unzip telegraf-1.7.3_windows_amd64.zip

Linux Binaries (64-bit)

wget https://dl.influxdata.com/telegraf/releases/telegraf-1.7.3_linux_amd64.tar.gz
tar xf telegraf-1.7.3_linux_amd64.tar.gz

Linux Binaries (32-bit)

wget https://dl.influxdata.com/telegraf/releases/telegraf-1.7.3_linux_i386.tar.gz
tar xf telegraf-1.7.3_linux_i386.tar.gz

Linux Binaries (ARM)

wget https://dl.influxdata.com/telegraf/releases/telegraf-1.7.3_linux_armhf.tar.gz
tar xf telegraf-1.7.3_linux_armhf.tar.gz
Configuring and Starting Telegraf

Generate the configuration file by running the command.

telegraf config > telegraf.conf

Now that it’s created we will edit this file to make it send metrics to BeeInstant. So let’s open the file using Vim.

vi telegraf.conf

Next let’s disable the influxdb output. To do this we will press ‘/’ which starts the search function, we will now type in ‘influxdb’ and press enter. Now we press ‘n’ until we reach the line containing:

[[outputs.influxdb]]

All we have to do is add a ‘#’ before it, thus making the whole line a comment. To do this press “i” to enter insert mode and insert a ‘#’ at the start of the line when the # is entered press ‘esc’ to exit insert mode.

# [[outputs.influxdb]]

Now we will create the link between BeeInstant and the server. To do this press the ‘/’ button on your keyboard, type “outputs.socket” and press enter. To enable this function we will remove all ‘#’ on the following two lines by entering insert mode (press ‘esc’) and pressing backspace just after the characters.

[[outputs.socket_writer]]
address = “tcp://127.0.0.1:8094”

Let’s save the file by typing “:x”, hit enter and the configuration file is now done.

Start Telegraf

Start the Telegraf service and direct it to the relevant configuration file:

OS X Homebrew

telegraf --config telegraf.conf

Linux (sysvinit and upstart installations)

sudo service telegraf start

Linux (systemd installations)

systemctl start telegraf

Metrics are currently being sent to BeeInstant! To view these metrics simply log in, and search for the IP address to view the infrastructure metrics

Collect infrastructure metrics via CollectD

collectd is a daemon which collects system and application performance metrics periodically and provides mechanisms to store the values in a variety of ways. collectd gathers metrics from various sources, e.g. the operating system, applications, logfiles and external devices, and stores this information or makes it available over the network.it’s written in C for performance and portability, allowing it to run on systems without scripting language or CRON daemon, such as embedded systems.

Ubuntu / Debian

Firstly make sure everything is updated then install the collectD package
sudo apt update
sudo apt upgrade  
apt install collectd
We will now open the configuration file which can be done by running the command
nano /etc/collectd.conf
Un-comment the line below and configure it to match your servers hostname
#Hostname "localhost"
sudo service collectd start

Fedora / CentOS

Firstly make sure everything is updated. The required packages are contained the EPEL repository so you will need to install that before installing collectD itself.
yum update
yum upgrade
yum install epel-release
yum install collectd
We will now open the configuration file which can be done by running the command
nano /etc/collectd.conf
Un-comment the line below and configure it to match your servers hostname
#Hostname "localhost"
Lastly, if you use CentOS 7 you will need to enable collectD, otherwise skip the first line
systemctl enable collectd
systemctl start collectd

Collect metrics through Graphite protocol

BeeInstant fully supports receiving data through the Graphite plain text protocol. When sending data through the plain text protocol you must ensure that the data is sent in the following format.

<metric path> <metric value> <metric timestamp>

On Unix, the nc program (netcat) can be used to create a socket and send data to BeeInstant, this can be done using the following line of code.

echo “local.random.diceroll 4 date +%s” | nc -c SERVER PORT

SERVER should be replaced with the server being used and PORT to use, in most cases the server will be “localhost” and the Port will be 8125 for UDP or 8094 for TCP.