What’s up? DIY monitoring with Prometheus

Most of the time you are wondering what is happening inside your application and there are enough monitoring apps on the market that one of them should fulfill your needs. However, I decided to go for a DIY approach, as it gives you a better understanding, control and it is almost free — well you still have to pay for the computation. And it can become a new hobby. :)

I’ve looked around for a project, and I found Prometheus. It’s Github repository has been downloaded only less than Kubernetes, which makes it the best candidate.

In this blog post, I’m going to demonstrate how to install Prometheus and how to get the metrics of each server. I will expand on this in the future posts.

To get started, you need a server to run the Prometheus engine on it and Node exporters on the servers you like to monitor.

Installing Prometheus:

To start Prometheus

Now the Prometheus is running with its default config, and you can have a look by typing http://SERVER'S_IP:9090 so let's move to the other component which gathers the system information such as Memory, CPU or network from the machine. The agent is called Node Exporter for which the installation process is the same.

Installing Node Exporter:

Now the Node exporter is running on port 9100 but Prometheus is not configured to read from it yet, so back to config:

Prometheus config file

If you remember there is a file /etc/prometheus/prometheus.yml which is used as config file.

The following is an example for a server that Prometheus and Node Exporter are running on it, as you need information even for the server you are running Prometheus.

Now if there is another server that you want to monitor, you’ll need to install Node Exporter on it and change the config so Prometheus will retrieve data from it:

As you can see targets are arrays that you can add the new servers as you go along. Just make sure that you restart Prometheus.

If you’ve followed this step by step there is an issue restarting the Prometheus as we sent it to the background (remember adding that & at the end of the Prometheus command). So we need to first find the process ID of Prometheus and after that send a TERM signal to stop it.

Finding the process ID (PID)

In the result you look for the line that shows the command you run, the second column is the PID. Now you need to send a TERM signal to the process.

After this, start Prometheus again like how we did before.

There is still a long way to go, but for now, this should keep you busy. Look and get familiar with the code before we move to the next stage.

Happy coding!

Originally published at blog.cloud66.com on January 23, 2018.

DevOps-as-a-Service to help developers build, deploy and maintain apps on any Cloud. Sign-up for a free trial by visting: www.cloud66.com