GO API monitoring using Prometheus, and Grafana Integration using Docker Compose

This post is Prometheus hands on, where we will setup API monitoring using Prometheus, integration with Grafana for data visualization. The entire setup will be achieved using docker compose.

If you are new to Prometheus, I would recommend to read my post on What is Prometheus?

We will go through step by step as below

Architecture

Exposing /metrics endpoint in GO API

Prometheus.yml

Docker compose

Running Docker Compose

Launching applications
go rest api, metrics endpoint, prometheus, grafana

Grafana configuration

Load test the api and monitor metrics on grafana

Show me the Code!

Architecture

We are using docker compose and running 3 containers

a) golang
b) prometheus
c) grafana

Exposing /metrics endpoint in GO API

Import prometheus client libraries

For exposing metrics from our GO api, we will need to import golang client libraries for Prometheus.

Prometheus has official client libraries for GO, Java or Scala, Python and Ruby.
Unofficial client libraries are also available for other languages.

Have a look here for list of Prometheus Client libraries

1
2
3
4



"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"

Creating Counter metrics and Middleware, registering counter metric to Prometheus

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23



var totalRequest = prometheus.NewCounterVec(
	prometheus.CounterOpts{
			Name: "http_request_total",
			Help: "Number of get request",
	},
	[]string{"path"},
)

func prometheusMiddleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
		
		next.ServeHTTP(w, r)

		totalRequest.With(prometheus.Labels{"path":"/products"}).Inc()

	})
}

func init(){
	prometheus.Register(totalRequest)
}

Registering Prometheus http handler to metrics endpoint

1
2
3
4
5
6
7



servMux := mux.NewRouter()

servMux.Use(prometheusMiddleware)

servMux.Path("/metrics").Handler(promhttp.Handler())

Prometheus.yml

This is the configuration where we tell Prometheus to scarp the data from an endpoint. In this case, we are asking prometheus to pull metrics from golang application on /metrics end point and the IP address.

This is static configuration. Prometheus also supports service discovery pattern to find it’s targets for pulling the metrics.

1
2
3
4
5
6
7
8
9


global:
  scrape_interval: 15s 

scrape_configs:

  - job_name: golang
    metrics_path: /metrics
    static_configs:
      - targets: ["192.168.99.100:8080"]

Docker compose

Let’s create our docker compose file.

The file is self explanatory, the only catch is at line 16, while mounting prometheus volumes.

And this is because, I am running docker on Windows 10 using Oracle Virtual VM.
With this docker setup, there is only one shared folder form your local host to virtual VM and it is c:\\users.

If we want to put prometheus.yml at some other location on local host then we have to make that folder sharable with Oracle VM box.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


version: '3.1'

services:
    golang:
        container_name: golang
        build:
            context: ./
            dockerfile: Dockerfile    
        restart: always
        ports:
            - 8080:7070
    prometheus:
        image: prom/prometheus:latest
        container_name: prometheus
        volumes:
            - //c/Users/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
            - prometheus_data:/prometheus
        command:
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--storage.tsdb.path=/prometheus'
            - '--web.console.libraries=/usr/share/prometheus/console_libraries'
            - '--web.console.templates=/usr/share/prometheus/consoles'
        ports:
            - 9090:9090
        restart: always

    grafana:
        image: grafana/grafana:latest
        container_name: grafana
        volumes: 
            - grafana_data:/var/lib/grafana
            - ./grafana/provisioning:/etc/grafana/provisioning
        environment:
            - GF_AUTH_DISABLE_LOGIN_FORM=false
            - GF_AUTH_ANONYMOUS_ENABLED=false
            - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
        ports:
            - 3000:3000        

volumes: 
    grafana_data:
    prometheus_data:

Running Docker Compose

starting docker compose

1

$ docker-compose up -d

-d is for running containers in detachable mode.

verifying containers

1

$ docker ps

We have 3 containers running now

golang at 8080,
prometheus at 9090 and
grafana at 3000

Launching applications

Launch application with below urls.
Note:- I am using “192.168.99.100” as host because that’s my Oracle VM ip address for docker.

golang end points

http://192.168.99.100:8080/products

Click to see product API response

http://192.168.99.100:8080/metrics

Click to see metrics response

Prometheus application

This will show the metrics which prometheus data retrieval service is scrapping from golang application over /metrics

http://192.168.99.100:9090/

Click to see how data looks on Prom UI

Grafana application
http://192.168.99.100:3000/
Login with default user and password as admin.

Grafana configuration

Once we login to Grafana using admin and admin as user and password.

Create DataSource

This is where, we tell Grafana to talk to Prometheus.
After this configuration, grafana pull data from Prometheus htp server component using PromQL.

Create dashboard

This is where you will create dashboard and then panel.
Inside Panel you can add your promQL query and there are many other configuration to play around.

Panel and it’s configuration

Load test the api and monitor metrics on grafana

Let’s add load to the API, so that we can see real time graph in Grafana.

I will use hey for adding load to my api. If you haven’t tried hey library, give a try. Hey is beautiful.

download hey

$ go get -u github.com/rakyll/hey to install hey

Verify download

$ which hey

You should see hey folder inside your $GO-PATH\bin\

Running the load

hey -n 10 -c 2 -m GET -T “application/json” http://192.168.99.100:8080/products

-n = Number of requests to run.
-c = Number of workers to run concurrently
-m = HTTP method
-T = Content-type

Grafana dashboard after load

Show me the Code!

Go API with Prometheus and grafana integration using Docker Compose