Distributed metrics in PHP using Go and Gob

25 Sep 2024 - by 'Maurits van der Schee'

In the previous post I showed how to do high frequency metrics in PHP with TCP sockets. In this post I'll show how to collect and combine metrics from multiple PHP application servers. Instead of sending the log lines from each server to a single node, the monotonically increasing counters from all PHP application server's metrics HTTP end-points are scraped and added up. The transfer of the metrics is done using Go's very efficient Gob protocol (over HTTP).

Go's Gob protocol

The Gob protocol is inspired by "protocol buffers" but it is strictly designed for "communicating between servers written in Go" and promises to be "much easier to use" and "possibly more efficient".

Prometheus metrics: 6 kilobytes
Prometheus metrics (gzipped): 1 kilobyte
Gob protocol: 105 bytes

As you see for this specific use case it makes a lot of sense to use the Gob protocol as it is clearly "more efficient" (as promised).

Much easier to use

Gob protocol is also promised to be "much easier to use". The following struct is accessed from different go routines and therefor has a private mutex, but otherwise it is just a struct with public members.

type Metrics struct {
    mutex     sync.Mutex
    Counters  map[string]uint64
    Durations map[string]float64    
}

Here is an example for decoding an arbitrary nested struct "m" (type Metrics) received over HTTP:

response, err := http.Get(url)
m := metrics.New()
err = m.ReadGob(response)

func (m *Metrics) ReadGob(resp *http.Response) error {
    m.mutex.Lock()
    defer m.mutex.Unlock()
    return gob.NewDecoder(resp.Body).Decode(m)
}

And here is an example for the encoding of such struct:

err := http.ListenAndServe(":9999", http.HandlerFunc(
    func(writer http.ResponseWriter, request *http.Request) {
        m.WriteGob(writer)
    },
))

func (m *Metrics) WriteGob(writer http.ResponseWriter) error {
    m.mutex.Lock()
    defer m.mutex.Unlock()
    return gob.NewEncoder(writer).Encode(m)
}

Note that there is no code change needed for supporting nested structs, maps of structs, slices or any other Go types. The code above is implemented in the metrics package of php-observability.

A network of metric scrapers

Since every PHP server logs to localhost, it will not be influenced by network congestion. Since the aggregation is done locally it is also very fast. The metrics are exposed in Prometheus (text) format for (human) analysis, but also in (binary) Gob format for efficient scraping by other servers. Any installation of the aggregation server can both expose and scrape metrics (from multiple servers), allowing you to create a complex network of scrapers.

See: https://github.com/mevdschee/php-observability

Inserting into the database

While you can let InfluxDB or Prometheus scrape the Prometheus end-point, you may also store the metrics in PostgreSQL (TimescaleDB) or MySQL. I started writing a small application that scrapes Gob metrics and inserts those metrics into the database. It creates a new table for each metric with a 30 day retention, allowing you to keep your indexes small and your insertion speeds high.

See: https://github.com/mevdschee/metrics-db-importer

Links

PS: Liked this article? Please share it on Facebook, Twitter or LinkedIn.

TQ
dev.com