motivation
This story of dish starts in the year 2016. I was developing my first “mature” Information System (based on sakalWeb IS 4). Out of blue I got an idea about how to monitor other services, and how cool it would be to see green a red labels jumping around to signalize service’s last status.
The latter version (around 2018) was mostly inspired by Zabbix interface. I just wanted a list of services, their last status, amount of time after last test. Plus some modification and deletion clicable little “X"s (no need to use icons in the minimalist’s world).
tiny-monitor v2 attempt
Around the year of 2020, I started to provide my friend (who had been managing a lot of sites at the time) with service monitoring, as that proto-monitor was able to send messages to Telegram groups via Telegram bot.
Fig. 2: Public page thumbnail of newer PHP8 version (tiny-monitor, deprecated now).
Next year, when it started to look awfully simple and badly managable, I decided to rewrite tiny monitor into a fresh, atomized PHP service (repo archive). This went well as I documented the whole API-first structure according to OpenAPI 3.0 and Swagger. The problem now was missing frontend for socket (basically a combination of hostname and some port) modification, editing, deletion. I even intended to write it in pure PHP 8 and according to OOP standards. Unfortunately, this was a dead end.
Fig. 3: PHP8 version’s (tiny-monitor) swagger API documentation thumbnail, cropped.
golang and metrics stack
But services need to be monitored, otherwise they went off silently to the time a customer shows at your door… So finally, at the start of year 2022, me and my colleague tack started to implement a very minimalistic golang module/package to see if it would work. After several reversed commits, merge conflicts (all credits to git and Linus Torvalds!), we ended wondering that little binary were functioning properly.
Golang is flexible and amazingly fast low-level language. Unfortunately it is sometimes “impossible” to set the dev setup the right way. This module is missing, that one requires version 1.18 while you are running 1.17… (Version 1.19 was out the time i was writing this post.) Fortunately, there is a magical tool called Docker container. Using this one, we were able to wrap the source code around a genuine dev environment and to successfully build the binary out of it.
From Docker container it took us a little leap forward to start sending metrics to Grafana. Due to the Prometheus no-push/post policy, we needed to rebuild the dish in a bit different way. As we didn’t want to make another HTTP service as there is already an ocean of HTTP servers, I deciedd to take a chance and use http.Client to POST basic metrics to some target. This target would be a Pushgateway instance, that is used as a workaround for Prometheus and that stores just one value at a time, so Prometheus can scrape it in its scraping intervals (e.g. every 30 seconds).
Fig. 3: Golang monitor (savla-dish) metrics stack interconnection schema.
Since Grafana provides cool and effective dashboard building interface, we decided to give it a try too. Also, we wanted to improve dish’s integration with Telegram instant messaging platform by using Prometheus alert (not necessary AlertManager, but it is possible this would be needed too in the future). This alert should give us, when firing, a status of dish not sending any data for a given period of time. Moreover, to pass some given theshold of failed sockets count (by default 0).
Fig. 4: Grafana dashboard showing two concurrent dish instances active, cropped.
In total, this might seem to be a redundant metric stack for just one little monitoring instance. Well, our stack is washed by numerous types of metrics at a time, as Prometheus scrapes around 12 targets now. From this point of view, there is only a dish, and a pushgateway instance that completes the additional stack.
Note: one could say, that this looks like Blackbox Exporter (BE) reloaded. If not mentioned already, dish is inspired by BE too, mainly in its simplicity and smallness. Moreover, dish could be understood as another implementation of the same principe/concept.
Table: Base Docker image sizes comparsion — dish and pushgateway by Prometheus (Aug 22, 2022).
GitHub dish golang repo.