Introduction

Recently Go has released Go 1.12, which added support to retrieve build module information via runtime/debug package. Therefore, I decided to build a Go module to expose this information for Prometheus. I’m happy to announce the release of prommod, which you can find here:

Background

I think it’s a really cool idea to put Go Module version information into Prometheus, as I had previously mentioned. This provides us with a global view of software packages we actually have running in production, as well as an up to date answer – considering this information is updated every minute or so (based on Prometheus scrape period).

Imagine you just learned that Go Module X with version v1.2.0 had just announced a critical security vulnerability. Now you would like to know which of your applications are currently being affected. This can become a hard question to answer, especially if you are utilizing microservice architecture. Considering you may have thousands of them in use, residing in multiple version control repositories, scattered all over the place, it may be nearly impossible to resolve on your own.

However, by using prommod package, it becomes pretty effortless to know what has been affected. Simply query Prometheus, by issuing this PromQL query:

Maybe you want to know how many different versions of package X we currently have running in production?

There is a PromQL for that:

Or, maybe you would like to get a list of different versions?

Hell, you could even write an alert using similar PromQL language. Maybe you don’t like having too much of a version spread? Let’s say you allow 3 versions to be supported by library X, then PromQL for this would look like:

There are tons of other interesting questions you could ask, for example, how long did it take us to update our packages? Or, how many terribly outdated packages do we have in production?

I think it also has a case for your company’s internal packages. Imagine you are a maintainer of a package and almost everyone in your company uses it. Now, as a maintainer, you would probably like to see what package version various teams are on, what the actual version spread is, or how many services are affected by the bug in Y release. By using prommod, you can do just that.

In practice, I use this approach all the time. Where I work, each team runs Prometheus & Thanos for our monitoring pipeline. From time to time I check for Thanos updates and make a Pull Request with the update for the teams. Once the teams approve, I merge it and expect the teams to actually deploy the changes. Obviously, not all of the teams have the time to do so, but, for me, it’s important that this deployment is not forgotten. After a couple of days or so, I check what our version spread is by issuing this PromQL:

From this query, I get a list of teams that did not apply the update and send them a quick reminder.

Inspiration

Prometheus Version package

This package was totally inspired by Prometheus version package, which has a very interesting API. In order to use it, you have to set Version, Revision, Branch, BuildUser,BuildDate in the global variables of that package. Once you have done that, you need to call version.NewExporter("application_name") to retrieve a Prometheus metrics from it.

Here is a simple code sample for it:

But wait, isn’t package scoped global state bad?

The reason version package uses global variables, instead of a function argument, is that you can populate these variables during build time. Go allows you to do that, using the -ldflags argument. This is how my typical Makefile looks:

To explain this code briefly, global variables like User in package github.com/prometheus/common/version gets populated with user information during build time. You can read more about this approach here.

The actual produced metric by this library looks like this:

As you can see, it just puts a “fake” metric with version information in the labels.

Printing version information

Another cool thing about version package is that it has a nice way of plugging version information with other tools. Let’s say you use the ever popular kingpin package, and would like to show version information to your users. Turns out, it’s really easy – take a look at this example:

This will make a --version flag of your application and display current and relevant version information. This is how the output looks:

Many Open Source projects like Prometheus, Kubernetes, Thanos or Grafana use this package or have a very similar approach. Let’s take a look at Kubernetes, for a moment. Although Kubernetes may not use this package directly, it utilizes its own little package, which adds a bit more content. For example, querying Prometheus with kubernetes_build_inforesults in:

prommod

Usage

Let’s get back to my newly created prommod package, one I built to have a very similar feel to version package. In order to use it, you just call prommod.NewCollector("app_name")and register your metric with Prometheus. Take a look at this code snippet:

prommod will produce a metric for each go module used. Here is how exported metrics look:

As you can see, we create go_mod_info metric, which stores your project dependency version information in Prometheus labels.

Also there is prommod.Print("app_name") and prommod.Info() calls to get a human-readable representation of dependency information, just like in the Prometheus version package. You can use prommod alone, or you can combine it with version package in order to get nice combined output. For example:

Gives you this nice output:

Personally, I find this very useful for development. You can ask your users to submit output of app --version, when submitting bug reports, which then allows you to quickly see all of the dependencies, as well as their versions.

Building prommod

Although this package is small and pretty trivial to implement, writing it was not so easy as it may seem, and I ended up using a lot of Go’s “hidden” features.

Build tags

I really wanted to make this package work on Go with versions lower than 1.12. Although Go modules were introduced in Go 1.11, the debug.ReadBuildInfocall, which is used by prommod, was only introduced in Go 1.12. Therefore, this wouldn’t work for users with older Go versions.

In order to solve this, I used Go build tags. For Go >=1.12 implementation I added Package comment // +build go1.12, which calls debug.ReadBuildInfo. For older Go versions I added // +build !go1.12 comment, and provided the same functions, with identical API. But, instead of calling debug.ReadBuildInfo, I would just return empty strings in prommod.Print("app_name") & prommod.Info() calls, and an empty Prometheus metric.

All in all, if you are using prommod with an older Go version, it will still compile, but won’t produce any useful output.

In order to continuously test this with different versions, I used Travis CI. And I think it’s the best tool for the job. My .travis.yml looks like this:

As you can see it’s just a list of Go versions, which I build and test against, and it all just works 🙂

Go examples are tests

When testing this library, I particularly liked writing Go Examples.
Not a lot of Go developers know, but Go examples are tests. In order to use Example test features, you simply need to print the resulting variable to Standard Output and check it with // Output: comment.

Take a look at this code snippet:

In this case, Go will run this code as a test expecting prommod.Info() to be equal to (). And, if the output is not equal, go test will give you a nice error:

I encountered weird Go 1.12 behavior, as Go behaves like a non Go module program in testing. What happens is runtime/debug.ReadBuildInfo() doesn’t return any of the Go module information, although compiled with Go 1.12, which makes this library a bit harder to unit test.

Using init() functions

Another lesser known feature are the Go init() functions. Basically, init() functions are called once per package, and they’re used to set up state. What I wanted to do is to always return the computed information from a global variable, instead of actually computing everything on Print()/Info() call.

I think it makes a lot of sense for this library, as this information never changes after the initial build. This also has performance benefits, as this library is mostly just returning constant global strings.

Semantic Versions

Lastly, and most importantly, this package tags versions using Semantic Versioning. Go Modules depend on the community using Semantic Versioning. Please use it – especially if you are an author of a Go package! Pretty please? We are all going to benefit from it. 🙂

Conclusion

Although I really like this approach of using the Prometheus monitoring system for storing dependency versions, I do think I might be hammering a nail with a screwdriver here. It feels like we are slightly abusing Prometheus, as metrics are “fake” (i.e. metric value never changes) and we just use labels to store information. Maybe it’s worth investigating other solutions, like Grafeas, for storing this type of information​.

On the other hand, I really like the Pull approach here. I like that it’s sort of an application telling you, “hey, I’m this thing and I use these dependencies to perform my function”, just like application in health checks tells you “hey I think I’m fine”, or “hey I can’t connect to DB”. Also, Prometheus information is as close to real-time as possible. This means that you can monitor application & library updates, live.

Most importantly, what do you think? I would really like to hear more opinions on this topic. And, by the way, if you want to support my work you can buy me a book here.

Thanks for reading! Quality open source is really important to me. prommod is available using MIT license, so feel free to use it. Just remember to star it on github if you really like the package. And, if you are missing something, please create an issue and then a PR if you feel like it.