Exposing Go Modules to Prometheus

Introduction

Recently Go has released Go 1.12, which added support to retrieve build module information via runtime/debug package. Therefore, I decided to build a Go module to expose this information for Prometheus. I’m happy to announce the release of prommod, which you can find here:

go get github.com/povilasv/prommod

Background

I think it’s a really cool idea to put Go Module version information into Prometheus, as I had previously mentioned. This provides us with a global view of software packages we actually have running in production, as well as an up to date answer – considering this information is updated every minute or so (based on Prometheus scrape period).

Imagine you just learned that Go Module X with version v1.2.0 had just announced a critical security vulnerability. Now you would like to know which of your applications are currently being affected. This can become a hard question to answer, especially if you are utilizing microservice architecture. Considering you may have thousands of them in use, residing in multiple version control repositories, scattered all over the place, it may be nearly impossible to resolve on your own.

However, by using prommod package, it becomes pretty effortless to know what has been affected. Simply query Prometheus, by issuing this PromQL query:

go_mod_info{name="X",version="v1.2.0"}

Maybe you want to know how many different versions of package X we currently have running in production?

There is a PromQL for that:

count(count by (version)(go_mod_info{name="X"}))

Or, maybe you would like to get a list of different versions?

count by (version)(go_mod_info{name="X"})

Hell, you could even write an alert using similar PromQL language. Maybe you don’t like having too much of a version spread? Let’s say you allow 3 versions to be supported by library X, then PromQL for this would look like:

count(count by (version)(go_mod_info{name="X"})) > 3

There are tons of other interesting questions you could ask, for example, how long did it take us to update our packages? Or, how many terribly outdated packages do we have in production?

I think it also has a case for your company’s internal packages. Imagine you are a maintainer of a package and almost everyone in your company uses it. Now, as a maintainer, you would probably like to see what package version various teams are on, what the actual version spread is, or how many services are affected by the bug in Y release. By using prommod, you can do just that.

In practice, I use this approach all the time. Where I work, each team runs Prometheus & Thanos for our monitoring pipeline. From time to time I check for Thanos updates and make a Pull Request with the update for the teams. Once the teams approve, I merge it and expect the teams to actually deploy the changes. Obviously, not all of the teams have the time to do so, but, for me, it’s important that this deployment is not forgotten. After a couple of days or so, I check what our version spread is by issuing this PromQL:

count(thanos_build_info) by (revision)

From this query, I get a list of teams that did not apply the update and send them a quick reminder.

Inspiration

Prometheus Version package

This package was totally inspired by Prometheus version package, which has a very interesting API. In order to use it, you have to set Version, Revision, Branch, BuildUser,BuildDate in the global variables of that package. Once you have done that, you need to call version.NewExporter("application_name") to retrieve a Prometheus metrics from it.

Here is a simple code sample for it:

import (
   "github.com/prometheus/common/version"
   "github.com/prometheus/client_golang/prometheus"
)

func main(){
    version.Version = "v1.11"
    version.Revision = "sha"
    version.Branch = "master"
    version.BuildUser = "povilas"
    version.BuildDate = "2019-03-08"
    prometheus.MustRegister(version.NewCollector("app_name"))
}

But wait, isn’t package scoped global state bad?

The reason version package uses global variables, instead of a function argument, is that you can populate these variables during build time. Go allows you to do that, using the -ldflags argument. This is how my typical Makefile looks:

DATE := $(shell date +%FT%T%z)
USER := $(shell whoami)
GIT_HASH := $(shell git --no-pager describe --tags --always)
BRANCH := $(shell git branch | grep * | cut -d ' ' -f2)

build:
go build -a -ldflags '-s
-X github.com/prometheus/common/version.Version=$(GIT_HASH)
-X github.com/prometheus/common/version.BuildDate="$(DATE)"
-X github.com/prometheus/common/version.Branch=$(BRANCH)
-X github.com/prometheus/common/version.Revision=$(GIT_HASH)
-X github.com/prometheus/common/version.BuildUser=$(USER)'

To explain this code briefly, global variables like User in package github.com/prometheus/common/version gets populated with user information during build time. You can read more about this approach here.

The actual produced metric by this library looks like this:

# HELP app_name_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which app_name was built.
# TYPE app_name_build_info gauge

app_name_build_info{branch="master",goversion="devel +5126feadd6 Sat Mar 2 18:28:10 2019 +0000",revision="32eb8c1",version="32eb8c1"} 1

As you can see, it just puts a “fake” metric with version information in the labels.

Printing version information

Another cool thing about version package is that it has a nice way of plugging version information with other tools. Let’s say you use the ever popular kingpin package, and would like to show version information to your users. Turns out, it’s really easy – take a look at this example:

app := kingpin.New("app_name", "My super awesome application")
app.Version(version.Print("app_name"))

This will make a --version flag of your application and display current and relevant version information. This is how the output looks:

app_name, version 32eb8c1 (branch: master, revision: 32eb8c1)
  build user:       povilasv
  build date:       "2019-03-08T17:59:47+0200"
  go version:       devel +5126feadd6 Sat Mar 2 18:28:10 2019 +0000

Many Open Source projects like Prometheus, Kubernetes, Thanos or Grafana use this package or have a very similar approach. Let’s take a look at Kubernetes, for a moment. Although Kubernetes may not use this package directly, it utilizes its own little package, which adds a bit more content. For example, querying Prometheus with kubernetes_build_inforesults in:

kubernetes_build_info{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",buildDate="2018-11-26T12:46:57Z",compiler="gc",failure_domain_beta_kubernetes_io_region="ax-aaa-1",failure_domain_beta_kubernetes_io_zone="ax-aaa-3a",gitCommit="435f92c719f279a3a67808c80521ea17d5715c66",gitTreeState="clean",gitVersion="v1.12.3",goVersion="go1.10.4",major="1",minor="12"}

prommod

Usage

Let’s get back to my newly created prommod package, one I built to have a very similar feel to version package. In order to use it, you just call prommod.NewCollector("app_name")and register your metric with Prometheus. Take a look at this code snippet:

import (
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main(){
    prometheus.MustRegister(prommod.NewCollector("app_name"))
}

prommod will produce a metric for each go module used. Here is how exported metrics look:

# HELP go_mod_info A metric with a constant '1' value labeled by dependency name, version, from which app_name was built.
# TYPE go_mod_info gauge
go_mod_info{name="github.com/beorn7/perks",program="app_name",version="v0.0.0-20180321164747-3a771d992973"} 1
go_mod_info{name="github.com/golang/protobuf",program="app_name",version="v1.2.0"} 1
go_mod_info{name="github.com/matttproud/golang_protobuf_extensions",program="app_name",version="v1.0.1"} 1
go_mod_info{name="github.com/povilasv/prommod",program="app_name",version="v0.0.11-0.20190309143328-e661980fc053"} 1
go_mod_info{name="github.com/prometheus/client_golang",program="app_name",version="v0.9.2"} 1
go_mod_info{name="github.com/prometheus/client_model",program="app_name",version="v0.0.0-20180712105110-5c3871d89910"} 1
go_mod_info{name="github.com/prometheus/common",program="app_name",version="v0.0.0-20181126121408-4724e9255275"} 1
go_mod_info{name="github.com/prometheus/procfs",program="app_name",version="v0.0.0-20181204211112-1dc9a6cbc91a"} 1

As you can see, we create go_mod_info metric, which stores your project dependency version information in Prometheus labels.

Also there is prommod.Print("app_name") and prommod.Info() calls to get a human-readable representation of dependency information, just like in the Prometheus version package. You can use prommod alone, or you can combine it with version package in order to get nice combined output. For example:

func main() {
    fmt.Println(prommod.Print(version.Print("app_name")))
}

Gives you this nice output:

app_name, version 32eb8c1 (branch: master, revision: 32eb8c1)
  build user:       povilasv
  build date:       2019-03-09T06:22:18+0200
  go version:       devel +5126feadd6 Sat Mar 2 18:28:10 2019 +0000
  github.com/beorn7/perks:  v0.0.0-20180321164747-3a771d992973
  github.com/golang/protobuf:  v1.2.0
  github.com/matttproud/golang_protobuf_extensions:  v1.0.1
  github.com/povilasv/prommod:  v0.0.10
  github.com/prometheus/client_golang:  v0.9.2
  github.com/prometheus/client_model:  v0.0.0-20180712105110-5c3871d89910
  github.com/prometheus/common:  v0.0.0-20181126121408-4724e9255275
  github.com/prometheus/procfs:  v0.0.0-20181204211112-1dc9a6cbc91a

Personally, I find this very useful for development. You can ask your users to submit output of app --version, when submitting bug reports, which then allows you to quickly see all of the dependencies, as well as their versions.

Building prommod

Although this package is small and pretty trivial to implement, writing it was not so easy as it may seem, and I ended up using a lot of Go’s “hidden” features.

Build tags

I really wanted to make this package work on Go with versions lower than 1.12. Although Go modules were introduced in Go 1.11, the debug.ReadBuildInfocall, which is used by prommod, was only introduced in Go 1.12. Therefore, this wouldn’t work for users with older Go versions.

In order to solve this, I used Go build tags. For Go >=1.12 implementation I added Package comment // +build go1.12, which calls debug.ReadBuildInfo. For older Go versions I added // +build !go1.12 comment, and provided the same functions, with identical API. But, instead of calling debug.ReadBuildInfo, I would just return empty strings in prommod.Print("app_name") & prommod.Info() calls, and an empty Prometheus metric.

All in all, if you are using prommod with an older Go version, it will still compile, but won’t produce any useful output.

In order to continuously test this with different versions, I used Travis CI. And I think it’s the best tool for the job. My .travis.yml looks like this:

language: go

go:
  - "1.x"
  - "master"
  - "1.12"
  - "1.11"
  - "1.10"

As you can see it’s just a list of Go versions, which I build and test against, and it all just works ūüôā

Go examples are tests

When testing this library, I particularly liked writing Go Examples.
Not a lot of Go developers know, but Go examples are tests. In order to use Example test features, you simply need to print the resulting variable to Standard Output and check it with // Output: comment.

Take a look at this code snippet:

func ExampleInfo() {
    fmt.Println(prommod.Info())
    // Output: ()
}

In this case, Go will run this code as a test expecting prommod.Info() to be equal to (). And, if the output is not equal, go test will give you a nice error:

$ go test
--- FAIL: ExampleInfo (0.00s)
got:
bad_output
want:
()
FAIL

I encountered weird Go 1.12 behavior, as Go behaves like a non Go module program in testing. What happens is runtime/debug.ReadBuildInfo() doesn’t return any of the Go module information, although compiled with Go 1.12, which makes this library a bit harder to unit test.

Using init() functions

Another lesser known feature are the Go init() functions. Basically, init() functions are called once per package, and they’re used to set up state. What I wanted to do is to always return the computed information from a global variable, instead of actually computing everything on Print()/Info() call.

I think it makes a lot of sense for this library, as this information never changes after the initial build. This also has performance benefits, as this library is mostly just returning constant global strings.

Semantic Versions

Lastly, and most importantly, this package tags versions using Semantic Versioning. Go Modules depend on the community using Semantic Versioning. Please use it – especially if you are an author of a Go package! Pretty please? We are all going to benefit from it. ūüôā

Conclusion

Although I really like this approach of using the Prometheus monitoring system for storing dependency versions, I do think I might be hammering a nail with a screwdriver here. It feels like we are slightly abusing Prometheus, as metrics are “fake” (i.e. metric value never changes) and we just use labels to store information. Maybe it’s worth investigating other solutions, like Grafeas, for storing this type of information‚Äč.

On the other hand, I really like the Pull approach here. I like that it’s sort of an application telling you, “hey, I’m this thing and I use these dependencies to perform my function”, just like application in health checks tells you “hey I think I’m fine”, or “hey I can’t connect to DB”. Also, Prometheus information is as close to real-time as possible. This means that you can monitor application & library updates, live.

Most importantly, what do you think? I would really like to hear more opinions on this topic. And, by the way, if you want to support my work you can buy me a book here.

Thanks for reading! Quality open source is really important to me. prommod is available using MIT license, so feel free to use it. Just remember to star it on github if you really like the package. And, if you are missing something, please create an issue and then a PR if you feel like it.

yaml url.URL parsing

I was using gopkg.in/yaml.v2 to unmarshal a .yaml file into a struct and couldn’t find a solution to unmarshal an url string into url.URL.

Trying to just use url.URL  fails with:

Error initializing configuration: yaml: 
unmarshal errors:\n  line 54:
cannot unmarshal !!str `http://...` into url.URL

This is because, url.URL doesn’t adhere to yaml.v2.Unmarshaler interface.

So here is my solution, feel free to copy paste it!

type YAMLURL struct {
    *url.URL
}

func (j *YAMLURL) UnmarshalYAML(unmarshal func(interface{}) error) error {
    var s string
    err := unmarshal(&s)
    if err != nil {
        return err
    }
    url, err := url.Parse(s)
    j.URL = url
    return err
}

And here is how to use it:

type Config struct{
    url YAMLURL `yaml:"url"`
}

Thanks for reading!

Think about your dependencies

A little copying is better than a little dependency, says Go proverb.

Go has a huge standard library, which has support for many, many things. I think many developers, don’t utilize it fully. I mean I have¬†seen many new gophers¬†try to use packages for everything. For example, do you really need a dependency on github.com/stretchr/testify ? ¬†If you are just using assert,¬†maybe you¬†can just can copy the assert function from benbjohnson/testing? Ben Johnson has ¬†published these convenience functions (MIT license):

func assert(tb testing.TB, condition bool, msg string) 
func ok(tb testing.TB, err error) 
func equals(tb testing.TB, exp, act interface{}) 

If you are using the TestSuite¬†stuff, maybe you can just avoid it entirely? The same goes for all the cool HTTP muxes. It’s cool, but if you don’t have a lot of endpoints, maybe you can just avoid that dependency and use standard library’s¬†http¬†package.

But why am I against dependencies? Well, for new developers learning a new language is a lot, if you complicate this by adding custom packages and frameworks it will be harder. And it will be harder for you too. If you get back to a project after a break, you will notice that you have to relearn the dependencies you have used.

With less dependencies you will spend less time managing them. Once, I spent 3 hours fixing a dependency problem with Glide. The problem was that one of my dependency (let’s call this dependency A) had a dependency on an older version of a package C¬†and another dependency (let’s call it B) had a dependency on a newer version of a package C.¬†The biggest problem was that package¬†C¬†hadn’t used semantic versioning, so I had to try random library C commits and try to get this thing working… After doing that for an hour, I took a break and¬†realized, that I can easily rewrite the code and remove a dependency on B.

But aren’t packages from standard library are dependencies too? Well yes, but by utilizing the standard library, you are learning the language. You¬†will start to remember public functions, structs and interfaces, after awhile. Also, Go has version 1 compatibility guarantee, that means that standard library’s API won’t break down on you, for go 1.* versions.

Of course don’t go and reimplement Go MySQL driver or anything crazy like that. Just think before adding another dependency!

That’s all from me. Do you enjoy these articles? Follow me on twitter @PofkeVe¬†and join my mailing list¬†ūüôā

Go schema migration tools

In this post I did a comparison of following tools:

Summary

TL;DR If your looking for schema migration tool you can use:

mattes/migrate, SQL defined schema migrations, with a well defined and documented API, large database support and a useful CLI tool. This tool is actively maintained, has a lot of stars and an A+ from goreport.

rubenv/sql-migrate, go struct based or SQL defined schema migrations, with a config file, migration history, prod-dev-test environments. The only drawback is that it got B from goreport.

markbates/pop, use this if you are looking for an ORM like library. It has awesome model generation capabilities and a custom DSL language for writing schema changes.

go-gormigrate/gormigrate, use this if you are using GORM, this helper adds proper schema versioning and rollback capabilities.

Why these tools? Read below:

Comparing by Stars

mattes/migrate 961
rubenv/sql-migrate 605
markbates/pop 605
liamstask/goose – bitbucket project; stars not available
DavidHuie/gomigrate 110
pressly/goose 80
BurntSushi/migration 56
tanel/dbmigrate 38
GuiaBolso/darwin 29
go-gormigrate/gormigrate 22
pravasan/pravasan 14

Note: I put goose into 4th place because it has a lot of watchers.

I use Github stars as a metric to see how widely project is used. Because it is almost impossible to see how many users actually use it. I bolded out the tools that win in this category.

Comparing by last activity

markbates/pop Feb 14, 2017
mattes/migrate Feb 10, 2017
GuiaBolso/darwin Feb 10, 2017
rubenv/sql-migrate Feb 7, 2017
go-gormigrate/gormigrate Feb 4, 2017
pressly/goose Dec 9, 2016
DavidHuie/gomigrate Aug 9, 2016
tanel/dbmigrate Feb 23, 2016
pravasan/pravasan Mar 20, 2015
liamstask/goose Jan 16, 2015
BurntSushi/migration Jan 25, 2014

Note: I scrapped this info on Wednesday, February 15, 2017 3:00 pm EET. 

More often than not you want to use projects that are maintained. So last activity can be seen as a measure of project’s maintainability. It is¬†important because if there is a bug you want to have an ability to submit a PR or create an issue, which¬†hopefully would get resolved. Bolded out tools win in this category.

Comparing by goreportcard

markbates/pop A+
GuiaBolso/darwin A+
go-gormigrate/gormigrate A+
mattes/migrate A
liamstask/goose A
pressly/goose A
DavidHuie/gomigrate A
tanel/dbmigrate A
rubenv/sql-migrate B
BurntSushi/migration B
pravasan/pravasan D

Note: I scrapped this info on Wednesday, February 15, 2017 3:00 pm EET. 

Goreportcard allows you to check the code quality of any open source project written in go and gives an overall score (A+, A, B,..). I bolded out the tools that win in this category.

Comparing by usability

In this comparison I will try out some of the tools and give my opinion.

I will be playing around with a table MyGuests, which looks like this:

CREATE TABLE MyGuests (
   id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
   firstname VARCHAR(30) NOT NULL,
   lastname VARCHAR(30) NOT NULL,
   email VARCHAR(50),
   reg_date TIMESTAMP
)

mattes/migrate

This is a simple tool, which does¬†migrations based on files. It comes with a Go library and a CLI¬†tool, which helps you to¬†create SQL migration¬†files and manages schema version. Let’s take a look at the example usage of CLI tool below:

$ migrate -url mysql://root@tcp(127.0.0.1:3306)/mattes -path ./migrations 
 create initial_users_table

creates 2 files and a table called schema_migrations:

Version 1487240220 migration files created in ./migrations:
1487240220_initial_users_table.up.sql
1487240220_initial_users_table.down.sql

I added CREATE TABLE MyGuests …¬†statement in file called *up.sql and DROP TABLE¬†MyGuests..¬†statement in *down.sql

Running migration using CLI is simple:

$ migrate -url mysql://root@tcp(127.0.0.1:3306)/mattes 
  -path ./migrations up

This creates a table and sets a row in schema_migrations table:

mysql> select * from schema_migrations;
+------------+
| version |
+------------+
| 1487240220 |
+------------+
1 row in set (0.00 sec)

Here is an example of running ¬†a “down” migration, which drops the table and removes a row from schema_migrations:

$ migrate -url mysql://root@tcp(127.0.0.1:3306)/mattes 
  -path ./migrations down

CLI tool also allows going to specific schema version,  rollbacking previous n migrations, etc.

The provided go library is also pretty simple, it allows you to run migrations from your code and provides you with synchronous and asynchronous implementations. Probably you will only be using UpSync function from your code. Take a look at the example below:

package main

import (
   "fmt"
   _ "github.com/mattes/migrate/driver/mysql"
   "github.com/mattes/migrate/migrate"
)

func main() {
   fmt.Println("Hello")
   allErrors, ok := migrate.UpSync("mysql://root@tcp(127.0.0.1:3306)/mattes", "./migrations")
   if !ok {
      fmt.Println(allErrors)
   }
}

I like¬†this library for it’s simplicity. It supports PostgreSQL, Cassandra, SQLite, MySQL, Neo4j, Ql, MongoDB, CrateDb. But it has a¬†caveat: MySQL support is only experimental.

liamstask/goose

Playing around with this library was a bit painful for me. For about 20 minutes I couldn’t figure out what was wrong with my connection info.¬†I was continuously getting Invalid DBConf¬†errors, with no explanations:

2017/02/16 13:14:54 Invalid DBConf: 
   {mysql  root@tcp(127.0.0.1:3306)/goose  }

It appears to me now that I had left a space after specifying database type!

So in goose you have to create a dir called db and add a file called dbconf.yaml , which contains connection information. This is how my file looked:

development: 
  driver: mysql
  open: root@tcp(127.0.0.1:3306)/goose

In this config you are also allowed to choose your SQL dialect and import a different db driver.

Creating a migration with goose is easy:

goose create initial_users_table

which creates a  file called 20170216132820_initial_users_table.go, which contains 2 go functions:

func Up_20170216132820(txn *sql.Tx) {

}

func Down_20170216132820(txn *sql.Tx) {

}

Here is how I filled these functions:

// Up is executed when this migration is applied
func Up_20170216132820(txn *sql.Tx) {
   res, err := txn.Exec(`CREATE TABLE MyGuests (
      id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
      firstname VARCHAR(30) NOT NULL,
      lastname VARCHAR(30) NOT NULL,
      email VARCHAR(50),
      reg_date TIMESTAMP
   )`)
  fmt.Println(res)
  fmt.Println(err)
}

// Down is executed when this migration is rolled back
func Down_20170216132820(txn *sql.Tx) {
   res, err := txn.Exec("DROP TABLE MyGuests;")
   fmt.Println(res)
   fmt.Println(err)
}

Executing up/down migrations is also easy:

goose up

goose down

Internally goose maintains a table called goose_db_version:

 mysql> select * from goose_db_version;
+----+----------------+------------+---------------------+
| id | version_id | is_applied | tstamp |
+----+----------------+------------+---------------------+
| 1 | 0 | 1 | 2017-02-16 13:37:20 |
| 2 | 20170216132820 | 1 | 2017-02-16 13:37:47 |
| 3 | 20170216132820 | 0 | 2017-02-16 13:39:32 |
| 4 | 20170216132820 | 1 | 2017-02-16 13:40:04 |
| 5 | 20170216134743 | 1 | 2017-02-16 13:51:30 |
| 6 | 20170216134743 | 0 | 2017-02-16 13:51:34 |
+----+----------------+------------+---------------------+
6 rows in set (0.00 sec)

This tools also allows you to specify migration using SQL files, by default this tool supports postgres, mysql, sqlite3 and has a go library. Mostly I liked that you specify connection info in a config file, which simplifies your work with the CLI. Also, writing db migrations as go code looks interesting! Overall, not a bad tool.

markbates/pop

pop is more like an “ORM”, which helps you to create models and sql schema for you. pop also comes with migration capabilities in a CLI tool called¬†soda and a DSL for specifying migrations called fizz.

At the start you have to specify database connection config in a database.yaml file. Mine looked like this:

development:
  dialect: "mysql"
  database: "pop"
  host: "localhost"
  port: "3306"
  user: "root"
  password: ""

Then you can create/drop a database using CLI tool:

soda create -e development
soda drop -e development

Generating a model with it’s migration script based on fizz¬†DSL is simple:

soda generate model MyGuest firstname:text lastname:text email:text 
  reg_date:timestamp

Generated DSL looks like this:

create_table("my_guests", func(t) {
   t.Column("id", "uuid", {"primary": true})
   t.Column("firstname", "text", {})
   t.Column("lastname", "text", {})
   t.Column("email", "text", {})
   t.Column("reg_date", "timestamp", {})
})

and model:

type MyGuest struct {
        ID uuid.UUID `json:"id" db:"id"`
        CreatedAt time.Time `json:"created_at" db:"created_at"`
        UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
        Firstname string `json:"firstname" db:"firstname"`
        Lastname string `json:"lastname" db:"lastname"`
        Email string `json:"email" db:"email"`
        RegDate time.Time `json:"reg_date" db:"reg_date"`
}

Migrate up/down:

soda migrate up 
soda migrate down

On migration Fizz DSL produced the following schema:

mysql> desc my_guests;
+------------+----------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+----------+------+-----+---------+-------+
| created_at | datetime | NO | | NULL | |
| updated_at | datetime | NO | | NULL | |
| id | char(36) | NO | PRI | NULL | |
| firstname | text | NO | | NULL | |
| lastname | text | NO | | NULL | |
| email | text | NO | | NULL | |
| reg_date | datetime | NO | | NULL | |
+------------+----------+------+-----+---------+-------+
7 rows in set (0.00 sec)

Internally this tool maintains a table called schema_migration, which holds schema version number.

I liked this library a lot, but it feel like you should only use this then you are looking for “ORM” like library. Generating models and migrations looks cool! Also, bonus points for Fizz DSL, which looks a lot like go ūüôā

One drawback is that pop supports only : PostgreSQL (>= 9.3), MySQL (>= 5.7) and SQLite (>= 3.x).

go-gormigrate/gormigrate

Gormigrate is a migration helper for GORM library. This helper adds proper schema versioning and rollback cababilities. I like schema versioning + schema migration definition in a list of structs. This is how this looks with MyGuests example:

func main() {
        db, err := gorm.Open("mysql",
    "root@tcp(127.0.0.1:3306)/gorm?charset=utf8&parseTime=True&loc=Local")
        if err != nil {
                panic("failed to connect database")
        }

        if err = db.DB().Ping(); err != nil {
                log.Fatal(err)
        }

        db.LogMode(true)

        defer db.Close()

        m := gormigrate.New(db, gormigrate.DefaultOptions, 
             []*gormigrate.Migration{
                {
                        ID: "201702200906",
                        Migrate: func(tx *gorm.DB) error {
                                type MyGuest struct {
                                        gorm.Model
                                        Firstname string
                                        Lastname  string
                                        Email     string
                                        RegDate   time.Time
                                }
                                return tx.AutoMigrate(&MyGuest{}).Error
                        },
                        Rollback: func(tx *gorm.DB) error {
                                return tx.DropTable("MyGuest").Error
                        },
                },
        })

        if err = m.Migrate(); err != nil {
                log.Fatalf("Could not migrate: %v", err)
        }
}

This migration creates a table, with the following schema:

mysql> desc my_guests;
+------------+------------------+------+-----+---------+----------------+
| Field      | Type             | Null | Key | Default | Extra          |
+------------+------------------+------+-----+---------+----------------+
| id         | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| created_at | timestamp        | YES  |     | NULL    |                |
| updated_at | timestamp        | YES  |     | NULL    |                |
| deleted_at | timestamp        | YES  | MUL | NULL    |                |
| firstname  | varchar(255)     | YES  |     | NULL    |                |
| lastname   | varchar(255)     | YES  |     | NULL    |                |
| email      | varchar(255)     | YES  |     | NULL    |                |
| reg_date   | timestamp        | YES  |     | NULL    |                |
+------------+------------------+------+-----+---------+----------------+
8 rows in set (0.00 sec)

I would definitely use this with GORM.

rubenv/sql-migrate

sql-migrate is a look like goose, you specify connection info in a yaml file, migrations are written in SQL files. It has a CLI, which generates a template for your migration:

$ sql-migrate new MyGuests

My schema change looked like this:

-- +migrate Up
CREATE TABLE MyGuests (
           id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
           firstname VARCHAR(30) NOT NULL,
           lastname VARCHAR(30) NOT NULL,
           email VARCHAR(50),
           reg_date TIMESTAMP
);

-- +migrate Down
DROP TABLE MyGuests;

Applying migration with CLI tool is pretty straightforward:

$ sql-migrate up
Applied 1 migration

It also has nice looking library, with well defined API, which supports having migrations in a struct or in a directory. The only drawback I can think of is that goreport card gave a B for this library.

DavidHuie/gomigrate

Is a really simple toolkit, which only allows you to run migrations from go code:

err := migrator.Migrate()
err := migrator.Rollback()

migration are defined in sql files named { id }}{{ name }}{{ “up” or “down” }}.sql, which you have to manage yourself, because¬†it’s only a library.¬†mattes/migrate seems to cover the same functionality and add much more, so I would prefer to use it over this library.

GuiaBolso/darwin

It’s a library, which tracks schema changes in a struct¬†and only allows up migrations. I love the idea of storing all migrations in a slice:

var (
   migrations = []darwin.Migration{
   {
   Version: 1,
   Description: "Creating table MyGuests",
   Script: `CREATE TABLE MyGuests (
              id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
              firstname VARCHAR(30) NOT NULL,
              lastname VARCHAR(30) NOT NULL,
              email VARCHAR(50),
              reg_date TIMESTAMP
            )`,
   },
   }
)

func main() {
   database, err := sql.Open("mysql", "root@tcp(127.0.0.1:3306)/darwin")

   if err != nil {
     log.Fatal(err)
   }

   driver := darwin.NewGenericDriver(database, darwin.MySQLDialect{})

   d := darwin.New(driver, migrations, nil)
   err = d.Migrate()

   if err != nil {
     log.Println(err)
   }
}

I don’t think there is more to say about this library, but I guess it’s a good thing.

tanel/dbmigrate

Simple library for PostgreSQL or Cassandra. Runs migrations (.sql or .cql files) sorted using their file name. mattes/migrate seems to cover the same functionality and add much more, so I would prefer to use it over this.

pressly/goose

It’s a fork of goose, which drops support for config files and custom drivers.¬†Migrations can be run with any driver that is compatible with database/sql. This tool looks like¬†liamstask/goose¬†and mattes/migrate¬†had a baby ūüôā It takes good things from both projects: good CLI, no configuration, write migrations using .sql or .go files,¬†store history of migrations in a goose_db_version table. But there are some things that this tools lacks: it doesn’t support Cassandra or any other non SQL database, CLI can migrate SQL files only.

Then trying to use it I had problems with this tool:

I created SQL based migration:

$ goose mysql "root@tcp(127.0.0.1:3306)/pressly" create initial_users_table sql
Created sql migration at 20170301085637_initial_users_table.sql

Filled 20170301085637_initial_users_table.sql file with:

-- +goose Up
-- SQL in section 'Up' is executed when this migration is applied

CREATE TABLE MyGuests (
    id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    firstname VARCHAR(30) NOT NULL,
    lastname VARCHAR(30) NOT NULL,
    email VARCHAR(50),
    reg_date TIMESTAMP
)


-- +goose Down
-- SQL section 'Down' is executed when this migration is rolled back
DROP TABLE MyGuests;
$ goose mysql "root@tcp(127.0.0.1:3306)/pressly" up
2017/03/01 08:57:41 WARNING: Unexpected unfinished SQL query: -- +goose Up
-- SQL in section 'Up' is executed when this migration is applied

CREATE TABLE MyGuests (
    id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    firstname VARCHAR(30) NOT NULL,
    lastname VARCHAR(30) NOT NULL,
    email VARCHAR(50),
    reg_date TIMESTAMP
). Missing a semicolon?
OK    20170301085637_initial_users_table.sql
goose: no migrations to run. current version: 20170301085637
$ goose mysql "root@tcp(127.0.0.1:3306)/pressly" down
2017/03/01 08:57:47 FAIL 20170301085637_initial_users_table.sql (Error 1051: Unknown table 'pressly.myguests'), quitting migration.
$ goose mysql "root@tcp(127.0.0.1:3306)/pressly" up
goose: no migrations to run. current version: 20170301085637

I did miss the semicolon, but this tool wrote into it’s table that migration ran successfully, so I was left at state where table doesn’t exist, but tool thinks it’s exist.

mysql> select * from goose_db_version;
+----+----------------+------------+---------------------+
| id | version_id     | is_applied | tstamp              |
+----+----------------+------------+---------------------+
|  1 |              0 |          1 | 2017-03-01 08:44:45 |
|  2 | 20170301085637 |          1 | 2017-03-01 08:57:41 |
+----+----------------+------------+---------------------+

I had to manually delete the row in goose_db_version the table and restart the migration…
After that everything worked normally:

$ pressly goose mysql "root@tcp(127.0.0.1:3306)/pressly" up
OK    20170301085637_initial_users_table.sql
goose: no migrations to run. current version: 20170301085637
$ pressly goose mysql "root@tcp(127.0.0.1:3306)/pressly" down
OK    20170301085637_initial_users_table.sql

Also I tried to run migration based on go code, but wasn’t successful:
I created the migration and compiled the example cmd file provided in the repo for running migrations:

$ goose mysql "root@tcp(127.0.0.1:3306)/pressly" create initial_users_table
 Created go migration at 20170301083810_initial_users_table.go

and filled with my migration code:

package migration

import (
    "database/sql"
    "fmt"

    "github.com/pressly/goose"
)

func init() {
    goose.AddMigration(Up_20170301083810, Down_20170301083810)
}

func Up_20170301083810(tx *sql.Tx) error {
    res, err := tx.Exec(`CREATE TABLE MyGuests (
           id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
           firstname VARCHAR(30) NOT NULL,
           lastname VARCHAR(30) NOT NULL,
           email VARCHAR(50),
           reg_date TIMESTAMP
       )`)
    fmt.Println(res)
    return err
}

func Down_20170301083810(tx *sql.Tx) error {
    res, err := txn.Exec("DROP TABLE MyGuests;")
    fmt.Println(res)
    return err
}

Then I tried to run:

./pressly --dir=migrations/ mysql "root@tcp(127.0.0.1:3306)/pressly" up
2017/03/01 09:02:24 FAIL 00002_rename_root.go 
(Error 1146: Table 'pressly.users' doesn't exist), quitting migration.

my migrations directory contains only 1 file called 20170301083810_initial_users_table.go, there is no file called 00002_rename_root.go, so I don’t know what the hell this tool is doing, but I really don’t like that it tries to run file called rename_root.go, which I didn’t write and don’t know nothing about.

So be careful with this tool!