Articles I recommend

Design

Solid go design by Dave Chaney

Goa: New Approach to Microservice Development in Go

  • Amazing way to design and generate boilerplate code: such as Rest API, validation, swagger UI, …

Functional options for friendly apis by Dave Cheney

  • tl;dr; make use variadic Functional options for your API, not huge list of params or Config object
  • Example:
func NewServer(uri string, options ...func(*Server)) (*Server, error){
  srv := Server{}
  ...
  for _, option := range options {
     option(&srv)
  }
}

func lookup(t *mapType, m*mapHeader, k unsafe.Pointer) unsafe.Pointer
  • You can also provide Public functions as configuration options

Do not fear first class functions by Dave Cheney

  • First class functions means an ability to treat functions as regular values.
  • There is a duality of a first class function and an interface with one method.
  • Function values allow you to pass behaviour to be executed, not the data.
  • Cool example:
func (m *Mux) SendMsg(msg string) error {
  result := make(chan error, 1)
  m.ops <- func(m map[net.Addr]net.Conn) {
    for _, conn := range m.conns {
      err := io.WriteString(conn, msg)
      if err != nil {
        result <- err
        return
      }
    }
    result <- nil
  }
  return <-result
}
  • This hides away error handling from ops channel, which executes the func(m map[net.Addr]net.Conn) function, which is pretty neat.
  • Use them in moderation!

Self-referential functions and the design of options by Rob Pike

  • Interesting way to do options that return previous options with self referential functions.

Don’t just check errors, handle them gracefully by Dave Cheney

  • If you need to check for error, never check err.Error()
  • checking error type via type assertion also has problems.
  • It’s better just to make an interface. Example:
type temporary interface {
        Temporary() bool
}
 
// IsTemporary returns true if err is temporary.
func IsTemporary(err error) bool {
        te, ok := err.(temporary)
        return ok && te.Temporary()
}
  • Wrap errors with errors package
  • errors.Print(err) pretty prints stack trace.
  • Sometimes you need to unwrap the error, this can be done via errors.Cause(err)
  • Example:
// IsTemporary returns true if err is temporary.
func IsTemporary(err error) bool {
        te, ok := errors.Cause(err).(temporary)
        return ok && te.Temporary()
}

A theory of modern Go by Peter Bourgon

  • Don’t use Package-global objects
  • Move each of the dependencies into signature as parameter.
  • It makes it easier to know the scope of the function.
  • For constructors making each parameters as an interface would capturing only the methods we use.
  • This way we allow callers to swap in alternative implementations.
  • Rules from Dave Cheney:
    • No package level variables
    • No func init

Go, without package scoped variables by Dave Cheney

  • Public vars in standard library:
    • Errors io.EOFsql.ErrNoRowscrypto/x509.ErrUnsupportedAlgorith are vars, can be consts though
  • Registration pattern:
    • Libs:
      • net/http,
      • database/sql,
      • flag,
      • log.
    • This way go register image decoders, database drivers and cryptographic schemes.
    • There is a problem with net/http/pprof package which registers via a side effect with net/http.DefaultServeMux.
  • Interface satisfaction assertions:
    • var _ SomeInterface = new(SomeType)
    • should be moved to tests.
  • Conclustions:
    • Firstly, public var declarations should be eschewed.
    • Public package var declarations are used, the type of those variables should be carefully constructed to expose as little surface area as possible

Go: Best Practices for Production Environments by Peter Bourgon

  • Don’t create project structure until you demonstrably need it.
  • go fmt on save
  • Avoid named return parameters.
  • Avoid make and new, unless they’re necessary or we know the size of the allocated thing in advance
  • Use struct{} as a sentinel value, rather than bool or interface{}.
  • For example, a set is map[string]struct{}; a signal channel is chan struct{}.
  • It unambiguously signals an explicit lack of information.
  • Use flags for config.
  • Define them in your func main.
  • Metrics pull model with expvar or expvar-style metrics.
  • Use plain package testing
  • For integration tests do a integration_test.go, and give it a build tag of integration.
  • When you do this Run this
    Save go fmt (or goimports)
    Build go vet, golint, and maybe go test
    Deploy go test -tags=integration
  • The ultimate best practice is to embrace simplicity

Go best practices, six years in by Peter Bourgon

  • Layout
    • go-build-template
    • standard-package-layout
    • Two top-level directories, pkg and cmd
    • And you have space and isolation for non-Go assets.
    • For example, Javascript can live in a client or ui subdirectory.
    • Dockerfiles, continuous integration configs, or other build helpers can live in the project root or in a build subdirectory.
    • And runtime configuration like Kubernetes manifests can have a home, too.
    • Always use fully-qualified import paths.
    • Defer to Andrew Gerrand’s naming conventions.
    • Avoid nil checks via default no-op implementations.
    • Make dependencies explicit!
    • Loggers are dependencies, just like references to other components, database handles, command line flags, etc.
  • Logging and instrumentation:
    • Log only actionable information, which will be read by a human or a machine.
    • Use structured logging.
  •  Metrics
    • Use Does instead of http.Client:
    • type Doer interface {
          Do(*http.Request) (*http.Response, error)
      }
      
  • Testing
    • Use many small interfaces to model dependencies.
  • Dependencies
    • Libraries should never vendor their dependencies.
  • Video

Twelve Go Best Practices – Francesc Campoy

  • slides
  • Tip: early return (Avoid nesting by handling errors first)
  • Deploy one-off utilitity types for Simpler code (binWriter)
  • Type switch with short variable declaration: switch x := v.(type)
  • Function adapters (e.g. Error Handler) func errorHandler(f func(http.ResponseWriter, *http.Request) error) http.HandlerFunc
  • Write important info first: License information, build tags, package documentation
  • Document your code
  • Shorter is better
  • Prefer MarshalIndent to MarshalWithIndentation
  • Package name appears in import path
  • Avoid very long files
  • It’s convention to create a doc.go  containing the package documentation
  • Two directories cmd with all the CMD and pkg
  • Smallest interface as you can as your function input
  • Keep independant packages independant
  • Have as little dependencies as you can
  • Avoid concurrency in your API
  • Avoid goroutine leak

Let’s talk about loggin by Dave Cheney

  • Don’t use log.Fatal, as it calls os.Exit(1), which doesn’t close buffers, removes tmp files, etc.
  • The log.Debug() and log.Info is preffered logging package.

Concurrency:

Go Concurrency Patterns: Pipelines and cancellation by Sameer Ajmani

select {
            case DO_WORK:
                ...
            case <-done:
                return
}
  • Also an interesting to make channel return custom struct, which has err in it:
type result struct {
    stuff string
    moreStuff string
    err  error
}

Naming:

What is in the Name? by Andrew Gerrand

Package names by Sameer Ajmani

Testing:

Go advanced tips and tricks by me

Testing Techniques by Andrew Gerrand

Tools

Go tool trace by Pusher

  • Use it to understand what your program is doing over time or to diagnose contention issues.
  • Go tool pprof
  • Use go tool pprof to track down slow functions and where your program is spending most of its CPU time.
  • Use env GODEBUG=scheddetail=1,schedtrace=1000 ./program to trace the scheduler. Explained in Scheduler tracing by William Kenedy

CPU, Memory, Blocking, Goroutine Profiling & GC, Memory Allocator, Scheduler Tracing by Dmitry Vyukov.

  • Garbage collector trace:
  • GODEBUG=gctrace=1 ./program
  • Memory allocator trace:
  • GODEBUG=allocfreetrace=1 ./program
  • Simple Scheduler trace:
  • GODEBUG=schedtrace=100 ./program
  • Escape analysis trace:
  • go build -gcflags=-m

Navigating Unfamiliar Code with the Go Guru by Alan Donovan

  • Go guru (original name oracle)
  • Using go guru
  • Features:
  • auto highlight variable
  • go to definition
  • GoRefferers show who references variable
  • Free variables tells what variables would be function parameters if it would be moved
  • Free names/Free identifiers
  • GoFreevars
  • GoDescribe – describes size of type, methods, interfaces it implements, and fields, a const, or package
  • GoImplements – shows who implements given interface
  • GoWhicherrs – shows what type of errors could happen here.

Dynamic Tools by Dmitry Vyukov

Data race detector
  • It occurs when two goroutines access same variable concurrently and at least one of the accesses is a write.
  • Any data race can destroy correctness, security of the program.
  • There are no “benign” data races
  • Compiler assumes that your code is race free and does aggressive optimization (assume “ownership” over written-to variables)
  • To enable data race detector: go run -race, go build -race, go install -race,go test -race
  • How it works?
  • Compiler instrumentation pass enabled by -race:
    func foo(p *int) {
      *p = 1
    }
    
  • Becomes:
    func foo (p *int){
      runtime.funcenter(caller_pc)
      runtime.racewrite(p)
      *p = 1
      runtime.funcexit()
    }
    
  • Runtime module handles: memory access, synchronization, function calls, goroutine creation/exit
  • Algorithm is based on dynamic modelling of happens-before relation
  • Do:
    • Write good concurrency tests
    • run continuous build with race enabled
    • run integration tests
    • run race-enabled canaries in production
Go-fuzz
    • No need to write any tests!
    • Randomized testing
    • Generate random blob of data -> feed into program -> see if it crashes -> profit
    • Useful for programs that parse complex data
    • The whole idea is to generate a lot of random data
    • Coverage-guided fuzzing (genetic algorithm)
    • Go fuzz has Sonar: finds all comparison operations and passes them into runtime
    • Sonar does low-level operation (bit-flipping)
    • Versifier: tries to reverse engineer the protocol of file and then does structural mutations to input
    • Slides: https://talks.golang.org/2015/dynamic-tools.slide#1
    • Example test:
      func Foo(data []byte) int{
        //Do something with bytes. Ex:
        gob.NewDecoder(bytes.NewReader(data))
        return 0
      }
      

 

 

 

  • build:
    go-fuzz-build
  • collect corpus
  • run:
    go-fuzz -bin=gob-fuzz.zip -workdir=examples/gob
    
Execution tracer
  • A part of 1.5 release
  • Captures with nano second precision:
    • goroutine creation/start/end
    • goroutine blocking/unblocking
    • network blocking
    • sys calls
    • GC events

Cover tool by Rob Pike

  • Rewrite the package’s source code before compilation to add instrumentation, compile and run the modified source, and dump the statistics.
  • What cover tool does is inserts the code to increment counters at the start of each basic block.
  • Its runtime overhead is modest, adds only about 3% when running a typical test.
  • Rewriting is done via cover tool which is launched by go test
  • go test -x shows what is happening
  • A big advantage of this source-level approach to test coverage is that it’s easy to instrument the code in different ways.
  • We can ask not only whether a statement has been executed, but how many times.
  • go test -covermode set,count,atomic:
    • set: did each statement run?
    • count: how many times did each statement run?
    • atomic: like count, but counts precisely in parallel programs

Debugging Go core dumps

  • Ulimit must be set before starting:
  • ulimit -c unlimited
  • Core files is a file that contains the memory dump of a running process and its process status.
  • GOTRACEBACK=crash ./app
  • Makes ctrl+\ crash the program and core dump file written.
  • gcore PID , makes core dump without crashing the process.
  • dlv core ./app core.PID, to debug core dump via delve

Libraries, Frameworks

Go build modes by David Crawshaw 

Introduction to gRPC by Shiju Varghese

  • gRPC is a high performance, open-source remote procedure call (RPC) framework.
  • uses protobufs
  • principles
  • supports: Request/Response model, server-side streaming RPC, client-side streaming RPC, bidirectional streaming RPC.

General

Writing go tool by Fatih Arslan

Looking inside a Race Detector by Kavya Joshi 

  • Go race detector is a happens-before detector.
  • Implemented via vector clocks.
  • race detector catches different events (goroutine created, memory accessed, etc) and has a state machine for each goroutine
  • Race detector inserts calls to runtime: runtime.raceread(), runtime.racewrite(),runtime.racefuncexit().
  • Mutex.go has race.Aquire()
  • Racedetector is based on ThreadSanitizer library (TSan) (C++)

Understanding Nil by Francesc Campoy

  • nil means zero
  • zero values
    bool false
    number 0
    string “”
    pointers, slices, channels, maps, functions, interfaces nil
    structs all it’s internal components are zero value.
  • nil has no type.
  • nil is not a keyword it is a predclared identifier.
  • Tip: do not declare concrete error vars
  • kinds of nil:
    pointers point to nothing
    slices have no backing array
    maps are not initialized
    channels are not initialized
    functions are not initialized
    interfaces have no value assigned, not even a nil pointer
  • Tip: nil pointer receivers are useful
  • slices: var s []slice
    len(s) 0
    cap(s) 0
    for range s zero times
    s[i] panic: index out of range
  • maps var m map[t]u
    len(m) 0
    for range m zero times
    v, ok := m[i] zero(u), false
    m[i] = x panic: assignment to entry in nil map
  • nil maps – read only, empty maps
  • channels var c chan t
    <- c blocks forever
    c <- x blocks forever
    close(c) panic: close of nil channel
  • closed channels var c chan t
    v, ok <- c zero(t), false
    c <- x panic: send on closed channel
    close(c) panic: close of nil channel
  • Tip: use nil channels to disable a select case
  • nil funcs used for lazy initialization
  • nil can imply default behaviour
  • The nil interface
  • The nil interface used as signal, if err != nil
  • Tip: nil *Person is not equal to nil interface
  • Tip: nil values can satisfy interfaces.
  • Tip: use nil interface to signal default
  • Nil is useful
    pointers methods can be called on nil receivers
    slices perfectly valid zero values
    maps perfect as read-only values
    channels essential for concurrency patterns
    functions needed for completeness

Go at Google: Language Design in the Service of Software Engineering by Rob Pike

Gophercon 2015 videos

Concurrency is not parallelism by Rob Pike

Five things that makes Go fast by Dave Cheney

Go has both make and new functions, what gives ? by Dave Cheney

        • make returns value not a pointer
        • new(T) always returns a *T pointing to an initialised T
var x1 *int
var x2 = new(int)
        • x2 can be dereferenced safely, while x1 not.

Inside the Map Implementation by Keith Randall

  • Slides
  • Associative containers mapping keys to values
  • key type must have == operation (no maps, no slices, no funcs)
  • value can be anything
  • Operations run in constant expected time.
  • Hash functions split the data into buckets (non crypto)
  • Required for correctness:
    • deterministic: k1 == k1 -> h(k1) == h(k2)
    • keys are uniform P[h(k) == b]  ~= 1/#buckets
    • Adversary safe: hard for attacker to find lots of k with h(k) == b
    • Such functions exist.
    • Go takes care of hash functions (you cannot choose it by yourself)
    • 8 slots for data.
    • overflow pointer points to another bucket if we need more than 8 slots
  • map actually points to map header, which contains (len, log(#buckets), bucket array, hash seed,…)
  • log(#buckets) – because buckets are power of 2 (this field is smaller that way)
  • if v = m[k] would compile to runtime.lookup(m, k) we wouldn’t know the type, so we needs Generics!
  • So thats why Go fakes generics:
  • All operations are done using unsafe.Pointer
  • pointer to key, pointer to value
  • also type information stored in type descriptor (size, equal function)
  • mapType has 2 fields
  • so actually v = m[k] compiles to :
    pk := unsafe.Pointer(&k)
    pv := runtime.lookup(typeOf(m),m,pk)
    v = *(*V)pv
    
  • where runtime function look like:
    func lookup(t *mapType, m *mapHeader, k unsafe.Pointer) unsafe.Pointer
  • Shows lookup implementation
  • Insert/delete looks similiar
  • On insert when bucket get too full we need to grow the map.
  • “too full”  = average of 6.5 entries per bucket.
    1.  Allocate a new array of buckets of twice the size
    2. Copy entries over (copying is done incrementally, not all at once)
    3. Use the new buckets.
  • In go you can modify maps in iteration.
  • You cannot provide equality operation
  • Go uses adversary safe hash functions.
  • Speed compares to java & c++
  • Go maps typically uses less memory
  • Space overhead about 2x, compared to raw data.
  • Access time on the order of ~100 cycles.

Understanding channels  by Kavya Joshi

  • buffered channel
  • ch := make(chan Task, 3)
  • ch: goroutine – safe, stores up to capacity elements, provides FIFO semantics, can cause block or unblock
  • Actually make initializes hchan struct and returns pointer to it.
  • hchan is made of:
    • buf – circlular queue
    • sendx – send index
    • recv – index
    • lock – mutex
    • sendq – waiting sender
    • recvq – waiting receivers
  • ch <- Task0 (send) happens in:
    1. Acquiring lock
    2. Enqueue with a memcopy to a buf (if not full)
    3. Releasing lock
  • t := <-ch (receive) happens in:
    1. Acquiring lock
    2. Dequeue with a memcopy from a buf to t
    3. Releasing lock
  • So channel is shared a cross goroutines, but chan values are copied.
  • What happens when sending to a full channel? ch <- Task4
  • Sender parks it self calling gopark (which sets goroutine’s status to waiting, removes association with M)
  • It turns out sender also modifies hchan by adding it’s g and element to sendq
  • The next t:= <-ch will pop of the sendq, copy element to a buffer and make a call into Go runtime goready(g)
  • What happens when receiving from an empty channel? t := <-ch
  • Receiver will add itself to a recvq and call gopark on itself.
  • The next send ch <- Task1 will pop of recvq, directly copy Task1 to waiting goroutines stack and call goready(waitingG)
  • This trick is clever, receiving goroutine doesn’t need to acquire lock and manipulate the buffer.
  • Unbuffered channel always work by “direct send” case:
  • receiver first -> sender writes to receiver’s stack
  • sender first -> receiver receives directly from sudog.
  • Select
  • all channels are locked
  • a sudog is put into sendq/recvq of all channels.
  • channel is unlocked, selecting is paused.
  • CAS operation so there’s one winning case.
  • resuming mirrors the pause sequence.

Forward Compatible Go Code

  • Slides
  • Encoding & other packages can produce unstable output:
    • archive/{tar,zip}
    • compress/{flate,gzip,lzw,zlib}
    • encoding/{csv,gob,json,xml}
    • image/{gif,jpeg,png}
    • net/http
    • math/rand
    • sort
  • the output is still compatible with format specification, ex. json or xml
  • For example, go used to store floats using exponential notation, which they switched to decimal notation
  • Not all variables are comparable. 
  • For example:
    • time.Time
    • errors
    • reflect.Value
    • Any types you don’t own
  • Then marshaling time.Time monotonic information is lost and the same time won’t be equal using ==.
  • Also map[time.Time]Record{} doesn’t work, better use map[int64]Record{} and time.NanoSecond().
  • Don’t rely on runtime details:
    • Order of goroutine scheduling
    • Iteration order of maps
    • How long functions take
    • The exact text of panic messages
  • Unsafe is not forward compatible
  • Race detector now randomizes scheduling.
  • Tips:
    • Read the documentation!
    • Be careful of what you hardcode
    • Use the right comparison
    • Use the race detector
    • Be willing to update unsafe code

My Go Resolutions for 2017 by Rus Cox

Go At Google by Rob Pike (2013)

Notes:

  • Go is here to make our life better
  • How does the language can help software engineering?
  • Go is not to be bleeding edge.

Speed of compilation:

  • C’s/C++’s ifdef guard problems.
  • Plan9 solves it by having developer to explicitly write out dependencies in docs.
  • Hugest problem of compilation dependencies for C/C++ are dependencies

Dependencies in Go:

  • Explicit, clear, computable
  • Unused dependencies are errors.
  • We traverse each file once.
  • If A depends on B, which depends on C, go does this trick that B object file contains information that it needs from C (This way go reads exponentially less files).
  • Circular dependencies are disallowed.
  • Great effort was spent to minimize dependencies in Go’s standard library. Copying little part of code is better than including, for example net package has it’s own itoa.
  • Package paths are unique, names are not.

Syntax

  • Clean syntax
  • 25 keywords
  • straightforward to parse
  • Pascal/Modula-style; name before type.
  • Go tried to play around with export keyword and different types for public/private visibility.
  • The current upper case solution is very simple and very easy to use.
  • Scope: universe -> package -> file -> function ->block

Semantics

  • very C like feels like procedural.
  • ++ are not expressions.
  • array bound checking (always)

Garbage collector

  • Developer chooses layout of memory via Structs
  • Heaps tend to be smaller, memory pressure is significantly better than Java
  • Uses parallel mark and sweep algorithm.

Interfaces

  • You should compose, not inherit
  • Go is OOP, but does not has classes or subtype inheritence
  • Rob Pike things that: Type hierarchy breaks OOP.
  • Traditional OOP leads to: over-design, abstraction very early.
  • Designing your type hierarchy is hard.
  • Go Interface: nobody has to declare that they implement that interface.

Errors

Hidden Pragmas of Go By Dave Cheney

  • Pragmas in other languages C:  #pragma pack(2); perl/js: “use strict;”

  • Go doesn’t have macros or preprocessor

  • Go has pragmas implemented in compiler by source code comments

  • Pragmas are not part of the language spec!

  • List of pragmas available in source code.
  • Syntax //go:directive
  • //go:noescape
  • no escape does not allow for variable to escape into heap
  • you can view variables escaping by:
  • go build -gcflags=m
  • //go:norace
  • when you compile in race mode, code goes through race detector book keeping.

  • go:norace makes function not to go through race detector book keeping.
  • shouldn’t be any reason to add it to your code.

  • //go:nosplit
  • skip the stack check before function

  • stack split check is a little preamble before function which checks if enough stack is available if not creates it

  • this is useful for optimization
  • //go:noinline
  • disables function inlining for particular func
  • //go:notinheap
  • set on type not on function
  • variables of this type are never heap allocated

  • if you are in gc constrained environments

  • //go:linkname time_now time.now
  • you must import _ “unsafe”
  • func time_now() will be actually be called to time.now

  • Don’t use it

  • you can get goroutineid by linking get_m to runtime.getm, and then casting m struct.

  • //import "rsc.io/pdf"
  • Not a compiler directive

  • checks that package is imported from the given addr
  • //line /foo/bar.go:123
  • renumbers line numbers in cgo

  • go build -gcflags='-l -l -l -l'
  • enables more aggresive inlining