Goroutines: from "it works" to "why is prod silently hanging"

There are two kinds of broken Go concurrency. The kind that screams at you, and the kind that doesn't.

The screaming kind is almost a kindness. It might still take down prod, but it leaves you a stack trace, a line number, a place to start. The quiet kind gives you nothing. No crash, no log, no error. Your service just stops answering some of the time, and you find out from a Slack message that starts with "hey is the API ok?"

This is the whole arc. We build up from "I want to do two things at once" to "why are 200 goroutines stuck and the runtime doesn't care". The easy parts are easy. Stick around for the last act.

Act 1: making concurrency actually run

The naive version

Everybody starts here. You have two tasks, you run them one after another, you pad it with sleeps to simulate real work.

func doSomething() {
	fmt.Println("Doing something important...")
	time.Sleep(time.Second * 2)
}
 
func doSomethingElse() {
	fmt.Println("Doing something else, equally important...")
	time.Sleep(time.Second * 3)
}
 
func main() {
	doSomething()
	doSomethingElse()
}

It's dumb and it's slow, because the second task sits around waiting for the first to finish. But hey, we all start somewhere.

"I'll just put go in front of everything"

Then you hear about goroutines. Lightweight threads. So you do the obvious thing.

func main() {
	go doSomething()
	go doSomethingElse()
	fmt.Println("I'm done")
}

$ go run main.go
I'm done

That's it. That's the whole output. The main goroutine does not give a damn about your two new goroutines. It hits the end of main, says "I'm done", and peaces out before they even start their engines. When main returns, the program dies, and everything still running dies with it.

So go is not the answer. go is the question. The answer is: how does main wait?

WaitGroup: the responsible adult

WaitGroup is a counter with three buttons.

• wg.Add(1) says "I'm starting one more task".
• wg.Done() says "this one's finished" (counter down by one).
• wg.Wait() blocks until the counter == 0.

func worker(id int, wg *sync.WaitGroup) {
	defer wg.Done()
	fmt.Printf("Worker %d starting\n", id)
	time.Sleep(time.Second)
	fmt.Printf("Worker %d done\n", id)
}
 
func main() {
	var wg sync.WaitGroup
	for i := 1; i <= 5; i++ {
		wg.Add(1)
		go worker(i, &wg)
	}
	wg.Wait()
	fmt.Println("All workers done")
}

defer wg.Done() is the move you want to internalize. It runs when the function returns, no matter how it returns. So even if the worker bails early, the counter still drops. Forget the Done and wg.Wait() blocks forever, which is a nice preview of where this article is heading.

Channels: when goroutines need to talk

WaitGroup tells you when everyone is finished. It does not carry what they produced. For that you need a channel.

func worker(id int, wg *sync.WaitGroup, result chan<- int) {
	defer wg.Done()
	time.Sleep(time.Second * 2)
	result <- id * 2
}
 
func main() {
	var wg sync.WaitGroup
	result := make(chan int, 3)
 
	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go worker(i, &wg, result)
	}
 
	go func() {
		wg.Wait()
		close(result)
	}()
 
	for r := range result {
		fmt.Printf("Received: %d\n", r)
	}
	fmt.Println("All workers done")
}

It works. Results come out, program ends. But what's that little go func() doing in the middle? Before getting clever, just read its two lines.

wg.Wait() you know: block here until the counter == 0.

close(result) is the one people get wrong. It doesn't delete the channel or empty it. It flips a "kitchen's closed" sign: nothing new comes through, but whatever's already inside stays readable. The for range loop keeps eating leftovers, and stops the instant the channel is closed and empty.

Together: wait until the workers are done, then tell the range loop the kitchen's closed. Reasonable enough. So the real question isn't what these lines do. It's why they're wrapped in a go func() instead of sitting plainly in main.

That wrapper looks optional. It isn't. Pull it out and you can wedge the whole program. Let's take it apart one change at a time, because the reason is sneakier than it looks.

The first deadlock (the loud one)

Here's what that wrapper was hiding. Two separate things were keeping this program alive, and you have to kill both to break it.

Net one: the go func(). It lets main jump straight into the for range and drain results while the workers are still sending. Send and drain happen at the same time, so the buffer never stays full for long.

Net two: a buffer big enough to hold every result (3 slots, 3 workers). Even if nobody drained until the very end, all the sends still fit, so no worker ever blocks.

Either net alone is enough. Really:

• Keep the wrapper, shrink the buffer to 2? Fine. main is draining concurrently, so a 2-slot buffer never overflows.
• Remove the wrapper, keep the buffer at 3? Fine. All 3 sends fit, nobody blocks, then main drains.

So to actually break it you have to cut both. Unwrap the lines (net one gone) and shrink the buffer (net two gone):

result := make(chan int, 2) // net two gone: too small for 3 senders
// ...
wg.Wait()                     // net one gone: main blocks here instead of draining
close(result)
for r := range result { ... }

fatal error: all goroutines are asleep - deadlock!

Now walk the circle:

1. Buffer holds 2, three workers want to send. Worker three can't fit, so it blocks on result <- ..., waiting for room.
2. A blocked worker never reaches defer wg.Done(). The counter never reaches 0.
3. So wg.Wait() waits forever, standing in main's way.
4. main is the only one who drains the channel, down in the for range. But it never gets there, stuck up top on wg.Wait().
5. Nobody drains, so no room ever frees up, so worker three stays blocked. Back to step 1.

Everyone is waiting for everyone. The runtime sees the whole program asleep, throws all goroutines are asleep - deadlock!, and pulls the plug.

The fix is to keep net one. Put the two lines back in their goroutine:

go func() {
	wg.Wait()
	close(result)
}()
for r := range result { ... }

Now main drains right away while the waiter waits in the background, and the buffer size stops mattering. That's why net one is the one to lean on: you can't always size a buffer to the workload, but you can always drain concurrently.

Now here is the part I actually want you to remember. That crash was a gift. The runtime caught it, named it, and stopped everything before it could do damage. You hear about it the instant you hit run.

Hold that thought. Because its evil twin makes no sound at all.

Act 2: when concurrency breaks

Goroutine code dies two ways.

Panic is loud and immediate. One goroutine blows up, prints a stack, and unless it recovers itself, takes the whole process down with it.

Deadlock is the opposite. It can be quiet, it can wait, and it can take down the whole system without a single line of output.

Panic, the loud one

Panic tells you everything: the error, the type, the exact line. That doesn't make it harmless. It makes it findable.

ch := make(chan int)
close(ch)
ch <- 1 // panic: send on closed channel
 
var p *Person
fmt.Println(p.Name) // panic: nil pointer dereference
 
a := []int{1, 2, 3}
_ = a[5] // panic: index out of range [5] with length 3
 
var i interface{} = "hello"
_ = i.(int) // panic: interface conversion: string, not int

It stops the goroutine, runs that goroutine's defers on the way up, and bubbles. A defer with recover() catches it; otherwise you get a stack trace pointing at the crime scene.

The part that bites in production: an unrecovered panic takes down the whole process, not just the one goroutine. And recover() only works inside the same goroutine that panicked, so main can't catch a worker's panic. net/http already wraps each request handler, but a goroutine you spawn is bare, and one stray panic in it sinks the whole service. So wrap the ones you start:

func goSafe(fn func()) {
	go func() {
		defer func() {
			if r := recover(); r != nil {
				log.Printf("goroutine panic: %v\n%s", r, debug.Stack())
			}
		}()
		fn()
	}()
}

Still, panic is the gentler problem: when it blows it leaves a body and a note, a stack trace that says what and where. The deadlock we're about to meet keeps the lights on and says nothing.

Deadlock, the boss fight

Deadlock comes in two flavors, and the difference between them is the whole point of this article.

Channel deadlock. You already met it in Act 1. The minimal version:

ch := make(chan int) // open, unbuffered
ch <- 1              // nobody is receiving, so this blocks forever
// fatal error: all goroutines are asleep - deadlock!

The runtime catches this. Loud, instant, fatal.

Worth pausing on, because the channel's state decides your fate here. Send to a closed channel and you get a panic (send on closed channel). Send to an open one with no receiver, like above, and you deadlock. Same ch <- 1, two completely different deaths.

Mutex deadlock. Two goroutines, two locks, grabbed in opposite order:

func mutexDeadlock() {
	var m1, m2 sync.Mutex
 
	go func() {
		m1.Lock()
		time.Sleep(100 * time.Millisecond)
		m2.Lock() // waiting for the other one to let go of m2
		m2.Unlock()
		m1.Unlock()
	}()
 
	go func() {
		m2.Lock()
		time.Sleep(100 * time.Millisecond)
		m1.Lock() // waiting for the other one to let go of m1
		m1.Unlock()
		m2.Unlock()
	}()
 
	time.Sleep(time.Second)
}

Goroutine 1 holds m1, wants m2. Goroutine 2 holds m2, wants m1. Neither will ever let go. They are stuck until the heat death of the universe.

And the runtime says... nothing.

Why one screams and the other stays quiet

Go does have a built-in deadlock alarm. But it fires under exactly one condition: every goroutine is blocked at the same time. No goroutine left that could possibly run, so the runtime gives up and shouts all goroutines are asleep.

A toy channel deadlock trips it instantly. There's only main, main is stuck, so everyone is stuck. Alarm.

A mutex deadlock in a real server never trips it. The server still has goroutines accepting requests, timers firing, the GC running. So even with two goroutines locked against each other, the runtime looks around, sees plenty still awake, and stays quiet. The alarm only knows how to spot total gridlock, not two stuck goroutines in an otherwise busy program.

That's the whole asymmetry: the loud failure trips the alarm, the quiet one walks right under it. So for the quiet one you're on your own. You need habits to keep it from forming, and tools to find it when it does. Both are coming up.

The fix ladder

There are four ways out, and they climb from "simple but fragile" to "Go idiomatic". Pick the lowest rung that holds your weight.

Rung 1: lock in the same order, always. The deadlock only happens because one goroutine grabs m1 then m2 and the other grabs m2 then m1. Force everyone to grab them in the same order and the cycle can't form.

// the rule is the LOCK order: everyone takes m1, then m2
m1.Lock()
m2.Lock()
// work
m2.Unlock() // unlock order is free, reverse is just habit
m1.Unlock()

Only the lock order matters. Unlocking never blocks, so you can release in any order you like. The reverse-order unlock above is just the tidy convention, and it's exactly what you get for free if you use defer m1.Unlock() right after each Lock(), since defers run last-in-first-out. Don't mistake that habit for a rule.

Simple, correct. Falls apart the moment you have a dozen locks and can't keep the global order straight in your head.

Rung 2: one lock to rule them all. Wrap the whole set behind a single lock-everything / unlock-everything pair.

type OrderedLocks struct{ m1, m2 sync.Mutex }
 
func (l *OrderedLocks) LockBoth()   { l.m1.Lock(); l.m2.Lock() }
func (l *OrderedLocks) UnlockBoth() { l.m2.Unlock(); l.m1.Unlock() }

Barbaric, but it works. You also just threw away most of your concurrency, because nobody can hold one lock while another holds the other. Sometimes that's the right call. Usually it's a sign you need rung 3.

Rung 3: TryLock with a timeout. Instead of blocking forever waiting for a lock, try, and give up if it doesn't come.

func tryLock(m *sync.Mutex, timeout time.Duration) bool {
	ctx, cancel := context.WithTimeout(context.Background(), timeout)
	defer cancel()
	for {
		select {
		case <-ctx.Done():
			return false // gave up, no deadlock
		default:
			if m.TryLock() {
				return true
			}
			time.Sleep(time.Millisecond)
		}
	}
}

TryLock landed in Go 1.18. The context here is just a stopwatch, don't overthink it. The point is a goroutine that can't get its second lock backs off instead of hanging. No permanent wait means no deadlock. This is the rung most real services live on.

Rung 4: don't share memory, pass messages. This one is a Go proverb, almost word for word from Effective Go: "Do not communicate by sharing memory; instead, share memory by communicating." Model each resource as a buffered channel holding one token. Holding the resource means holding the token. select with a default means "grab it if you can, otherwise move on".

resource1 := make(chan struct{}, 1)
resource1 <- struct{}{} // the token
 
select {
case <-resource1:
	// got it, do work, then put it back
	resource1 <- struct{}{}
default:
	// someone else has it, don't block
}

No mutexes, no lock ordering to babysit. You've turned "wait on a lock" into "ask a channel, take no for an answer". Whether this is cleaner than rung 3 depends on the problem, but it's the one that feels most like Go.

Debugging cheat sheet

The sticky little facts, in one place.

Read the symptom first:

Then the facts that bite:

• There is no wg.Count(). To watch outstanding work, keep your own atomic.Int64 next to the WaitGroup, or watch runtime.NumGoroutine() for the trend (it counts every goroutine, so read the direction, not the absolute number).

• recover() only saves its own goroutine, and only inside a defer. Every goroutine you spawn needs its own. net/http already wraps each request handler for you; your bare go statements get nothing.

• pprof for stuck locks: import _ "net/http/pprof", run a server (go http.ListenAndServe("localhost:6060", nil)), then open /debug/pprof/goroutine?debug=1. A pile of goroutines parked like this is your deadlock:

  sync.runtime_SemacquireMutex+0x27  runtime/sema.go:77
  sync.(*Mutex).Lock+0x16f           sync/mutex.go:90
  main.mutexDeadlock.func1+0x11c     main.go:35
  

  sync.runtime_SemacquireMutex is the internal name for "stuck in Mutex.Lock"; the line numbers tell you which locks.

• Static tools are thin. go vet catches a copied Mutex / WaitGroup. staticcheck catches some provable nil derefs and impossible type assertions. Neither catches a deadlock. The compiler only flags a constant index out of range on an array, never a slice.

• Lock-order deadlock in dev: swap sync.Mutex for sasha-s/go-deadlock, a drop-in that prints a report at runtime when the lock order inverts. Take it back out for prod.

The one table to remember

Panic is the coworker who tells you to your face. Deadlock is the one who quietly stops doing their job and lets you find out in the standup three days later.

The lesson from the whole arc: the failures that crash are doing you a favor. The one that worries me, the one worth building habits around, is the mutex deadlock that keeps the lights on while half your requests quietly die in the dark. Lock in one order, or stop sharing memory and pass a message instead.

References

• Effective Go, "Share by communicating" — go.dev/doc/effective_go#Share_by_communicating
• Rob Pike, "Go Proverbs" — go-proverbs.github.io