Doing many things at once in Go: from one go keyword to a worker pool

2024/10/06
golangconcurrencygoroutinescontexterrgroupdeadlockmutex

The simplest way to make Go do many things at once is to put go in front of a function call. That is where the problems start, though: main doesn't wait for it, the result can't come back, and when several goroutines change the same variable, the count comes out wrong.

One scene runs through the whole article: a restaurant kitchen.

KitchenGo
a cooka goroutine
the head chef, who locks up and turns off the lights when leavingmain (when main returns, the program ends)
the counter on the whiteboard: a tick for every dish handed out, crossed off when it is donesync.WaitGroup
the pass, the shelf where cooks put finished platesa channel
the "kitchen closed" sign on the passclose(ch)
one pen, one wok: while someone uses it, everyone else waitssync.Mutex
watching the pass and the kitchen timer at once, and dealing with whichever moves firstselect
the front of house calls in "the customer cancelled", and every cook stopscontext
only three burners, so no matter how many dishes, only three cook at oncea worker pool
one dish burns, the whole table is redone, and the other cooks can stoperrgroup
the kitchen manager: when every cook is waiting on someone else and nobody can move, he closes the kitchen and writes down what each cook was waiting forthe runtime's deadlock detector (prints fatal error and where each goroutine is stuck, then ends the program)

All programs below were run on Go 1.26.1 (darwin/arm64); errgroup is golang.org/x/sync v0.20.0.

Part 1: running things at the same time

The naive version

Start with the simplest case. Two tasks, one after the other, with sleep standing in for real work.

func doSomething() {
	fmt.Println("Doing something important...")
	time.Sleep(time.Second * 2)
}
 
func doSomethingElse() {
	fmt.Println("Doing something else, equally important...")
	time.Sleep(time.Second * 3)
}
 
func main() {
	doSomething()
	doSomethingElse()
}

It takes 5 seconds, because the second task waits for the first to finish. One cook, two dishes: finish the first, then start the second.

"I'll just put go in front of everything"

A goroutine is a function the Go runtime runs concurrently, and starting one is much cheaper than starting an OS thread. So the natural next step is:

func main() {
	go doSomething()
	go doSomethingElse()
	fmt.Println("I'm done")
}
$ go run main.go
I'm done

Out of 1000 runs, 911 printed only I'm done. The other 89 printed a line or two more, but neither task finished its sleep.

In the kitchen, the head chef hands out two dishes, says "I'm done", turns off the lights and leaves, and the cooks go home with him wherever they were. When main returns, the program ends, and every goroutine still running ends with it.

So how does main wait for the goroutines to finish?

WaitGroup: the counter on the whiteboard

sync.WaitGroup is a counter. The head chef puts a tick on the whiteboard for every dish handed out, and each cook crosses one off when done. Until every tick is crossed off, the head chef cannot leave.

  • wg.Add(1): add one, one more task.
  • wg.Done(): subtract one, this task is finished.
  • wg.Wait(): wait until the counter reaches 0.
func worker(id int, wg *sync.WaitGroup) {
	defer wg.Done()
	fmt.Printf("Worker %d starting\n", id)
	time.Sleep(time.Second)
	fmt.Printf("Worker %d done\n", id)
}
 
func main() {
	var wg sync.WaitGroup
	for i := 1; i <= 5; i++ {
		wg.Add(1)
		go worker(i, &wg)
	}
	wg.Wait()
	fmt.Println("All workers done")
}

Five workers, each sleeping 1 second, finish in 1.0 second total.

Write Done with defer. Then the counter goes down however the worker returns. Forget it, and one tick stays on the board forever, so wg.Wait() waits forever.

Since Go 1.25 you can also use wg.Go(f), which does the add, the go, and the done in one call. Most existing code still uses the form above.

Channels: when goroutines need to send back results

A WaitGroup tells you when everyone is finished, not what they produced. For results you need a channel, which is the pass: cooks put plates on it, someone takes them off the other side. make(chan int, 3) is a pass with 3 slots. When all slots are full, a cook stands there holding the plate.

func worker(id int, wg *sync.WaitGroup, result chan<- int) {
	defer wg.Done()
	time.Sleep(time.Second * 2)
	result <- id * 2
}
 
func main() {
	var wg sync.WaitGroup
	result := make(chan int, 3)
 
	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go worker(i, &wg, result)
	}
 
	go func() {
		wg.Wait()
		close(result)
	}()
 
	for r := range result {
		fmt.Printf("Received: %d\n", r)
	}
	fmt.Println("All workers done")
}

This runs fine. The small go func() in the middle has two lines: wg.Wait() waits for every cook, and close(result) hangs the "kitchen closed" sign.

close does not delete or empty the channel. After the sign goes up, nobody can put a new plate on the pass, but plates already there can still be taken. So the for range takes the remaining plates and stops once the channel is empty.

Why are those two lines wrapped in go func() instead of sitting directly in main?

The first deadlock

Two things keep this program from getting stuck:

  • The go func(): it lets main go straight to the for range, taking plates off while cooks put them on, so the pass is rarely full.
  • A big enough pass: 3 slots for 3 workers, so every plate fits even if nobody takes one.

Keep either one and the program finishes:

go func()?buffer sizeresult
yes3finishes
yes2finishes
yes0 (unbuffered)finishes
no3finishes
no2fatal error: all goroutines are asleep - deadlock!
no0 (unbuffered)fatal error: all goroutines are asleep - deadlock!

Remove both:

result := make(chan int, 2) // 3 senders, only 2 slots
// ...
wg.Wait()                     // main blocks here instead of draining
close(result)
for r := range result { ... }

This is how it gets stuck:

  1. The pass holds 2 plates. The third cook finds it full and stands at result <- ... holding the plate.
  2. He never reaches defer wg.Done(), so his tick is never crossed off.
  3. The head chef keeps standing at wg.Wait().
  4. The only one who takes plates off the pass is the head chef, and he never reaches the for range.
  5. Nobody takes a plate, no slot opens, and the third cook keeps standing. Back to step 1.

Everyone is waiting on someone else. The runtime sees that every goroutine is stuck, prints all goroutines are asleep - deadlock!, and ends the program.

The fix is to put the go func() back. Then main starts taking plates right away, and the buffer size stops mattering. A buffer is hard to size to the workload, while taking plates as they arrive always works, so the go func() is the safer of the two.

Mutex: several cooks updating one number

On the kitchen wall is a count of dishes served today. Each cook, after finishing a dish, walks over and adds one. In Go, that is 1000 goroutines each doing orders++:

orders := 0
var wg sync.WaitGroup
for i := 0; i < 1000; i++ {
	wg.Add(1)
	go func() {
		defer wg.Done()
		orders++
	}()
}
wg.Wait()
fmt.Println(orders)

It should end at 1000. Over 30 runs, the result was between 935 and 983, and never 1000.

orders++ looks like one step but is three: read the current value, add one, write it back. If two cooks both read 41, each adds one and both write 42, and one dish goes uncounted. This is a data race. Run it with go run -race and it prints WARNING: DATA RACE, plus the two lines that touched the same variable.

The fix is a sync.Mutex. After one goroutine calls Lock(), anyone else calling Lock() waits until the holder calls Unlock(). It is like having one pen by the wall: only the cook holding the pen can change the number.

var mu sync.Mutex
// ...
go func() {
	defer wg.Done()
	mu.Lock()
	orders++
	mu.Unlock()
}()

With the lock, 10 runs all printed 1000. If all you need is to add to a number, atomic.Int64 from sync/atomic also works: call Add(1), no lock needed.

Part 2: waiting, cancelling, limiting

select: waiting on several things at once

So far main waits on one thing at a time: wg.Wait() for the counter to reach zero, <-result for a plate on the pass. Sometimes you need to wait on two things and handle whichever comes first. The most common case is a timeout: the customer waits at most 1 second, and if the dish isn't ready, someone tells them.

func cook(d time.Duration) <-chan string {
	ch := make(chan string, 1)
	go func() {
		time.Sleep(d)
		ch <- "steak"
	}()
	return ch
}
 
func main() {
	select {
	case dish := <-cook(2 * time.Second):
		fmt.Println("got", dish)
	case <-time.After(1 * time.Second):
		fmt.Println("timeout")
	}
}

select blocks until one of its cases has a value. The steak takes 2 seconds and time.After(1 * time.Second) sends a value after 1 second, so this prints timeout after 1.0 seconds.

The channel in cook has 1 slot because nobody will take the steak after the timeout. With no slot, that cook would stand holding the plate forever, and the goroutine would never end.

select can also have a default. If no channel is ready, it takes default and doesn't wait.

context: the customer cancelled

A timeout stops waiting when time runs out. Sometimes the signal comes from outside instead, "no need to finish": the user closes the page and the HTTP request is cancelled. Cooks still working on it should stop too, or the work is wasted.

context carries that message. context.WithCancel gives you a ctx and a cancel function. After cancel(), the ctx.Done() channel is closed, and every goroutine waiting on it in a select sees it:

func cook(ctx context.Context, id int) error {
	select {
	case <-time.After(2 * time.Second):
		fmt.Printf("dish %d done\n", id)
		return nil
	case <-ctx.Done():
		fmt.Printf("dish %d stopped: %v\n", id, ctx.Err())
		return ctx.Err()
	}
}
 
func main() {
	ctx, cancel := context.WithCancel(context.Background())
	var wg sync.WaitGroup
	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			cook(ctx, i)
		}()
	}
	time.Sleep(500 * time.Millisecond)
	cancel() // the customer cancelled
	wg.Wait()
}

Three dishes take 2 seconds each, and the customer cancels at 0.5 seconds. All three cooks print context canceled and stop, and the whole run takes 0.5 seconds.

context.WithTimeout is the version that cancels itself when time is up. In net/http, every request carries a ctx, r.Context(), which is cancelled when the client disconnects. Pass that ctx down to the work a handler starts, and the work stops along with the request.

Only three burners: limiting how many run at once

So far every dish gets its own goroutine. With 5 dishes that's fine, but with 100,000 requests to an external API all sent at once, the other side's rate limit will probably start rejecting them. The kitchen has the same limit: three burners, so only three dishes cook at a time.

The first way is a worker pool: start only three cooks, and have them take orders from one channel.

dishes := make(chan int) // no size: it holds no orders, each one goes straight into a cook's hands
var wg sync.WaitGroup
for stove := 1; stove <= 3; stove++ {
	wg.Add(1)
	go func() {
		defer wg.Done()
		for d := range dishes {
			cookDish(d) // 1 second per dish
		}
	}()
}
for n := 1; n <= 10; n++ {
	dishes <- n // if no cook is waiting at range, main stops on this line
}
close(dishes) // no more orders; cooks finish what is left and go home
wg.Wait()

The loop at the bottom is the head chef handing out orders 1 to 10, one at a time, into dishes. dishes has no size, so it can't hold a single order. dishes <- n only completes when some cook is sitting at range ready to receive. With make(chan int, 10) instead, all 10 orders would go in at once, and the head chef would go straight on to close(dishes) and wg.Wait() without waiting. The cooks still cook at most three dishes at once, though, so it finishes in 4 seconds, the same as the unsized version below. Printing a timestamp where the head chef hands out an order, where a cook takes one, and where a cook goes home gives this order of events:

0 s  head chef hands out 1, 2, 3; each cook takes one and starts
     head chef waits holding 4, since all three cooks are busy
1 s  the three cooks finish and take 4, 5, 6
2 s  they take 7, 8, 9
3 s  one cook takes 10; head chef calls close(dishes)
     the other two cooks' for range gets no more orders, ends, they go home
4 s  the last cook finishes 10 and goes home; wg.Wait() returns

Which cook gets which order changes from run to run. For the first three seconds three dishes cook at once; in the last second only dish 10 does.

The second way still starts one goroutine per dish, but each has to get a burner before cooking. The burners are a channel with 3 slots, a pattern called a semaphore:

stoves := make(chan struct{}, 3)
for d := 1; d <= 10; d++ {
	wg.Add(1)
	go func() {
		defer wg.Done()
		stoves <- struct{}{}        // take a burner; wait if all three are taken
		defer func() { <-stoves }() // give it back when done
		cookDish(d)
	}()
}
wg.Wait()

With 10 dishes at 1 second each, both versions take 4.0 seconds, with at most 3 dishes cooking at once. 10 dishes split into rounds of 3, 3, 3 and 1, so 4 seconds.

The number of goroutines is different, though. The worker pool has 3 the whole time; the semaphore version starts 10 right away, and 7 of them wait for a burner.

errgroup: one dish burns, the whole table stops

A WaitGroup only waits; it doesn't care who failed. But if one dish on a table burns, the whole table is redone, and the other cooks can stop. So as soon as one goroutine returns an error, the others should stop their work, and that error should go back to main.

golang.org/x/sync/errgroup bundles a WaitGroup, a context and error collection. It is not in the standard library; run go get golang.org/x/sync first.

g, ctx := errgroup.WithContext(context.Background())
g.SetLimit(3) // three burners
for i := 1; i <= 5; i++ {
	g.Go(func() error {
		if i == 2 {
			time.Sleep(300 * time.Millisecond)
			return errors.New("dish 2 burned")
		}
		select {
		case <-time.After(2 * time.Second):
			return nil
		case <-ctx.Done():
			return ctx.Err()
		}
	})
}
err := g.Wait()

g.Go is like wg.Go, except the function returns an error. The ctx from errgroup.WithContext is cancelled as soon as the first error comes back; g.SetLimit(3) is the three burners from the previous section; g.Wait() waits for everyone, then returns the first error.

Of 5 dishes, dish 2 burns at 0.3 seconds, and the other cooks see the cancellation on ctx.Done() and stop. g.Wait() returns dish 2 burned after 0.3 seconds.

Part 3: when it goes wrong

When these tools are misused, the program either crashes and prints why, or keeps running while some requests never get an answer.

The crash is actually the easier one. It can still take down prod, but at least it leaves a stack trace and a line number. The hang is harder: no crash, no log, no error. Often nobody notices until someone asks in Slack, "hey, is the API ok?"

Panic

A panic prints a stack trace the moment it happens, with the line number and the error, so it is easy to find.

ch := make(chan int)
close(ch)
ch <- 1 // panic: send on closed channel
 
var p *Person
fmt.Println(p.Name) // panic: runtime error: invalid memory address or nil pointer dereference
 
a := []int{1, 2, 3}
_ = a[5] // panic: runtime error: index out of range [5] with length 3
 
var i interface{} = "hello"
_ = i.(int) // panic: interface conversion: interface {} is string, not int

In the kitchen, a panic is a fire at one cook's station. The panic stops that goroutine, runs its defers, and moves up the call stack. If a deferred function calls recover(), the panic stops there; otherwise the program prints the stack trace and exits.

The trouble is that an unrecovered panic ends the whole process, not just that goroutine. A fire at one station sets off the sprinklers in the whole kitchen. And recover() only works in the goroutine that panicked, so main cannot catch a worker's panic. net/http recovers panics in each request handler for you, but goroutines you start with go get nothing, so wrap them yourself:

func goSafe(fn func()) {
	go func() {
		defer func() {
			if r := recover(); r != nil {
				log.Printf("goroutine panic: %v\n%s", r, debug.Stack())
			}
		}()
		fn()
	}()
}

Deadlock

A deadlock is goroutines waiting on each other, forever.

Channel deadlock. You already saw one in Part 1. The smallest version is two lines:

ch := make(chan int) // open, unbuffered
ch <- 1              // nobody is receiving, so this blocks forever
// fatal error: all goroutines are asleep - deadlock!

The same ch <- 1 panics if the channel is closed. If it is open and nobody receives, it waits forever, as above.

Mutex deadlock. The most common case is two locks taken in opposite orders. In the kitchen, two cooks grab the wok and the spatula in opposite order:

func main() {
	var wok, spatula sync.Mutex
	var wg sync.WaitGroup
	wg.Add(2)
 
	go func() { // cook 1
		defer wg.Done()
		wok.Lock()
		time.Sleep(100 * time.Millisecond)
		spatula.Lock() // waits for cook 2 to put the spatula down
		spatula.Unlock()
		wok.Unlock()
	}()
 
	go func() { // cook 2
		defer wg.Done()
		spatula.Lock()
		time.Sleep(100 * time.Millisecond)
		wok.Lock() // waits for cook 1 to put the wok down
		wok.Unlock()
		spatula.Unlock()
	}()
 
	wg.Wait()
}

Cook 1 holds the wok and waits for the spatula; cook 2 holds the spatula and waits for the wok. Neither lets go. Run on its own, this program is caught too:

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [sync.WaitGroup.Wait]:
...
goroutine 18 [sync.Mutex.Lock]:
...
goroutine 19 [sync.Mutex.Lock]:

When the runtime catches a deadlock, and when it does not

There are two kinds of stopped goroutine. One is blocked on a channel or a mutex and needs another goroutine to wake it, by receiving a value or calling Unlock(). The other is waiting on the network or a timer, and wakes up by itself when a packet arrives or the time is up.

Go's deadlock detector fires only when every goroutine is the first kind, because then nobody can wake anybody. In the kitchen, the manager sees everyone waiting on someone else and closes up. The runtime prints fatal error and where each goroutine is stuck, then ends the program.

A real server never gets there. Some goroutine is always waiting for the next request, which is the second kind, so the runtime never decides the program is stuck. Two cooks are frozen at the stove, but the front of the house is still taking orders, so the manager sees movement and stays out of it.

Each program in the table below has two goroutines blocked forever plus one goroutine that stays alive, and is killed after 6 seconds:

the extra goroutineblocked on a channelblocked on two mutexes
none (main ends in select {})fatal error: all goroutines are asleep - deadlock!fatal error: all goroutines are asleep - deadlock!
a time.Tick loopno output until killedno output until killed
http.ListenAndServeno output until killedno output until killed

Only the first row triggers the detector. Rows two and three each have a goroutine waiting on a timer or the network, and the runtime reports nothing. A production server is row three.

Four ways out

1. Lock in the same order, always. The deadlock happens because cook 1 takes the wok first and cook 2 takes the spatula first. If the kitchen rule is that everyone takes the wok first, two cooks can never each hold one tool and wait for the other.

// the rule is the LOCK order: everyone takes wok, then spatula
wok.Lock()
spatula.Lock()
// work
spatula.Unlock() // unlock order is free, reverse is just habit
wok.Unlock()

Only the order of taking matters; Unlock never blocks, so release in any order. This is simple and correct, but with a dozen locks it gets hard to remember the order. sasha-s/go-deadlock catches wrong orders during development, see the cheat sheet below.

2. Take everything at once. Wrap both locks as one set and always take them together, like a tool cabinet you open as a whole.

type Tools struct{ wok, spatula sync.Mutex }
 
func (t *Tools) LockBoth()   { t.wok.Lock(); t.spatula.Lock() }
func (t *Tools) UnlockBoth() { t.spatula.Unlock(); t.wok.Unlock() }

Crude, but it works. A cook who needs only the wok now blocks a cook who needs only the spatula, though. If that is not acceptable, go back to option 1.

3. TryLock with a timeout. Instead of waiting forever, try for a while and give up.

func tryLock(m *sync.Mutex, timeout time.Duration) bool {
	ctx, cancel := context.WithTimeout(context.Background(), timeout)
	defer cancel()
	for {
		select {
		case <-ctx.Done():
			return false // gave up, no deadlock
		default:
			if m.TryLock() {
				return true
			}
			time.Sleep(time.Millisecond)
		}
	}
}

TryLock was added in Go 1.18. It returns true if it gets the lock and false right away if not, without waiting. The context.WithTimeout around it is the timer that decides when to give up. Giving up is not enough: the cook who cannot get the spatula also has to put the wok back so the other cook can finish.

for {
	wok.Lock()
	if tryLock(&spatula, 50*time.Millisecond) {
		// cook
		spatula.Unlock()
		wok.Unlock()
		break
	}
	wok.Unlock() // put the wok back
	time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
}

That last wait has to be random. If both cooks always wait the same time, they come back together and collide again. Both goroutines keep running but make no progress; this is called livelock. Over 10 runs each: with both cooks following the same formula, the cook who finished second needed 4 to 45 attempts, median 14; with a random 0 to 100 ms wait, it took 3 or 4 attempts every time.

The Go documentation is direct about TryLock: "while correct uses of TryLock do exist, they are rare, and use of TryLock is often a sign of a deeper problem in a particular use of mutexes." Use it only when you cannot fix a lock order.

4. Pass messages instead of sharing memory. From Effective Go: "Do not communicate by sharing memory; instead, share memory by communicating." Model a tool as a channel with one slot holding one token, like a wok on its hook. select with default means "take it if it is on the hook, otherwise do something else":

wokHook := make(chan struct{}, 1)
wokHook <- struct{}{} // the wok is on its hook
 
select {
case <-wokHook:
	// got the wok, cook, then hang it back
	wokHook <- struct{}{}
default:
	// someone else has it, don't wait
}

With one tool there is no order to get wrong. With more, hooks deadlock just like mutexes: one cook takes the wok, the other takes the spatula, each waits for the second with a plain <-hook, and the result is the same fatal error. So option 1's ordering rule still applies.

Panic and deadlock side by side

PanicDeadlock
Whenimmediatelywhen goroutines start waiting on each other, which may be rare
Outputstack trace, error messagefatal error only when nothing at all can run; in a server, nothing
Recoverableyes, recover(), same goroutine onlyno, restart the process
Damagewhole process, unless that goroutine recoversrequests that need those locks, until restart
Finding itread the errorNumGoroutine, pprof

Debugging cheat sheet

Start from the symptom:

What you seeWhat it usually isFirst move
stack trace, process exitsunrecovered panicread the trace, it names the line
process dies, trace points into a goroutine you startedpanic with no recover() in that goroutinewrap the goroutine (goSafe above)
fatal error: all goroutines are asleepevery goroutine blockedthe trace shows where each one is waiting
service hangs, no error, goroutine count climbssome goroutines blocked forever (mutex or channel)runtime.NumGoroutine() trend, then pprof
flaky, occasional garbage datadata racego test -race

Facts that are easy to get wrong:

  • There is no wg.Count(). To see how much work is left, keep your own atomic.Int64 next to the WaitGroup, or watch whether runtime.NumGoroutine() goes up or down.
  • pprof for stuck locks: import _ "net/http/pprof", start a server (go http.ListenAndServe("localhost:6060", nil)), then open /debug/pprof/goroutine?debug=1. A goroutine waiting for a lock looks like this (file paths shortened):
    #	internal/sync.runtime_SemacquireMutex+0x27	runtime/sema.go:95
    #	internal/sync.(*Mutex).lockSlow+0x16f		internal/sync/mutex.go:149
    #	internal/sync.(*Mutex).Lock+0xd3		internal/sync/mutex.go:70
    #	sync.(*Mutex).Lock+0x80				sync/mutex.go:46
    #	main.startCooks.func1+0x88			main.go:18
    
    The main. line at the bottom is the Lock() call in your code that is waiting.

    Waiting for a lock does not always mean stuck; many requests may just be queueing. debug=1 merges goroutines with the same stack into one entry, so you cannot tell them apart. ?debug=2 lists each goroutine separately with its ID. Open it twice, 10 seconds apart; the same ID still waiting is probably stuck.
  • Static checks help little. go vet catches a copied Mutex or WaitGroup, but it does not report deadlocks.
  • Catching lock order in development: replace sync.Mutex with sasha-s/go-deadlock. When it sees two locks taken in opposite orders, it prints a report. Switch back before production.

What to use for what

What you wantWhat to use
run at the same timego
wait for everything to finishsync.WaitGroup, wg.Go
bring results backa channel, close when done sending
several goroutines change one variablesync.Mutex; for adding to a number, sync/atomic
wait on several things, set a timeoutselect, time.After
stop when told from outsidecontext
limit how many run at oncea worker pool, or a buffered channel as a semaphore
stop everything when one failserrgroup

References