Imagine you're running a busy coffee shop. If every single customer who walked through the door demanded their own personal barista, you'd run out of floor space and go bankrupt in ten minutes. Instead, you hire three experienced baristas. They stand behind the counter, and there's a single line of orders (the queue). As soon as a barista finishes one latte, they look at the line and grab the next order. This keeps your resource usage predictable and prevents the shop from collapsing under its own weight.

In Go, a worker pool is exactly that. The "orders" are data sent over a channel, the "baristas" are a fixed number of goroutines, and the "counter" is the channel itself. We do this because while goroutines are cheap, they aren't free. If you spawn 100,000 goroutines to process 100,000 API calls simultaneously, you'll likely hit rate limits, exhaust file descriptors, or blow through your memory. Throttling that concurrency is a mark of production-ready code.

Setting up the Conveyor Belt

First, we need a way to define what "work" is and a way to transport it. I usually start by defining a simple Job struct. We also need a channel to send these jobs to the workers. The key here is that the workers will all be listening to the same channel. Go's channel implementation ensures that each job is received by exactly one worker—no two baristas will try to make the same latte.

type Job struct {
	ID    int
	Value int
}

func worker(id int, jobs <-chan Job, results chan<- int) {
	for j := range jobs {
		// Simulate some heavy lifting
		fmt.Printf("Worker %d started job %d\n", id, j.ID)
		time.Sleep(time.Millisecond * 100) 
		results <- j.Value * 2
	}
}

Notice that I used range jobs. This is a clean way to tell the worker: "Keep taking jobs until the channel is closed." Once the channel closes, the loop terminates, and the goroutine exits gracefully.

Hiring the Baristas

Now we need to actually spawn the workers. I like to do this in a loop before I start sending any data. If we spawn them after sending data, we might fill the channel buffer and block our main thread before the workers even start. I've seen people make this mistake in interviews all the time—they forget that the "pool" needs to be ready and waiting for the work.

func main() {
	const numJobs = 10
	const numWorkers = 3

	jobs := make(chan Job, numJobs)
	results := make(chan int, numJobs)

	// This is our "hiring" phase
	for w := 1; w <= numWorkers; w++ {
		go worker(w, jobs, results)
	}

	// Send the work
	for j := 1; j <= numJobs; j++ {
		jobs <- Job{ID: j, Value: j}
	}
	close(jobs) // Tell workers no more jobs are coming
}

Knowing when the shift is over

Here is where most developers trip up: how do you know when all the workers are actually done? Just closing the jobs channel tells the workers to stop looking for new work, but they might still be processing their final task. If your main function exits too early, you'll lose that last bit of data.

I always reach for a sync.WaitGroup here. It's the most explicit way to say, "Wait for these specific N goroutines to signal they are finished." I wrap the worker logic so the WaitGroup can be decremented right before the goroutine dies.

func worker(id int, jobs <-chan Job, results chan<- int, wg *sync.WaitGroup) {
	defer wg.Done() // Mark this barista as 'off the clock'
	for j := range jobs {
		results <- j.Value * 2
	}
}

// In main...
var wg sync.WaitGroup
for w := 1; w <= numWorkers; w++ {
	wg.Add(1)
	go worker(w, jobs, results, &wg)
}

// ... after sending jobs and closing the channel:
go func() {
	wg.Wait()
	close(results) // Now it's safe to close results
}()

By closing the results channel in a separate goroutine, we can range over the results in our main thread without deadlocking. It's a pattern you'll see in almost every high-performance Go project.

📋 Practical Task

Implement a Concurrent Image Metadata Processor

You are building a tool that processes a list of "image files" to extract metadata. Since reading files is I/O intensive, you need to implement a worker pool to avoid overwhelming the system.

Requirements:

• Create a FileJob struct that contains a Filename (string) and a FileSize (int).
• Create a Metadata struct that contains the Filename and a ProcessedHash (string).
• Implement a processor worker function that:
  • Receives FileJob from a jobs channel.
  • Simulates a delay using time.Sleep (e.g., 50ms).
  • Generates a dummy hash (you can just return the filename + "_processed").
  • Sends the Metadata result to a results channel.
• In your main function:
  • Initialize a pool of 5 workers.
  • Feed 20 dummy files into the jobs channel.
  • Use a sync.WaitGroup to ensure all workers finish before the program exits.
  • Print the final metadata results as they come in.