You've probably already felt the temptation. You have a list of a thousand tasks—maybe you're pinging a list of servers to check their health or processing a batch of image uploads—and your first instinct in Go is to just wrap the function call in a go keyword inside a loop. It feels like the "Go way." Why not just let the runtime handle it? After all, goroutines are cheap.

The danger of the "Unlimited" approach

Let's look at how most people first try to solve a batch processing problem. Imagine we're writing a tool to check the status of 10,000 different URLs. You might write something like this:

for _, url := range urls {
    go func(u string) {
        status := checkURL(u)
        fmt.Println(u, status)
    }(url)
}

On a small list of 10 or 20 URLs, this is lightning fast. But when you scale that to 10,000, you're going to hit a wall. I've seen this crash production environments more times than I care to admit. You aren't just limited by your own CPU or RAM; you're limited by the operating system's open file descriptors and the network stack. You'll likely start seeing "too many open files" errors or, worse, the remote servers will identify your sudden burst of 10,000 simultaneous requests as a DDoS attack and block your IP.

The problem here is that you have no backpressure. You're dumping every single job into the scheduler as fast as the loop can run, regardless of whether your system (or the network) can actually handle the throughput.

Taming the load with a fixed pool

The better way is to decouple the submission of work from the execution of work. Instead of creating a new goroutine for every task, we create a fixed number of "workers" that live for the duration of the program and pull tasks from a shared queue. In Go, that queue is just a channel.

Here is how I usually structure this. We create a jobs channel to send the URLs and a results channel to collect the outcomes. We then spawn a specific number of workers—say, 5—who all listen to that same jobs channel.

func worker(id int, jobs <-chan string, results chan<- string) {
    for url := range jobs {
        // Each worker pulls a job, processes it, and sends the result
        results <- fmt.Sprintf("worker %d finished %s", id, url)
    }
}

func main() {
    urls := []string{"https://google.com", "https://golang.org", "https://github.com"} // imagine 10,000 of these
    
    jobs := make(chan string, len(urls))
    results := make(chan string, len(urls))

    // We start exactly 3 workers. No more, no less.
    for w := 1; w <= 3; w++ {
        go worker(w, jobs, results)
    }

    // Send the jobs into the channel
    for _, url := range urls {
        jobs <- url
    }
    close(jobs) // Important: tells workers there is no more work coming

    // Collect the results
    for i := 0; i < len(urls); i++ {
        fmt.Println(<-results)
    }
}

Why this wins (and what it costs)

By doing this, you've capped your resource usage. No matter if you have 10,000 or 10 million URLs, you will only ever have 3 active network connections at any given moment. This makes your application predictable. I can tell my ops team exactly how much memory and how many sockets this process will consume, which is a huge win for stability.

The trade-off is a bit of extra boilerplate. You have to manage the lifecycle of your channels and be very careful about closing the jobs channel; if you forget to close it, your workers will hang forever waiting for more input, causing a goroutine leak. You also have to decide on the "magic number" of workers. Too few, and you're under-utilizing your hardware; too many, and you're back to the resource exhaustion problem we started with.

Usually, for I/O bound tasks (like HTTP requests), you can set the worker count relatively high. For CPU-bound tasks (like image processing), I usually set the worker count to the number of available CPU cores using runtime.NumCPU(). Any more than that, and you're just wasting time on context switching.

📋 Practical Task

Build a Concurrent Log Parser

You have a large set of simulated log files (represented as a slice of strings) that need to be scanned for the keyword "ERROR". Instead of processing them sequentially or spawning a goroutine for every single line, implement a worker pool to handle the load.

• Create a LogLine struct that holds the line content and a line number.
• Implement a worker pool with a fixed size of 4 workers.
• The workers should scan each line; if "ERROR" is found, they should send the line number and the content to a results channel.
• Ensure the program exits cleanly only after all lines have been processed and all errors have been printed to the console.