Wait, how do I stop the crawler from visiting the same page a thousand times?

This is the first thing that usually trips people up. In a sequential crawler, a simple map works fine. But once you go concurrent, you'll hit a race condition the moment two goroutines try to update that map at the same time. Go maps are not thread-safe, and your program will panic and crash faster than you can say "concurrency."

I usually recommend wrapping a standard map in a struct with a sync.Mutex. It's explicit and easy to reason about. You check if the URL is in the map; if it is, you skip it. If not, you mark it as visited and proceed. I've seen people use sync.Map for this, but for a crawler, a mutex-protected map is usually more than enough and gives you more control.

type SafeMap struct {
	mu    sync.Mutex
	sites map[string]bool
}

func (s *SafeMap) Visit(url string) bool {
	s.mu.Lock()
	defer s.mu.Unlock()
	if s.sites[url] {
		return false
	}
	s.sites[url] = true
	return true
}

How do I keep the workers running without creating an infinite loop of goroutines?

A common mistake I see is spawning a new goroutine for every single link found on a page. If you hit a page with 200 links, you suddenly have 200 goroutines. If those each find 200 links... well, you've just created a self-inflicted DDoS attack on the target server and likely crashed your own memory. Not a great look.

Instead, use a worker pool. Create a fixed number of workers (say, 10 or 20) that all read from a single chan string. This limits your concurrency to a manageable level regardless of how many links you discover. The workers just sit there, waiting for a URL to pop into the channel, processing it, and then sending any new links they find back into that same channel.

func worker(id int, jobs <-chan string, results chan<- string, sm *SafeMap) {
	for url := range jobs {
		if !sm.Visit(url) {
			continue
		}
		// Imagine fetchLinks(url) returns a slice of strings
		links := fetchLinks(url)
		for _, link := range links {
			results <- link
		}
	}
}

How do I actually know when the crawler is finished if it's constantly finding new links?

This is the "termination problem," and it's honestly the hardest part of this exercise. If your workers are reading from a channel and writing back to it, how do you know when the queue is truly empty and no more work is coming? You can't just close the channel, because a worker might still be processing a page that will produce five more URLs.

I personally find that a sync.WaitGroup combined with a "pending" counter is the cleanest way to handle this. Every time you send a URL into the channel, you increment the WaitGroup. Every time a worker finishes processing a URL (regardless of whether it found new links), you call Done(). Once the WaitGroup hits zero, you know for a fact that there are no active requests and no pending URLs. That's your signal to close the channel and shut down the workers.

Just be careful: you'll need to trigger that initial Add(1) for your starting seed URL, or your program will exit immediately before the first worker even wakes up.

📋 Practical Task

Exercise: Build a Depth-Limited Domain-Specific Crawler

Your task is to build a concurrent web crawler that adheres to two strict constraints: it must only crawl pages within a specific domain (e.g., golang.org), and it must stop once it reaches a specified depth (e.g., 3 levels deep from the seed URL).

• The Domain Filter: Ensure your crawler doesn't wander off to Twitter or GitHub. Use the net/url package to parse the host of every discovered link.
• The Depth Limit: You'll need to change your "job" from a simple string to a struct that tracks the current depth of the URL. If a URL is at the maximum depth, the worker should process the page but NOT add any discovered links back into the queue.
• The Concurrency: Use a worker pool of exactly 5 goroutines and a sync.Mutex protected map to track visited URLs.
• The Termination: Implement a mechanism (like a sync.WaitGroup) that gracefully shuts down the program once all reachable pages within the depth limit have been visited.