I remember the first time I tried to write a crawler. I figured it was simple: find a link, follow it, find more links, repeat. I wrote a basic while loop that processed a list of URLs one by one. It worked, but it was agonizingly slow. I was spending 99% of my CPU time just sitting there, waiting for a server in another time zone to send back some HTML. It felt like a waste of a perfectly good machine.

The naive "more threads" approach

My first instinct was to just throw threads at the problem. If one thread is waiting on I/O, why not have ten? Or a hundred? I tried spawning a new Thread every time I encountered a new URL. It looked something like this in my head:

// This was a disaster
for (String url : discoveredLinks) {
    new Thread(() -> crawl(url)).start();
}

Within seconds, my console was a mess of stack traces. I hit a java.lang.OutOfMemoryError: unable to create new native thread. I had essentially created a fork-bomb. Every page had five links, those five pages had twenty-five links, and before I knew it, I was trying to launch thousands of threads. The OS just gave up on me. I realized that "more threads" isn't a strategy; it's a recipe for a crash.

Taming the chaos with a Pool

I needed a way to limit the concurrency. Instead of spawning threads wildly, I switched to an ExecutorService with a fixed thread pool. I decided on 10 threads—enough to keep the network pipe full without getting my IP banned by the target server.

But then I hit a new problem: cycles. I noticed my crawler was visiting the same "About Us" page over and over again. Page A linked to Page B, and Page B linked back to Page A. My threads were stuck in an infinite loop of rediscoveries. I needed a way to remember where I'd already been, but since multiple threads were checking and updating this list simultaneously, a standard HashSet started throwing ConcurrentModificationException all over the place.

Solving the shared state puzzle

I had to stop thinking about the crawler as a sequence of events and start thinking about it as a shared state problem. I swapped the HashSet for a ConcurrentHashMap.newKeySet(). This gave me a thread-safe set that allowed me to check add(url) atomically. If add returns false, it means the URL was already there, and the thread can just bail out immediately.

Here is how the logic evolved into something actually usable:

public class MultiThreadedCrawler {
    private final Set<String> visited = ConcurrentHashMap.newKeySet();
    private final ExecutorService executor = Executors.newFixedThreadPool(10);
    private final BlockingQueue<String> queue = new LinkedBlockingQueue<>();

    public void startCrawling(String seedUrl) {
        queue.add(seedUrl);
        
        // We keep a count of active tasks to know when to shut down
        while (true) {
            try {
                String url = queue.take(); // Blocks until a URL is available
                executor.submit(() -> {
                    if (visited.add(url)) {
                        System.out.println("Crawling: " + url);
                        List<String> links = fetchLinks(url);
                        queue.addAll(links);
                    }
                });
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                break;
            }
        }
    }

    private List<String> fetchLinks(String url) {
        // Imagine the HTTP logic here returning a list of strings
        return new ArrayList<>(); 
    }
}

The "When do I stop?" headache

Now I had a working engine, but I had a new problem: the program never exited. Because I used a BlockingQueue and a FixedThreadPool, the main thread just kept waiting for more work, and the worker threads stayed alive.

I realized that in a real-world crawler, you need a termination condition. Maybe it's a maximum number of pages, or a specific depth limit. For this exploration, I found that tracking the number of "in-flight" tasks was the cleanest way. If the queue is empty AND no threads are currently processing a page, the crawl is finished. I used an AtomicInteger to track active tasks, incrementing it before executor.submit() and decrementing it in a finally block inside the worker task.

It's a subtle shift, but this is where Java concurrency gets interesting. You aren't just writing logic; you're managing the lifecycle of resources and ensuring that your threads communicate their status without locking each other into a deadlock.

📋 Practical Task

Exercise: Implementing the Depth-Limited Crawler

Your task is to modify the MultiThreadedCrawler logic to prevent it from wandering too deep into the web. Currently, the crawler will follow links indefinitely as long as they are new.

Requirements:

• Modify the queue to store a custom PageRequest object instead of a simple String. This object should contain the url and the currentDepth.
• Introduce a constant MAX_DEPTH = 3.
• Update the worker logic: If a PageRequest has a depth equal to MAX_DEPTH, the crawler should record the page but not add any newly discovered links back into the queue.
• Ensure that when a link is discovered and added to the queue, its depth is parentDepth + 1.

Goal: Ensure that your crawler stops expanding the search tree once it reaches the 3rd level of links, while still using the ExecutorService to process those levels in parallel.