A few years ago, I was reviewing a PR for a junior dev who was building a report generator. The task was simple: read 2,000 small XML files from a network share, parse them, and aggregate the totals. He had written a standard foreach loop. It worked perfectly in development with three files, but in staging with the full dataset, it took nearly six minutes to run. He tried to "fix" it by manually spawning a new Thread for every single file. The result? The application crashed the server's memory and triggered a frantic call from the Ops team. He had fallen into the classic trap of thinking that "more threads equals more speed," without realizing that managing those threads is where the real work happens.

This is exactly why the Task Parallel Library (TPL) exists. Instead of you manually juggling threads—which is a recipe for race conditions and memory leaks—the TPL provides a high-level abstraction. It handles the partitioning of your data and the scheduling of tasks across the available CPU cores using the .NET Thread Pool. You tell the TPL what needs to be done in parallel, and it decides how to distribute that work efficiently.

Parallelizing the Heavy Lifting

The most immediate win you'll get from the TPL is Parallel.ForEach. If you have a collection of independent items—like those XML files or a list of images to resize—you can swap your foreach for a Parallel.ForEach. The TPL will automatically split the collection into chunks and process them across multiple cores.

using System.Threading.Tasks;
using System.Collections.Generic;

public void ProcessData(List<string> filePaths)
{
    // This will utilize multiple cores automatically
    Parallel.ForEach(filePaths, filePath => 
    {
        // Imagine this is a heavy IO or CPU bound operation
        var content = File.ReadAllText(filePath);
        var result = AnalyzeContent(content);
        SaveToDatabase(result);
    });
}

One thing to keep in mind: Parallel.ForEach is synchronous. It will block the calling thread until the entire loop is finished. If you're doing this on a UI thread, your application will freeze. In those cases, you'd wrap the whole Parallel call inside a Task.Run(() => { ... }).

Taming the CPU with ParallelOptions

In a perfect world, you'd just let the TPL use every bit of power available. But in the real world, you usually have a database or an API at the other end of your loop. If you have a 32-core processor and you fire off 32 simultaneous requests to a legacy database, you might accidentally DDoS your own infrastructure. I've seen this happen more times than I care to admit.

To prevent this, we use ParallelOptions. This allows you to cap the MaxDegreeOfParallelism, effectively telling the TPL, "I know we have 32 cores, but please only use 4 at a time."

var options = new ParallelOptions 
{ 
    MaxDegreeOfParallelism = 4 
};

Parallel.ForEach(filePaths, options, filePath => 
{
    ProcessFile(filePath);
});

The Danger of Shared State

Here is the catch: parallelism is only "free" if your operations are independent. The moment you try to update a shared variable—like a total count or a shared list—you've introduced a race condition. A standard List<T> is not thread-safe. If two threads try to .Add() to the same list at the exact same microsecond, you'll either get a corrupted list or a NullReferenceException that only happens once every ten runs.

When you need to aggregate results from a parallel loop, don't use a lock if you can avoid it, as that creates a bottleneck that can make your parallel code slower than a sequential loop. Instead, reach for the System.Collections.Concurrent namespace. ConcurrentBag<T> or ConcurrentDictionary<K, V> are designed specifically for this scenario, allowing multiple threads to add items without crashing the runtime.

📋 Practical Task

Build a Parallel File Metadata Scanner

Your goal is to create a utility that scans a directory for all .txt files and calculates the total character count across all of them as quickly as possible.

Requirements:

• Create a folder on your machine and fill it with 20-50 text files of varying sizes.
• Use Directory.GetFiles() to retrieve the paths.
• Use Parallel.ForEach to iterate through the files.
• Inside the loop, read the content of each file and calculate its length.
• Store the length of each file in a ConcurrentBag<long> to ensure thread safety.
• After the parallel loop completes, use LINQ .Sum() on the ConcurrentBag to print the final total character count to the console.
• Bonus: Implement ParallelOptions to limit the parallelism to 2 cores and compare the execution time (using a Stopwatch) against a version with no limit.