I've got a messy text file here—about 500MB of server access logs. Each line is just a string of data: IP address, timestamp, the HTTP method, the requested path, and the status code. I want to find out which endpoints are getting hammered the most, but if I try to load this into a standard Scala List, my JVM is going to scream and die. This is where Spark's RDD (Resilient Distributed Dataset) comes in.

Wrestling with a raw text file

First, I'll just get the data into Spark. I'm using sc.textFile to create my RDD. At this point, it's just an RDD of strings—one string per line.

val logs = sc.textFile("server_logs.txt")
val firstLine = logs.first() 
// Result: "192.168.1.1 [2023-10-01 10:00:01] GET /api/v1/users 200"

Alright, the data is there. Now I want to extract just the paths (like /api/v1/users). I'll use a map to split the string by spaces and grab the 4th element.

val paths = logs.map(line => line.split(" ")(3))
println("Transformation applied!")

Here is where most people get tripped up the first time. I ran that code, and it finished almost instantaneously. I thought, "Wait, it can't have processed 500MB of text in 10 milliseconds." I checked my CPU usage, and it was flat. Nothing happened. This is Spark's "Lazy Evaluation" in action. The map didn't actually move a single byte of data; it just recorded a plan in a DAG (Directed Acyclic Graph) saying, "When the user finally asks for a result, I need to split these strings."

Actually triggering the work

To actually see if my map worked, I need an Action. Actions are the trigger that tells Spark to stop planning and start executing. I'll use take(5) to grab a few samples.

val samples = paths.take(5)
samples.foreach(println)
// Output: 
// /api/v1/users
// /index.html
// /api/v1/products
// /api/v1/users
// /favicon.ico

Now I see the work actually happening. But wait—I've got some junk in here. I don't care about /favicon.ico or /robots.txt. I'll chain a filter transformation to get rid of the noise.

val cleanPaths = paths.filter(path => !path.contains("favicon") && !path.contains("robots"))

Again, no work is done yet. I'm just building a pipeline: textFile → map → filter.

The count and the shuffle

Now for the hard part: counting occurrences. I want a result like ("/api/v1/users", 450). The standard pattern in Spark for this is the "Map-Reduce" approach. First, I transform every path into a pair where the value is 1.

val pathPairs = cleanPaths.map(path => (path, 1))

Now I have an RDD of tuples: ("/api/v1/users", 1), ("/index.html", 1), ("/api/v1/users", 1). I need to sum these up by the key (the path). I could use groupByKey, but I've learned the hard way that groupByKey is a performance killer because it sends every single record across the network to the reducer. Instead, I'll use reduceByKey.

val pathCounts = pathPairs.reduceByKey(_ + _)

reduceByKey is smarter. It performs a "local combine" on each partition before shuffling the data across the cluster, which saves a massive amount of network bandwidth. Let's trigger the final action and see the top 5 endpoints.

val topEndpoints = pathCounts.takeOrdered(5)(Ordering.by((a, b) => b._2.compare(a._2)))
topEndpoints.foreach { case (path, count) => println(s"$path: $count") }

Everything clicked. I went from a raw text file to a ranked list of endpoints by defining a series of transformations and triggering them with a single action. The beauty here is that if one of the worker nodes had crashed halfway through, Spark would just look at the DAG and re-run the transformations on the missing partition to recover the data.

📋 Practical Task

Analyzing E-commerce Order Categories

You have a dataset of order logs where each line follows the format: OrderID, Category, Amount, Status (e.g., "101,Electronics,299.99,Completed"). Some orders are "Cancelled" and should be ignored.

Write a Spark program that does the following:

• Loads the text file using sc.textFile.
• Filters out any rows where the status is "Cancelled".
• Transforms the data into a pair RDD where the key is the Category and the value is the Amount (converted to Double).
• Uses a transformation to calculate the total spend per category.
• Collects and prints the final results.

Constraints: Ensure you use reduceByKey for the aggregation to optimize network shuffle, and remember that you must call an action (like collect()) to actually execute the pipeline.