You've likely been there: a production incident hits, a customer says their checkout failed, and you're staring at a mountain of logs from five different microservices. You search for a userId or a orderId, and you find a few scattered entries, but you can't tell for sure if the delay happened in the database, the payment gateway, or the network hop between your Order Service and your Shipping Service. It's like trying to reconstruct a crime scene from five different blurry polaroids taken by five different people.

The Correlation ID Rabbit Hole

The first instinct for most of us is to implement "manual correlation." I've seen this in a dozen codebases. You create a RequestContext case class containing a correlationId: UUID, and then you pass that object into every single function call in your entire call stack. It starts out innocent enough: def processOrder(order: Order, ctx: RequestContext). But then you realize your PaymentClient needs it, your InventoryService needs it, and your EmailNotifier needs it.

// The "Naive" Way: Manual Propagation
def checkout(cart: Cart, ctx: RequestContext): Response = {
  logger.info(s"[${ctx.correlationId}] Starting checkout for ${cart.id}")
  val order = orderService.createOrder(cart, ctx) // Passing ctx manually...
  val payment = paymentService.authorize(order, ctx) // ...and again...
  paymentService.capture(order, ctx) // ...and again.
  Response.Ok
}


This approach is a nightmare for a few reasons. First, it pollutes your business logic. Your domain services now depend on a tracing object that has absolutely nothing to do with the actual logic of "checking out a cart." Second, it's fragile. The moment a junior dev adds a new helper method and forgets to pass the ctx parameter, your trace chain breaks, and you're back to searching for needles in a haystack. Most importantly, a correlation ID tells you what happened, but it doesn't tell you how long each step took or the parent-child relationship between calls.

Letting OpenTelemetry Handle the Context
This is where OpenTelemetry (OTel) comes in. Instead of passing IDs around like hot potatoes, OTel uses a "Context" that lives outside your immediate function arguments—usually managed via a ThreadLocal (in blocking code) or integrated into the effect system (like ZIO or Cats Effect) in functional Scala. Instead of a flat ID, we use Spans. A span represents a single operation: it has a start time, an end time, and a parent span ID.

When you use the OTel SDK, you wrap your logic in a span. If a span is already active when you start a new one, OTel automatically marks the new span as a child of the current one. This creates a directed acyclic graph (DAG) of your entire request flow.

// The Better Way: OpenTelemetry Spans
import io.opentelemetry.api.GlobalOpenTelemetry
import io.opentelemetry.api.trace.Span

val tracer = GlobalOpenTelemetry.getTracer("order-service")

def checkout(cart: Cart): Response = {
  val span = tracer.spanBuilder("checkout_process").startSpan()
  try {
    span.setAttribute("cart.id", cart.id)
    
    // These internal calls will automatically detect the active span
    val order = orderService.createOrder(cart) 
    paymentService.authorize(order)
    
    Response.Ok
  } catch {
    case e: Exception => 
      span.recordException(e)
      span.setStatus(StatusCode.ERROR, "Checkout failed")
      throw e
  } finally {
    span.end()
  }
}


Notice how orderService.createOrder no longer needs to take a RequestContext. The OTel SDK handles the propagation. If createOrder makes an HTTP call to another service, the OTel instrumentation for your HTTP client (like sttp or Akka HTTP) will automatically inject the traceparent header (following the W3C Trace Context standard) into the request. The receiving service picks up that header and starts its own span as a child of yours. Magic. Well, not magic—just a standardized header and a shared context.

The Trade-off: Instrumentation Overhead
Now, I won't tell you this is free. Implementing OTel adds some complexity to your infrastructure. You now need a Collector—a separate process that receives these spans and pushes them to a backend like Jaeger, Zipkin, or Honeycomb. You also have to be careful about "span bloat." If you create a span for every single tiny helper function, you'll generate gigabytes of telemetry data and potentially slow down your application due to the overhead of creating and exporting these objects.

The rule of thumb I use is: instrument the boundaries. Wrap your API endpoints, your database queries, and your external HTTP calls. Only go deeper into the business logic if a specific area is a known performance bottleneck. It's far better to have a few high-quality, meaningful spans than a million tiny ones that make your trace visualization look like a bowl of spaghetti.

📋 Practical Task

Exercise: Instrumenting the Payment Gateway Handshake

You are given a Scala project with two services: OrderService and PaymentService. Currently, they communicate via HTTP, but there is no tracing. When a payment fails, you can't tell if the failure happened during the OrderService's request construction or inside the PaymentService's processing logic.

Your task:

• Integrate the OpenTelemetry Java SDK into both services.
• Wrap the checkout method in OrderService in a span named "process_checkout".
• Wrap the handlePayment method in PaymentService in a span named "authorize_payment".
• Ensure that the PaymentService span is correctly identified as a child of the OrderService span by configuring the HTTP client and server to propagate the W3C Trace Context headers.
• Add a custom attribute "payment.amount" to the span in the PaymentService.

Success Criteria: When you run the provided test suite and trigger a checkout, the exported trace in the local Jaeger UI should show a single trace containing two nested spans, with the PaymentService span indented under the OrderService span.