Wait, can't I just use DNS for this?

You'll hear this a lot, especially if you're working in a Kubernetes environment where K8s handles the internal DNS for you. In many cases, the answer is actually "yes." If your services are relatively static or managed by a sophisticated orchestrator, a simple DNS lookup is the cleanest way to go. No extra libraries, no extra infrastructure.

But here's the catch: DNS is often too slow for highly dynamic environments because of caching (TTL). If you're spinning up and shutting down instances of an InventoryService every few minutes based on traffic spikes, waiting for a DNS record to propagate can lead to a lot of ConnectionRefused errors. That's where a dedicated Service Registry—like Consul or Zookeeper—comes in. These tools provide a real-time "phone book" that services check before making a call, allowing for near-instant updates when a new instance joins the cluster.

How do I actually structure a client-side discovery mechanism in Scala?

In client-side discovery, the service calling the API (the client) is responsible for figuring out where the target service lives. I usually recommend abstracting this behind a trait so you can swap the implementation based on the environment (e.g., using a hardcoded map for local dev and a Consul client for production).

Let's say our OrderService needs to find the InventoryService. I'd set it up something like this:

case class ServiceInstance(host: String, port: Int)

trait ServiceDiscovery {
  def resolve(serviceName: String): Option[ServiceInstance]
}

// A production implementation might call an external API
class ConsulServiceDiscovery(consulClient: ConsulClient) extends ServiceDiscovery {
  override def resolve(serviceName: String): Option[ServiceInstance] = {
    consulClient.getHealthyInstance(serviceName)
      .map(info => ServiceInstance(info.address, info.port))
  }
}

// In your OrderService logic:
class OrderService(discovery: ServiceDiscovery) {
  def placeOrder(itemId: String): Unit = {
    discovery.resolve("inventory-service") match {
      case Some(instance) => 
        println(s"Calling Inventory at ${instance.host}:${instance.port}")
        // Perform actual HTTP call here
      case None => 
        throw new RuntimeException("Inventory service is currently unavailable!")
    }
  }
}

By keeping the ServiceDiscovery trait separate, your business logic doesn't care if you're using a fancy registry or just a config file.

What's the best way to handle "stale" service data?

This is the part that usually bites people in production. You resolve a service, you get an IP, you try to call it, and—boom—the instance just crashed. Your registry thinks it's healthy, but it's actually a zombie.

I've found that the most robust approach is a combination of Heartbeats and Client-side Load Balancing. The service should send a "still alive" signal to the registry every few seconds. If the registry misses a few beats, it marks the instance as unhealthy.

On the Scala side, you shouldn't just resolve one instance; you should resolve a list of healthy instances and use a simple round-robin or random selection. If the call fails, you immediately mark that instance as "suspect" in your local cache and try the next one. I usually wrap this in a Try or use a library like Pekko/Akka's built-in discovery mechanisms to handle the retries automatically. Don't trust the registry blindly—verify the connection at the moment of the call.

📋 Practical Task

Build a Round-Robin Service Resolver for the Order System

You have been provided with a ServiceDiscovery trait and a mock ConsulServiceDiscovery that returns a list of available instances. Your task is to create a new class called RoundRobinResolver that implements the ServiceDiscovery trait.

Requirements:

• The RoundRobinResolver should take a ConsulServiceDiscovery instance as a dependency.
• It must maintain an internal state (e.g., an atomic integer) to keep track of which instance was called last.
• Every time resolve(serviceName: String) is called, it should return the next available instance in the list provided by the Consul client, cycling back to the start when it reaches the end.
• If the Consul client returns an empty list, the resolver should return None.

Goal: Ensure that traffic to the InventoryService is distributed evenly across all healthy nodes rather than hitting the first one in the list every time.