I remember early in my career, I wrote a payment handler that looked perfectly fine on my local machine. It passed every single-threaded test I threw at it. Then we hit production, and suddenly, users were reporting that their balances weren't updating correctly, or worse, payments were being triggered twice while the balance only dropped once. When I looked at the logs, it was a nightmare of race conditions.

Here is a snippet of the kind of code that leads to those midnight emergency calls. This is a simplified version of a payment service using Cats Effect, but it contains two critical flaws that are incredibly common when you're first moving from imperative Scala to functional effects.

case class Account(id: String, balance: Double)

class PaymentService {
  private var accounts = Map("user1" -> Account("user1", 100.0))

  def processPayment(userId: String, amount: Double): IO[Unit] = {
    val account = accounts(userId)
    if (account.balance >= amount) {
      // Bug 1: Mutating state unsafely
      accounts = accounts.updated(userId, account.copy(balance = account.balance - amount))
      
      // Bug 2: The "Dangling IO"
      IO.println(s"Charging $amount to $userId") *> 
      externalPaymentGateway.charge(userId, amount) 
    } else {
      IO.raiseError(new Exception("Insufficient funds"))
    }
  }
}

The Danger of Mutating State in Fibers

The first thing that should jump out at you is that var accounts. In a standard Scala app, a var is just a variable. But in a Cats Effect application, your code is likely running across multiple fibers on a multi-threaded executor.

When two fibers call processPayment at the exact same time, they both read the same initial balance, subtract the amount, and then overwrite each other. One of those payments effectively "disappears" from the balance record, even though the money was charged. I've seen this happen in high-throughput systems where the bug only manifests once every ten thousand transactions, making it a total pain to debug.

To fix this, we need Ref. A Ref is essentially a purely functional atomic reference. It ensures that updates to your state are linearizable and thread-safe without you having to manually manage synchronized blocks or locks, which usually just lead to deadlocks anyway.

The Ghost of the Unrun IO

Now, look at the if block again. Notice how I'm updating the map and then returning an IO? This is a classic "impure" leak. The state update happens immediately when the function is called, but the externalPaymentGateway.charge call is wrapped in an IO.

Remember: an IO is just a description of a program. It does nothing until it is run. If the charge call fails later in the effect chain, the balance has already been deducted because the mutation happened outside the IO. You've just stolen money from your user without actually processing the payment. We need to wrap the state transition itself into the IO chain.

Implementing Atomic Transitions

Here is how we actually build this. We'll use Ref to manage the account map and ensure that the balance check and the deduction happen as one single, atomic operation using modify.

case class Account(id: String, balance: Double)

class PaymentService(accountsRef: Ref[IO, Map[String, Account]]) {
  
  def processPayment(userId: String, amount: Double): IO[Unit] = {
    for {
      // Atomically check and update the balance
      updated <- accountsRef.modify { accounts =>
        val account = accounts.getOrElse(userId, Account(userId, 0.0))
        if (account.balance >= amount) {
          val newAccount = account.copy(balance = account.balance - amount)
          (accounts.updated(userId, newAccount), Right(()))
        } else {
          (accounts, Left(new Exception("Insufficient funds")))
        }
      }
      
      // Handle the result of the atomic update
      _ <- updated match {
        case Right(_) => 
          IO.println(s"Charging $amount to $userId") *> 
          externalPaymentGateway.charge(userId, amount)
        case Left(err) => 
          IO.raiseError(err)
      }
    } yield ()
  }
}

Why is this better? First, accountsRef.modify ensures that no other fiber can sneak in and change the balance between the time we check it and the time we update it. Second, the entire sequence—from checking the balance to calling the gateway—is now one continuous IO chain. If the service that calls processPayment doesn't actually run the resulting IO, nothing happens. No money is moved, and no state is changed. That's the power of referential transparency.

• Ref handles the concurrency for us.
• modify allows us to perform a "check-and-set" operation atomically.
• Returning IO ensures that side effects only happen when the program is explicitly executed.

📋 Practical Task

Implement a Payment Retry Mechanism with Timeout

In a real payment service, external gateways fail all the time. Your task is to extend the PaymentService logic to make the payment process more resilient.

Modify the processPayment method (or create a wrapper) that implements the following requirements:

• The externalPaymentGateway.charge call should be wrapped in a timeout of 2 seconds. If it takes longer, it should be considered a failure.
• If the charge fails (either via a timeout or a thrown exception), the service should retry the call exactly 3 times with a 100ms delay between attempts.
• Crucially: If all 3 retries fail, you must "refund" the balance by atomically adding the amount back to the user's account in the Ref before finally raising the error.

Your solution should use IO.timeout, IO.sleep, and Ref.modify to ensure that the user is not charged if the external gateway is permanently unreachable.