I've seen this exact scenario play out in countless technical interviews and, more frustratingly, in actual production outages. A developer is told to build a "fast" backend using FastAPI or Sanic, so they go all-in on async. They feel great because the code looks modern, but then the system falls over the moment it hits real traffic. Let's look at a snippet that looks correct at a glance but is actually a ticking time bomb.

from fastapi import FastAPI
import requests

app = FastAPI()

@app.get("/user-profile/{user_id}")
async def get_profile(user_id: int):
    # Simulating a call to an external legacy User Service
    response = requests.get(f"https://api.legacy-service.com/users/{user_id}")
    return response.json()

The Event Loop Freeze

If you run this locally with one user, it works perfectly. But here is the system design failure: requests.get() is a synchronous, blocking call. In a standard synchronous framework like Flask, this is fine because each request gets its own thread. But in an async environment, there is only one event loop running on the main thread.

When requests.get() is called, it doesn't "pause" the function and let other requests through; it stops the entire heart of your application. If that legacy service takes 2 seconds to respond, your entire API is dead for everyone else for those 2 seconds. You've essentially turned your high-performance asynchronous server into a very expensive sequential processor. I call this "the async illusion"—writing async def without actually using non-blocking I/O.

Implementing Non-blocking I/O with HTTPX

To fix this, we need a library that knows how to await the network response, handing control back to the event loop so it can process other incoming requests while waiting for the bytes to come back from the wire.

from fastapi import FastAPI
import httpx

app = FastAPI()

# We create a single client to reuse connections (Connection Pooling)
# This is a key system design detail to avoid socket exhaustion
client = httpx.AsyncClient()

@app.get("/user-profile/{user_id}")
async def get_profile(user_id: int):
    # Now the loop can handle other requests while this one waits
    response = await client.get(f"https://api.legacy-service.com/users/{user_id}")
    return response.json()

By switching to httpx and using await, the server can now handle thousands of concurrent requests even if the downstream service is slow. I also added a global AsyncClient. A common rookie mistake is creating a new client inside the function; doing that forces a new TCP handshake for every single request, which destroys your latency and can lead to "Too many open files" errors under load.

Decoupling with Message Queues

Non-blocking I/O solves the "waiting" problem, but it doesn't solve the "heavy lifting" problem. If your backend needs to generate a PDF or process an image, await won't save you because those are CPU-bound tasks. If you block the event loop with a heavy calculation, you're back to square one.

In a real-world system design, you move that work out of the request-response cycle entirely. Instead of making the user wait, you push a message into a broker like RabbitMQ or Redis and let a worker (like Celery) handle it in the background. The API simply returns a 202 Accepted and a task ID. This is the fundamental shift from a monolithic "do it all now" approach to a distributed, asynchronous architecture.

Scaling Horizontally vs. Vertically

When your Python backend still struggles despite async I/O, you have two levers to pull. Vertical scaling (adding more RAM/CPU) has a ceiling—and in Python, you hit the Global Interpreter Lock (GIL) ceiling pretty quickly.

Horizontal scaling is where the real system design happens. You run multiple instances of your app behind a Load Balancer (like Nginx or an AWS ALB). But remember: once you have multiple instances, you can no longer store state (like user sessions) in local memory. You have to move that state to a shared external store, typically Redis. If you're still using a local Python dictionary for caching in a distributed system, your users will experience "ghosting" where they are logged in on Server A but logged out when the load balancer hits Server B.

📋 Practical Task

Implement a Redis-Backed Distributed Cache for Expensive Queries

You have a function fetch_complex_report() that takes 5 seconds to run. Your goal is to implement a caching layer so that subsequent requests for the same report are returned instantly. To ensure this works across multiple scaled server instances, you cannot use a local variable; you must use a shared cache.

Requirements:

• Create a function get_report_data(report_id).
• Check if the report_id exists in a Redis cache.
• If it exists, return the cached value immediately.
• If it doesn't, call fetch_complex_report(), store the result in Redis with an expiration time (TTL) of 300 seconds, and then return the result.
• Use the redis-py library.

import redis
import time

# Mocking the expensive operation
def fetch_complex_report(report_id):
    time.sleep(5) 
    return f"Detailed Report Data for {report_id}"

# YOUR CODE HERE: Implement get_report_data(report_id) using a redis client