Listen, the biggest headache in distributed systems isn't usually the logic—it's the data changing under your feet. You deploy a new version of a service, it starts writing data to Kafka using a new schema, and suddenly your downstream consumers start crashing because they don't know what to do with the new fields. That's where Avro's schema evolution comes in.

Avro is designed specifically to handle this "version drift." The magic happens because Avro requires both the writer's schema (what the data was written with) and the reader's schema (what the current code expects). If they differ, Avro tries to resolve them using a set of predefined rules.

Defining our first UserProfile

Let's build a simple user profile system. I'm using avro4s here because writing raw JSON schema files by hand is a recipe for typos and misery. In Scala, we can just use case classes.

import avro4s._
import avro4s.AvroSchema

case class UserProfileV1(id: Long, username: String)

val schemaV1 = AvroSchema[UserProfileV1]
// This generates a schema with two required fields: id and username


At this point, everything is simple. We write a UserProfileV1, we read it back. No drama.

Adding an optional email without breaking the world
Now, imagine the business decides we need to capture user emails. If I just add a field, any old data sitting in my logs or Kafka topics won't have an email. If the reader expects one and doesn't find it, the whole process blows up.

To evolve this safely, we need a default value. This is the golden rule of Avro evolution: if you add a field, you must give it a default so the reader knows what to plug in when processing old records.

case class UserProfileV2(
  id: Long, 
  username: String, 
  email: Option[String] = None // The Option + default value is key here
)

val schemaV2 = AvroSchema[UserProfileV2]


Because I used Option and provided None, Avro marks this field as having a default of null in the schema. When UserProfileV2 reads a V1 record, it sees the email is missing and simply fills it with None. This is backward compatible.

The "Oops" moment: The required field trap
Here is where I usually trip up when I'm rushing a feature. Let's say I want to add a accountTier field (like "Bronze", "Silver", "Gold"). I decide that every user must have a tier, so I add it as a plain String.

// I'm making a mistake here...
case class UserProfileV3(
  id: Long, 
  username: String, 
  email: Option[String] = None, 
  accountTier: String // Oops! No default value.
)


I deploy this. The new code starts reading old V2 data. Suddenly, I get a AvroTypeException: "Found UserProfileV2, expecting UserProfileV3. Field accountTier is missing."

I forgot that "required" in the reader's schema means "must exist in the writer's schema." If the old data doesn't have it, and there's no default to fall back on, Avro has no choice but to fail. I've just broken my production pipeline.

Fixing the evolution with a sensible default
To fix this, I have to ensure that any new field added to the schema is either optional or has a fallback value. If accountTier is mandatory for the business logic, I should still provide a "safe" default for legacy data.

case class UserProfileV3Fixed(
  id: Long, 
  username: String, 
  email: Option[String] = None, 
  accountTier: String = "Bronze" // Now it's safe!
)


Now, when the V3Fixed reader encounters a V1 or V2 record, it says: "I don't see an accountTier here, but the schema tells me to use 'Bronze' if it's missing." Everything flows smoothly again.

Just remember: if you're evolving a schema in a production environment, never add a field without a default, and never remove a field that didn't have a default (because old readers won't know how to handle its absence). Keep it simple, keep it optional, or keep it defaulted.

📋 Practical Task

Exercise: Implementing a Compatible Version 3 for OrderEvents

You are maintaining an event-driven system for an e-commerce store. You currently have the following case class representing an order event:

case class OrderEvent(orderId: String, totalAmount: Double)

Your task is to evolve this schema to version 2 and version 3 while ensuring backward compatibility (meaning the newest code can read the oldest data).

• Requirement 1: Create OrderEventV2. Add a field currency (String). Since old orders don't have this, it must default to "USD".
• Requirement 2: Create OrderEventV3. Add a field discountCode. This field should be optional (use Option[String]) and default to None.

Write the Scala case classes for OrderEventV2 and OrderEventV3. Ensure that both follow the rules of Avro schema evolution so that a V3 reader can successfully process a V1 record without throwing an exception.