I've spent the last two weeks tuning a new Random Forest model to predict customer churn. On my local test set, the metrics look fantastic—roughly a 3% lift in precision over our current production model. But here's the thing: I've been burned before. I once deployed a "better" model that crashed our conversion rate because it performed great on historical data but failed miserably on the actual live stream of users.

Before I let this new model (Model B) touch a single real customer, I want to run a "shadow" A/B test. I'm going to feed both the old model (Model A) and the new one the same incoming data, but I'll only act on Model A's predictions while I record what Model B would have done. Let's see if that 3% lift actually holds up when we simulate this.

The "It looks better on my screen" problem

First, I'll pull a sample of recent production data that neither model has seen. I'll start by simply comparing the accuracy of the two. I've already got my model objects loaded as model_prod and model_challenger.

# Simulating a holdout set of 1000 recent users
set.seed(42)
test_data <- data.frame(
  customer_id = 1:1000,
  actual_churn = sample(c(0, 1), 1000, replace = TRUE, prob = c(0.8, 0.2))
)

# Generating predictions for both models
test_data$pred_prod <- sample(c(0, 1), 1000, replace = TRUE, prob = c(0.75, 0.25))
test_data$pred_challenger <- sample(c(0, 1), 1000, replace = TRUE, prob = c(0.78, 0.22))

# Quick check of accuracy
mean(test_data$pred_prod == test_data$actual_churn)
# [1] 0.812
mean(test_data$pred_challenger == test_data$actual_churn)
# [1] 0.835

Okay, so Model B is at 83.5% and Model A is at 81.2%. In a vacuum, that's a win. But as a software engineer, "higher" isn't the same as "statistically significant." If I run this again with a different 1,000 users, could that gap vanish? I can't risk a deployment based on a coin flip.

Actually splitting the traffic

To do this properly, I need to treat this like a real experiment. I'll assign each user to a group. Even though both models are predicting on everyone in this shadow test, I want to analyze the performance as if they were separate cohorts. This helps me ensure there's no weird bias in how I'm calculating the lift.

# Assigning users to A or B groups
test_data$group <- sample(c("Control", "Treatment"), 1000, replace = TRUE)

# Let's calculate the error rate for each group specifically
library(dplyr)

performance <- test_data %>%
  group_by(group) %>%
  summarise(
    accuracy = mean(pred_challenger == actual_churn),
    n = n()
  )

print(performance)

Wait, I noticed something. When I split the data, the accuracy for the "Treatment" group in my simulation is actually lower than the overall average. This is a reminder that sampling noise is real. If I just looked at a small slice, I might mistakenly think the new model is worse.

The p-value reality check

I need to know if the difference between 81.2% and 83.5% is a result of a better algorithm or just the luck of the draw. Since I'm comparing two proportions (correct vs. incorrect), a prop.test is my best bet here.

# Successes (correct predictions) and totals
successes <- c(sum(test_data$pred_prod == test_data$actual_churn), 
               sum(test_data$pred_challenger == test_data$actual_churn))
totals <- c(nrow(test_data), nrow(test_data))

# Running the proportion test
test_result <- prop.test(successes, totals)
print(test_result)

If the p-value comes back as 0.15, I'm stopping right there. That means there's a 15% chance this "improvement" is just noise. In a production environment, I usually look for p < 0.05. If I don't hit that, I don't deploy. It's better to stay with a known, stable model than to move to a "better" one that provides no actual value.

Looking past the aggregate

Even if the p-value is great, there's one more thing I always check: Where is it failing? A model can have higher overall accuracy but fail catastrophically on your most valuable customers.

# Let's assume we have a 'customer_value' column
test_data$customer_value <- runif(1000, 100, 10000)

# Compare errors for high-value customers (top 10%)
high_value_cutoff <- quantile(test_data$customer_value, 0.9)
high_value_data <- test_data %>% filter(customer_value >= high_value_cutoff)

prod_error <- mean(high_value_data$pred_prod != high_value_data$actual_churn)
challenger_error <- mean(high_value_data$pred_challenger != high_value_data$actual_churn)

cat("Prod Error (High Value):", prod_error, "\n")
cat("Challenger Error (High Value):", challenger_error, "\n")

This is where the real insights happen. If Model B is 3% better overall but 10% worse on high-value customers, it's a failure. I'd rather have a slightly less accurate model overall if it protects my most important revenue streams. This kind of "slice analysis" is the difference between a data scientist who just runs scripts and an engineer who manages a product.

📋 Practical Task

Validating the Credit-Score Model Lift

You have been handed two models that predict whether a loan applicant will default: model_legacy and model_v2. You have a dataset of 2,000 applicants called loan_data with a column actual_default (1 for default, 0 for no default).

Write an R script to perform the following:

• Generate predictions for both models (you can simulate these using sample() for the sake of the exercise, but ensure the model_v2 predictions have a slightly higher accuracy).
• Calculate the accuracy for both models.
• Perform a prop.test() to determine if the difference in accuracy is statistically significant at the 0.05 level.
• Identify the "False Negative" rate (where the model predicted 0 but the actual was 1) for both models. In lending, a False Negative is much more expensive than a False Positive.
• Based on both the p-value and the False Negative rate, print a final recommendation: "DEPLOY", "REJECT", or "FURTHER TUNING".