I've seen this specific mistake more times than I care to admit. A junior analyst comes to me with a "successful" A/B test result, but when I look at the code, the math is fundamentally skewed because they've fallen into the global mean trap.

# The "Broken" Analysis
# df contains 'group' (A or B) and 'converted' (0 or 1)
conversion_b <- mean(df$converted[df$group == "B"])
overall_mean <- mean(df$converted)

# Calculating lift
lift <- conversion_b - overall_mean
print(paste("The lift for Group B is:", lift))

The "Global Mean" Trap

At first glance, this looks fine. The analyst is trying to see how much better Group B performed compared to the average. But here is the problem: overall_mean includes the data from Group B. By subtracting the global mean from Group B's mean, you aren't measuring the lift relative to the control (Group A); you're measuring the lift relative to a weighted average of both groups. This shrinks the perceived effect and gives you a mathematically incorrect lift value.

Isolating the Control Baseline

To fix this, we have to be explicit. In A/B testing, the control group is your ground truth. Everything must be measured relative to Group A, not the dataset as a whole. Here is how we actually handle that logic:

# The Fixed Analysis
conv_a <- mean(df$converted[df$group == "A"])
conv_b <- mean(df$converted[df$group == "B"])

# True lift: (Treatment - Control) / Control
lift <- (conv_b - conv_a) / conv_a
print(paste("The true relative lift is:", round(lift * 100, 2), "%"))

Now we're comparing apples to apples. I prefer calculating relative lift (the percentage increase) over absolute lift because it provides a much better sense of the business impact, regardless of whether your baseline conversion is 1% or 20%.

Modularizing the Analysis into a Pipeline

Doing this manually for one metric is easy. Doing it for five metrics across ten different experiments is a nightmare. To make this sustainable, we need a pipeline. I usually build this as a function that takes a raw dataframe and a metric column name, returning a clean summary. This prevents "copy-paste errors" where you accidentally reference conv_a in a conv_b calculation.

analyze_ab_test <- function(data, metric_col) {
  # Extract values for each group
  group_a <- data[[metric_col]][data$group == "A"]
  group_b <- data[[metric_col]][data$group == "B"]
  
  # Basic stats
  m_a <- mean(group_a)
  m_b <- mean(group_b)
  lift <- (m_b - m_a) / m_a
  
  # Statistical significance using a proportion test
  # prop.test expects counts of successes and total trials
  test_result <- prop.test(x = c(sum(group_a), sum(group_b)), 
                           n = c(length(group_a), length(group_b)))
  
  return(data.frame(
    metric = metric_col,
    control_rate = m_a,
    treatment_rate = m_b,
    lift = lift,
    p_value = test_result$p.value,
    significant = test_result$p.value < 0.05
  ))
}

# Running the pipeline on multiple metrics
metrics <- c("converted", "clicked_upsell", "completed_profile")
results <- do.call(rbind, lapply(metrics, function(m) analyze_ab_test(df, m)))

Adding Rigor with Confidence Intervals

A p-value tells you if there is an effect, but it doesn't tell you the magnitude of that effect. If you tell a stakeholder "it's significant," their first question will be "by how much?"

I always append the confidence interval of the difference to my pipeline. Since we are dealing with proportions, we can extract the conf.int from the prop.test object. If the interval is very wide, it means your sample size is likely too small, and you shouldn't trust the lift value even if the p-value looks promising. If the interval is tight and stays above zero, you've got a winner.

📋 Practical Task

Building a Multi-Metric Conversion Pipeline for E-commerce

You have been handed a dataset called ecommerce_data with the following columns: user_id, group (either "A" or "B"), purchased (0 or 1), and signed_up (0 or 1).

Your task is to create a robust analysis pipeline. Write a function called run_ab_pipeline that:

• Accepts the dataframe and a vector of metric column names.
• Calculates the conversion rate for both groups for every metric provided.
• Calculates the relative lift (Treatment - Control) / Control.
• Performs a prop.test to determine the p-value for each metric.
• Returns a single combined dataframe containing the metric name, the lift, and the p-value.

Test your function by passing c("purchased", "signed_up") as the metrics. Ensure your output is a tidy dataframe where each row represents a different metric.