I've seen this exact script on a dozen different dashboards across a dozen different companies. A developer wants to be "agile," so they write a loop that checks the p-value of an A/B test every day. The moment the p-value dips below 0.05, the script triggers an alert, and the team declares a winner. It feels efficient, but it's mathematically fraudulent.

# The "Peeking" Disaster
check_results <- function(data) {
  # Assuming data has 'group' and 'converted' columns
  test <- prop.test(x = c(sum(data$converted[data$group == "A"]), 
                          sum(data$converted[data$group == "B"])), 
                    n = c(sum(data$group == "A"), 
                          sum(data$group == "B")))
  
  if (test$p.value < 0.05) {
    return("Stop test: Significant result found!")
  } else {
    return("Continue collecting data...")
  }
}

The Peeking Trap: Inflating your Type I Error

The code above looks fine if you're used to basic statistics, but here is the problem: you are performing multiple hypothesis tests on the same accumulating data. Every time you "peek" at the p-value and decide whether to stop, you are giving the data another chance to accidentally cross the 0.05 threshold due to random noise.

If you peek 10 times during a test, your actual probability of a Type I error (a false positive) isn't 5%—it's closer to 15-20%. You're essentially rolling the dice over and over again until you get the result you want, then stopping. In a production environment, this leads to "winning" features that actually do nothing or, worse, hurt your metrics.

Allocating Alpha with the O'Brien-Fleming Boundary

To fix this, we use Alpha Spending. Instead of using a flat 0.05 for every peek, we "spend" a tiny portion of our total alpha budget at each look. If we plan to peek five times, we might only allow a p-value of 0.001 for the first peek, 0.01 for the second, and so on, ending with something close to 0.05 for the final look.

The gsDesign package is the gold standard for this in R. It allows us to define a "group sequential design," which tells us exactly what the critical value for the test statistic should be at each interval.

library(gsDesign)

Let's rewrite our logic. Instead of a blind 0.05 check, we define a design with 5 planned looks and use the O'Brien-Fleming spending function, which is conservative early on to prevent premature stopping.

# Define a sequential design
# k = number of looks, alpha = total budget, beta = power (1 - 0.8)
design <- gsDesign(k = 5, test.type = 2, alpha = 0.05, beta = 0.2, sfu = "O'Brien-Fleming")

# This 'design' object now contains the 'upper' and 'lower' boundaries 
# for the Z-score at each look.
print(design$upper$bound) 
# Output will show boundaries like 4.0, 2.8, 2.3, 2.0, 1.96

Now, instead of checking the p-value, we check if the Z-score of our current result exceeds the boundary for that specific look. The Z-score is simply the test statistic; for a proportion test, it's the difference in proportions divided by the pooled standard error.

check_results_sequential <- function(data, look_number) {
  # Calculate Z-score manually for the current data
  pA <- mean(data$converted[data$group == "A"])
  pB <- mean(data$converted[data$group == "B"])
  nA <- sum(data$group == "A")
  nB <- sum(data$group == "B")
  
  pooled_p <- sum(data$converted) / sum(data$group)
  se <- sqrt(pooled_p * (1 - pooled_p) * (1/nA + 1/nB))
  z_score <- abs((pB - pA) / se)
  
  # Get the boundary for the current look from our gsDesign object
  boundary <- design$upper$bound[look_number]
  
  if (z_score > boundary) {
    return("Stop: Statistically significant based on alpha spending!")
  } else {
    return("Continue: Boundary not yet crossed.")
  }
}

By using this approach, the total Type I error across all five looks remains exactly 0.05. You get the benefit of stopping early if there's a massive effect, but you protect yourself from the "noise" of daily fluctuations. I usually recommend the O'Brien-Fleming boundary because it's very strict early on, which forces the team to gather enough data before making a call.

📋 Practical Task

Implementing an Alpha-Spending Monitor for a Checkout Page Test

You are tasked with monitoring a conversion rate test for a new checkout flow. You have decided to use a sequential design with 4 planned looks. Use the gsDesign package to create a monitoring system.

Requirements:

• Create a gsDesign object with k = 4, alpha = 0.05, beta = 0.2, and sfu = "O'Brien-Fleming".
• Write a function called evaluate_peek that takes three arguments: the current z_score, the look_index (1 through 4), and your design object.
• The function should return TRUE if the z_score exceeds the boundary for that specific look, and FALSE otherwise.
• Test your function with a z_score of 3.5 at look_index = 1 (should likely be FALSE) and a z_score of 2.1 at look_index = 4 (should likely be TRUE).