Why am I getting "429 Too Many Requests" errors?

If you've seen a 429 error, it means you've effectively been put in "API timeout." Most servers have a rate limit to prevent a single user from hogging all their resources or accidentally DDoS-ing them. If the API says you can only make 5 requests per second and your R loop is hitting them at 50 per second, the server is just going to shut the door on you.

The quickest, dirtiest way to fix this is using Sys.sleep(). It's not elegant, but it works. If you know the limit is 2 requests per second, you just force R to pause for half a second between calls.

# The "quick and dirty" approach
city_ids <- c("NYC", "LDN", "TKY", "PAR", "BER")
results <- list()

for (id in city_ids) {
  results[[id]] <- httr::GET(paste0("https://api.citydata.com/pop/", id))
  # Pause for 0.5 seconds to stay under the limit
  Sys.sleep(0.5) 
}


Is there a better way than putting sleep commands inside every loop?
Honestly, manual sleeps are a pain because they slow your code down more than necessary and clutter your logic. I prefer using the ratelimitr package. It lets you "wrap" a function with a limit, so the timing happens automatically in the background. You just call the function normally, and ratelimitr ensures the calls don't happen too fast.
Here is how I usually set that up:
library(ratelimitr)
library(httr)

# 1. Define the basic function
get_population <- function(city_id) {
  res <- GET(paste0("https://api.citydata.com/pop/", city_id))
  return(content(res))
}

# 2. Create a rate-limited version: 2 calls per 1 second
limited_get_pop <- limit_rate(get_population, rate(n = 2, period = 1))

# Now you can use it in a map or a loop without worrying about Sys.sleep()
cities <- c("NYC", "LDN", "TKY", "PAR", "BER")
pop_data <- lapply(cities, limited_get_pop)


What if the server still blocks me even with a timer?
Sometimes a static timer isn't enough. Network jitter happens, or the server might have a "burst" limit that's different from its "sustained" limit. In those cases, you need a retry strategy—specifically "exponential backoff." This is where you wait a little bit after a failure, and if it fails again, you wait twice as long.
I don't write this logic from scratch anymore because httr::RETRY handles it beautifully. It will attempt the call, and if it sees a 429 (or a 503), it will pause and try again automatically.
# Instead of GET(), use RETRY()
# max_tries = 3 means it will try 4 times total
# pause_base = 1 means it waits 1s, then 2s, then 4s...
response <- httr::RETRY(
  url = "https://api.citydata.com/pop/NYC",
  times = 3,
  pause_base = 1
)

if (http_status(response)$category == "Error") {
  message("The server is really not having it today.")
}

My advice? Combine these. Use ratelimitr to be a good citizen and stay under the limit, but wrap your calls in RETRY just in case the server has a bad day.

📋 Practical Task

Exercise: Building a Throttled Weather History Scraper

You need to fetch weather data for 10 different cities from a hypothetical API. The API has a strict limit of 1 request every 2 seconds. If you exceed this, you will be blocked for an hour.

Your Task:

• Create a vector of 10 city names.
• Write a function called fetch_weather that takes a city name and uses httr::GET to call https://api.weather-example.com/history?city=[CITYNAME].
• Use the ratelimitr package to create a version of this function that is limited to 1 call every 2 seconds.
• Use lapply to execute the limited function across your vector of cities.
• Wrap the internal GET call inside httr::RETRY with 3 attempts to ensure that any intermittent 429 errors are handled gracefully.