I've seen this play out in plenty of code reviews: a developer hits a RecursionError, discovers sys.setrecursionlimit(), bumps the number up to 10,000, and assumes the problem is solved. They think they've just "unlocked" more capacity for their algorithm. This is a dangerous misunderstanding of how Python handles memory.

The Myth: Increasing the Limit Fixes Everything

The misconception is that the recursion limit is the only thing standing between you and a deep recursive call. It isn't. The limit is actually a safety rail designed to prevent a much nastier failure: a hard crash of the entire Python interpreter.

import sys

# The "dangerous" fix
sys.setrecursionlimit(100000)

def deep_dive(n):
    if n == 0:
        return 0
    return deep_dive(n - 1)

# This might not raise RecursionError anymore, 
# but it might just kill your process entirely.
deep_dive(100000) 

Here is what's actually happening. Every time you call a function, Python pushes a "frame" onto the C stack. This frame contains the local variables and the return address. The OS allocates a fixed amount of memory for this stack. While sys.setrecursionlimit() tells Python, "Yes, you're allowed to keep going," it doesn't tell the operating system to allocate more physical memory for the stack. If you push too many frames, you'll hit a Stack Overflow—a segmentation fault that crashes the program instantly without a helpful Python traceback. I've spent way too many hours debugging "silent" crashes that turned out to be exactly this.

The Hope: Tail Call Optimization as a Silver Bullet

If you've spent time with functional languages like Haskell or Scheme, you're probably looking for Tail Call Optimization (TCO). You've likely heard that if the recursive call is the very last action of a function, the compiler can "reuse" the current stack frame instead of adding a new one, effectively turning recursion into a loop.

Here is the cold truth: Python does not support TCO. Period. Guido van Rossum, Python's creator, has been very clear about this. He values the clarity of the traceback—the ability to see exactly how you got to a specific line of code—over the optimization of recursive calls. If Python optimized the tail call, the frame would vanish, and your stack trace would be missing the very history you need to debug a production crash.

Because of this, writing "tail-recursive" code in Python is a stylistic choice, not a performance one. Whether you write your recursive call as the final line or wrap it in another operation, you are still consuming stack space. If your data depth is unpredictable or potentially massive, recursion in Python is simply the wrong tool for the job.

The Correct Path: Manual Stack Management

When you hit the limits of the stack, the solution isn't to move the limit; it's to move the data. By using an explicit list as a stack (stored on the heap), you are limited only by your total available RAM, not by the OS's narrow thread-stack allocation.

Instead of letting the language manage the frames for you, you manage the state yourself. It's a bit more boilerplate, but it's the only way to ensure your code doesn't spontaneously combust when it encounters a deeply nested data structure.

📋 Practical Task

Converting a Deep Recursive Directory Walker to an Iterative Stack

You have been given a recursive function designed to list every file in a deeply nested directory structure. However, it crashes with a RecursionError (or a segmentation fault) when run against a file system with thousands of nested folders.

Your Task: Rewrite the walk_directories function to be iterative. Instead of calling itself, use a Python list as a manual stack to keep track of the directories that still need to be visited.

import os

# This is the BROKEN recursive version
def walk_directories_recursive(path):
    print(f"Visiting: {path}")
    try:
        for entry in os.scandir(path):
            if entry.is_dir():
                walk_directories_recursive(entry.path) # This is where it fails on deep paths
    except PermissionError:
        pass

# TODO: Implement the iterative version below
def walk_directories_iterative(start_path):
    # Your code here
    pass

# Test it with a path on your machine
# walk_directories_iterative("/your/deep/path")