Python Iteration Protocol: Iterables vs Iterators

If you use a Python list to load a 100 GB log file all at once, the process will quickly run out of memory (OOM) and the operating system will terminate it. Switch the code to stream the file through an iterator, and memory usage immediately drops to a constant level. However, if you then call len() to get the total number of records, Python raises a TypeError. If you traverse the same stream a second time, downstream computations silently receive no data.

If you are new to Python—or simply approaching the code with conventional assumptions—these behaviors can be confusing. (AI-generated code may not expose these issues directly, but you still need to understand the underlying mechanics.)

Many engineers can comfortably write for x in data: to traverse a sequence. However, they often conflate the data owner with the progress tracker. This conceptual ambiguity can easily cause production defects—for example, nested loops interfering with each other’s traversal state, or one-shot streams being mistakenly treated as reusable collections.

The key to understanding Python’s iteration model is recognizing the separation of responsibilities between an Iterable and an Iterator.

The rest of this article works through three steps:

To develop a complete mental model for stream processing, we must first answer a fundamental architectural question: why does Python not allow a container to track its own read position directly? We will start with the decoupling contract between containers and cursors.

If a list tracked its own traversal progress, nested loops would fail immediately. Once the outer loop reached its second element, starting the inner loop would reset the container’s internal cursor or advance it to the end, directly overwriting the outer loop’s state.

To prevent this state conflict, Python separates data ownership from progress tracking into two distinct roles: iterables and iterators.

An iterable is the data owner. It is responsible for storing data and providing access to it. An iterator is a forward-only cursor that tracks only the current read position.

Mental model: An iterable is like a published book, while an iterator is a bookmark placed inside it by a reader. Multiple readers can use separate bookmarks to read the same book without affecting one another’s progress.

Common misconception: anything you can put in a for loop is an iterator. In fact, list, tuple, and dict are only iterables—they carry no cursor state of their own and cannot answer 'where am I in the traversal?'.

This decoupling introduces some state-management overhead. Each traversal requires the runtime to instantiate a new iterator cursor, which means allocating a new object on the heap in interpreters such as CPython. While this creates a small allocation cost, the per-traversal allocation provides independent cursors that won’t interfere with each other’s iteration state—the same property that keeps nested loops predictable.

Python’s for loop is not fundamentally an index-based counting loop. It is syntactic sugar built on the iteration protocol. It completes traversal in three steps: acquire a cursor, advance it repeatedly, and handle the termination signal.

The following code reproduces the underlying logic of a for loop using the iteration protocol directly:

Mechanics and trade-offs:

A standard iteration protocol separates two distinct roles in code: a data holder and an independent cursor.

Under Python’s iteration protocol, the data holder stores the data and returns a fresh cursor for each iteration request. The cursor is responsible for maintaining its own forward-only traversal state.

Python’s iteration protocol uses duck typing and does not require explicit inheritance. A class satisfies the protocol contract as long as it implements the required magic methods. A cursor class must implement both __next__() and __iter__().

The rationale for self-reference: NumberCursor.__iter__() returns self directly. As a result, NumberCursor can serve as both an Iterator and an Iterable.

When developers pass an already-created cursor directly to a for loop, zip(), or enumerate(), the interpreter calls iter(cursor) and correctly receives the cursor itself. The collections.abc.Iterator abstract base class provides this same __iter__() implementation, which returns self.

Common misconceptions and trade-offs: Developers often assume that an ordinary iterable container can also return self from __iter__(). In practice, once a container returns itself, it becomes a single-use consumable stream. Nested loops then share and overwrite the same traversal state.

The hand-written two-class pattern fully decouples data storage from traversal state, but it also introduces additional boilerplate. When the iteration logic does not require complex state transitions, manually defining classes is more verbose than using generator syntax.

Python’s built-in container iterators differ fundamentally in their underlying memory layouts and their handling of concurrent structural modifications.

CPython's dict iterator uses an internal version stamp (ma_version), not a size comparison, so it detects structural changes even when the new size happens to match the old one.

Boundaries and applicable conditions: Dictionary iterator validation primarily detects changes to the number of entries. Updating the value of an existing key does not change the table structure, so the iterator keeps running.

Although hand-written cursor classes provide clear separation of responsibilities, each data stream requires its own class. The next chapter explains how generators simplify this process.

At the implementation level, a generator object fully satisfies the iterator protocol contract. It is not a separate technology or concept outside the iteration model; instead, it is Python’s higher-level abstraction for eliminating the boilerplate of hand-written cursor implementations.

In my early development, whether I was working with TypeScript or Python, I kept mistaking generators for a standalone concept that exists outside the iteration model. In reality, in Python’s type system and runtime, a generator is fundamentally just a standard iterator.

Calling a function that contains yield does not immediately execute its body. Instead, it returns a generator object.

Causal chain and mechanisms:

Common misconception: Generators are sometimes treated as a separate mechanism outside the iteration model. In practice, their interface and runtime behavior match an iterator exactly.

A function containing yield is merely a factory for generator objects. The generator instance returned by calling that function is the actual forward-only iterator cursor.

Boundaries and trade-offs: Generators eliminate the boilerplate required to implement cursor classes manually. However, the runtime stack frame fully encapsulates their internal cursor position. Generators hide their cursor position inside the runtime frame—there is no public cursor.index equivalent to peek at or reset.

The essence of stream processing is trading time for space. Generators produce one element on demand by suspending and resuming their execution stack frames at runtime.

Underlying principles and execution model (based on CPython 3.8+): When execution reaches a yield statement, the interpreter marks the generator’s stack frame as suspended. The runtime preserves the current frame’s local-variable table and the most recent instruction offset (f_lasti), then returns the yielded value to the caller. The function does not destroy its execution stack frame at this point.

When the caller next calls next(gen), the interpreter restores the generator's stack frame and resumes execution from the instruction immediately following f_lasti, continuing until it reaches another yield or the function completes and returns.

Mental model:

Trade-offs and costs: Lazy evaluation drops working-space complexity from O(N) to O(1)—the iterator only holds the current frame, never the full collection.

However, this space optimization comes at a CPU-throughput cost. Each generated element requires the generator stack frame to be suspended and resumed. If a generator pipeline becomes deeply nested, frequent stack-frame context switches and function-dispatch overhead can make total CPU time noticeably higher than a one-shot batch computation over contiguous memory.

Stream-based iterators reduce memory complexity to O(1). The trade-off is that they give up random access, known length, and repeatable iteration. This design is highly efficient for processing unbounded data streams. However, in typical application code, re-consuming an exhausted iterator does not raise an error; it simply returns no elements. As a result, downstream computations can silently omit data.

An iterator cursor stores only its current position in memory. It neither retains previously visited records nor prefetches elements that have not yet been traversed. This lightweight design directly eliminates the capabilities normally associated with sequences.

Causal chain and cost analysis:

What goes wrong with itertools.tee?: Many developers mistakenly assume that itertools.tee can clone multiple independent streaming iterators at no cost.

itertools.tee shares the underlying data source across branch iterators. It uses a first-in, first-out (FIFO) queue to buffer elements that have not yet been consumed by every branch.

If one branch consumes quickly while another lags, the FIFO buffer balloons—space complexity effectively becomes O(N) for however long the lag persists.

Three failure modes show up repeatedly in production: state exhaustion, concurrency conflicts, and resource leaks. Each calls for a specific defensive pattern.

When a generator is passed as an argument across multiple business functions, an earlier function may fully traverse it or invoke consuming built-in functions such as any() or max(), thereby exhausting its cursor. When a later function attempts to read from it again, it receives empty data and produces incorrect results.

Defensive pattern: If the dataset is known to be small and requires repeated access, explicitly convert it to a static list at the entry point with list(stream). If the dataset is too large to load entirely into memory, refactor the functions to accept a generator factory—one that returns a fresh iterator each time—instead of accepting an iterator instance directly.

Advancing a Python iterator is not an atomic operation. In a multithreaded environment, if multiple worker threads concurrently call next(iter) on the same iterator, the internal cursor state can experience a race condition. This can cause some elements to be processed more than once or skipped entirely.

Defensive pattern: Never share a bare iterator across threads. When distributing tasks across multiple threads, push data into a thread-safe queue.Queue in advance, or explicitly synchronize calls to next() with a mutex lock (threading.Lock).

Generators often encapsulate external I/O resources, such as file descriptors or database cursors. If the caller exits iteration early with break, the generator function remains suspended. System resources held internally may not be released promptly.

Defensive pattern: Protect any underlying system resource acquired inside a generator with a try...finally block or a with context manager. When the caller exits early, the generator receives a GeneratorExit (from close() or eventual GC). The finally block then runs and releases the handle.

Python separates the Iterable and Iterator protocols to support multiple independent cursors and lazy streaming computation. It achieves O(1) memory usage at the cost of random access and replayability.