Splitting One Input Stream into Multiple Processing Paths with `itertools.tee`

When processing a 5 GB payment reconciliation file, a program may need to run account matching and discrepancy auditing at the same time. Using list() lets both branches reuse the data, but loading the entire file into memory can trigger an out-of-memory (OOM) error. Sharing a raw generator avoids the memory overhead, but both consumers then advance the same cursor, causing the later branch to miss records consumed by the earlier one.

This creates a real engineering constraint: the input stream can only be read once, in order, but multiple downstream steps must each see the complete, consistent sequence.

The key to using itertools.tee is understanding how it balances data reuse with memory usage through a controlled shared buffer. The fundamental challenge: when an input stream can only be read once, how can multiple downstream steps each obtain a complete view of the data?

tee takes a single-consumer stream—where the first reader claims each item—and exposes it as several views that independent steps can consume in order. It works well when the input can't be rewound and fetching it again isn't practical.

In a batch CSV import, both field validation and invalid-row auditing need to inspect the same rows. Converting the file to a list lets both steps iterate over the data repeatedly. But as the file grows, all rows remain in memory for longer. The process may eventually run out of memory (OOM). Use tee to split validation and auditing into separate branches and finish both within each batch, so the file never has to be fully materialized in memory.

In payment reconciliation, transaction matching and discrepancy reports need to read the same records. If each step reads from object storage and parses the text independently, the upstream I/O and parsing happen twice. tee lets both steps share one sequential read. But if report generation stays slower than reconciliation, records that the report hasn't consumed yet still occupy the shared buffer.

Customer-support ticket exports face the same slow-branch and buffer-growth constraints. You can parse a JSON Lines stream—one JSON object per line—once, then generate a redacted download file and internal quality metrics from the separate branches. Generate the download file in a controlled batch process. Streaming directly to an unbounded, slow client causes network backpressure that stalls the download branch, preventing the shared buffer from being reclaimed.

Having two steps alternately call the same iterator isn't distribution. Once one step retrieves an element, the iterator has advanced. The other step can see only later elements, so the application skips records. tee creates multiple independent iterators. Each one can observe the same input sequence in order.

Engineers often assume tee runs concurrently out of the box, but it handles neither threading, scheduling, failure isolation, nor retries. Its purpose is solely to eliminate duplicate upstream reads—not to provide zero-memory broadcasting.The farther one branch falls behind, the more elements the system must retain for it. Memory usage depends mainly on the consumer lag between branches.

Use tee when you have only a few branches, their consumption rates stay reasonably close, and all processing can finish within the current process. For cross-process distribution, durable replay, or consumers that may remain far behind for a long time, use a message queue, files on disk, or a database.

The core idea behind itertools.tee is simple: it treats the upstream iterator as a conveyor belt that can move only forward, then gives each branch its own bookmark. This lets multiple branches observe the same sequence without reading the upstream data more than once.

When you call a, b = tee(source, 2), nothing is read from source immediately. The next item is pulled from source only when one branch, such as a, calls next(a).

That item goes into a shared buffer. If a is the first branch to reach it, a adds the item to the buffer and advances its own bookmark. When the slower branch, b, reads the same item later, it gets a reference from the buffer instead of triggering another read from source.

The bookmark model explains why both branches see the same data in the same order. But the buffer is not unbounded. The slowest branch's bookmark determines how much data the system must retain. A lagging bookmark forces the system to keep the items in between. An item can be released only after every branch has moved past it.

Boundary: The two branches receive references to the same objects. They do not receive deep copies. If an item is mutable, changes made by one branch will be visible to the other.

tee memory usage depends entirely on consumer lag between branches. In production code, advance both branches together in bounded batches. Don't fully consume one branch before processing the other.

For example, if you call list(validation_branch) first, the validation branch keeps pulling from upstream while the audit branch's bookmark remains at the beginning. The shared buffer must then retain nearly the entire input for the audit branch. This effectively becomes full loading again and can lead to Out of Memory (OOM).

The second argument to tee controls the number of branches, not the buffer size. There is also no public parameter for setting a buffer limit. But you can use external batch-processing logic to bound the maximum consumer lag.

The following example uses islice to make both branches consume the same batch in lockstep. This approximately limits the maximum buffer size to the batch size:

This batch coordination means the two branches cannot be consumed independently without limits. If your workload naturally requires one branch to process data hours or days after the other, use durable intermediate storage such as a database or message queue. Don't rely on tee's in-process buffer.

itertools.tee duplicates iteration capability without copying the entire dataset. It achieves this by buffering values in shared data blocks while each branch independently tracks its own read position.

In CPython 3.12 and 3.13, the core of tee is the teedataobject, a segmented buffer. The upstream iterator is called only when a branch requests a new element for the first time. tee then writes that element to a shared teedataobject block. These blocks form a linked list that stores elements already consumed by leading branches but still retained for lagging ones.

As the fast branch moves forward, it keeps pulling values from the upstream iterator and filling new buffer blocks. Each branch iterator maintains only a pointer to its current read position.

The cost: A data block can be released only after every live branch has moved past it. The slowest branch therefore determines the memory retention window. This design avoids calling the upstream iterator more than once for the same element. But every element still needed by a lagging branch must remain in memory. The risk of an Out of Memory (OOM) condition rises sharply when elements are large, branches fall far behind, or a branch stops consuming data for a long time.

tee is intended for controlled, synchronous, or coordinately paced single-process iteration. It also works when the branches can coordinate their progress. It should not be treated as a multithreaded fan-out queue.

The Python documentation explicitly states that using iterators returned by the same tee instance concurrently from multiple threads may raise RuntimeError. The pointers and buffer management inside tee are not synchronized for concurrent access. When consumers need to run concurrently, use a mechanism with synchronization semantics, such as queue.Queue.

Decision rules:

Counterexample: Don’t use tee to duplicate an HTTP streaming response into “real-time broadcast” and “offline archiving” paths. Network backpressure can make the broadcast branch’s throughput unpredictable. If either branch falls behind, the buffer may grow continuously and eventually cause an Out of Memory (OOM) condition.

tee splits a single-pass input into multiple independent iterators through a shared linked-list buffer. Memory usage depends on the consumer lag between the fastest and slowest branches.

Core strengths

Distributes non-rewindable streaming data to multiple consumers within a single process. You don't need to pull the upstream input repeatedly. You also don't need to write the entire stream to disk in advance.

Limits and trade-offs

When to use it

Use tee for non-rewindable stream processing with a small number of branches, typically two or three. Keep the work within a single process. The branches should consume data at roughly the same rate, such as when validation and billing run in sync.

How to avoid common pitfalls