Python: Iterators and Generators

Iterable vs Iterator

These two terms are related but distinct, and conflating them leads to real bugs. An iterable is any object that can produce an iterator when you call iter() on it. Lists, tuples, strings, dictionaries, and sets are all iterables. An iterator is the object that actually does the work of stepping through values one at a time. It must have a __next__ method.

A key practical difference: you can iterate over a list as many times as you like because each call to iter(list) returns a fresh iterator. An iterator, by contrast, is stateful and one-directional. Once it is exhausted, it stays exhausted. Calling iter() on an already-exhausted iterator just returns the same spent object.

__iter__ and __next__ Protocol

When you write a for loop, Python executes a specific sequence of steps behind the scenes. It calls iter(obj) to get an iterator, then callsnext() on that iterator repeatedly until StopIteration is raised, at which point the loop stops cleanly. You never see the exception because thefor loop catches it internally.

Understanding this sequence is what lets you build custom objects that plug directly into for loops, comprehensions, sum(), max(), and any other Python construct that consumes sequences.

Built-in Iterators

Python ships with several built-in functions that return iterators directly rather than materializing a full list. This is the memory-efficient choice for large datasets: range(), enumerate(), zip(), map(),filter(), and reversed() all return lazy iterator objects. In Python 3, dict.keys(), dict.values(), and dict.items()also return view objects that are iterable.

Creating Custom Iterators

You build a custom iterator by defining a class with both __iter__ and__next__. The class manages its own internal state. __iter__ typically returns self, and __next__ advances the state and returns the next value. When the sequence is done, it raises StopIteration.

A practical pattern is to separate the iterable (the data holder) from the iterator (the cursor). This lets you create multiple independent iterators over the same data, just like a list allows.

The StopIteration Exception

StopIteration is the signal that an iterator has run out of values. It is not an error condition in the traditional sense; it is a protocol. The forloop, comprehensions, list(), tuple(), and similar constructs all catch it silently to know when to stop.

One important rule: as of Python 3.7, raising StopIteration inside a generator function is converted into a RuntimeError. This prevents subtle bugs where a StopIteration from an inner call accidentally terminates an outer generator. Use return to signal the end of a generator instead.

Generator Functions and the yield Keyword

A generator function looks like a normal function but contains at least oneyield statement. Calling it does not execute any of its code; it returns a generator object immediately. Execution begins only when you call next() on that object. The function runs until it hits a yield, hands the yielded value back to the caller, and then suspends itself, preserving all its local state. The next call to next() resumes from exactly that point.

This makes generators the natural, Pythonic way to implement iterators. The class-based approach from the previous section requires you to manage state manually. Generators handle that bookkeeping automatically.

Generator Expressions

A generator expression is the concise, inline equivalent of a generator function. The syntax is identical to a list comprehension but uses parentheses instead of square brackets. The result is a generator object, not a list, so values are produced lazily on demand.

Use a generator expression when you only need to iterate once and do not need to store all the results. Use a list comprehension when you need random access, need to iterate multiple times, or need to check the length. For large datasets, the memory savings of a generator expression can be substantial.

The yield from Statement

yield from iterable delegates to another iterable or generator. It is more than a shorthand for for x in sub: yield x. When delegating to another generator, yield from also passes send() and throw() calls directly through to the sub-generator, and captures the sub-generator's finalreturn value as its own expression result.

The most common use is flattening nested iterables or composing multiple generators into a pipeline without writing manual loops.

Infinite Generators

Because generators are lazy, they can produce values indefinitely without ever running out of memory. An infinite generator simply never raises StopIteration. The caller is responsible for pulling exactly as many values as it needs, typically using next() directly, a for loop with a break, oritertools.islice() to take a fixed number of values.

send() and throw() in Generators

Generators are not just one-way pipes. You can communicate back into a running generator using gen.send(value). The sent value becomes the result of theyield expression inside the generator. This is how coroutine-style programming worked in Python before async/await was introduced, and it is still useful for stateful data pipelines.

Important: the very first call must be next(gen) or gen.send(None)to advance the generator to its first yield. Only after it is paused at a yield can you send a non-None value.gen.throw(ExcType) injects an exception at the current yield point, allowing the generator to catch and handle it internally.

close() on Generators

Calling gen.close() throws a GeneratorExit exception into the generator at the current yield point. The generator can catch this with atry/finally block to run cleanup code. If the generator does not catch it, Python silently ignores the exception and the generator is finalized. Once closed, calling next() on it raises StopIteration.

This is particularly important when you have a generator managing an open resource. Wrapping the yield in try/finally guarantees the resource is released even if the caller stops consuming values early.

The itertools Module

The standard library's itertools module contains a set of fast, memory-efficient iterator building blocks. They are all implemented in C, so they are typically faster than hand-written Python loops. The module is organized around three conceptual groups: infinite iterators, terminating iterators, and combinatoric iterators.

count(), cycle(), and repeat()

These three are the infinite iterators. count(start, step) produces an endless arithmetic sequence. cycle(iterable) loops over the input forever.repeat(obj, times) yields the same object repeatedly; if you omit times, it runs forever.

chain(), islice(), and tee()

chain(*iterables) concatenates multiple iterables into a single sequence without building a new list. islice(iterable, stop) takes a slice of a lazy iterator, which is the standard way to limit infinite generators.tee(iterable, n) clones an iterator into n independent copies. Use tee carefully: once you have teed an iterator, you should not use the original any further.

product(), permutations(), and combinations()

These are the combinatoric iterators. product() computes the Cartesian product (all ordered pairs from two or more sets), equivalent to nested loops.permutations() yields all ordered arrangements of the input.combinations() yields all subsets of a given length without repetition and without considering order. There is also combinations_with_replacement()for the case where repetition is allowed.

groupby() and starmap()

groupby(iterable, key) groups consecutive elements that share the same key. The critical requirement: the input must be sorted by that key ahead of time, otherwise items with the same key but separated by different keys will appear in multiple groups. starmap(func, iterable) is like map() but unpacks each element of the iterable as arguments to the function, which is convenient when your data is already stored as tuples of arguments.

Putting It All Together: A Data Pipeline

One of the most practical applications of generators is building processing pipelines where each stage transforms data and passes it to the next, all lazily. The example below reads numbers from a source, filters out negatives, squares the rest, and reports a running total, all without building any intermediate lists.

Quiz - Test Your Knowledge

Ten questions covering the iterator protocol, generator mechanics, and the itertools module. A few questions have deliberately close options, so read each one carefully before answering.