PYTHON / ITERATORS, GENERATORS, AND COMPREHENSIONS
Essential itertools recipes
Compose islice, chain, groupby, accumulate and tee into standard recipes for chunking, sliding windows, dedup and grouping without building lists.
What you will learn
- Chunk any iterable by calling islice repeatedly on one shared iter() cursor
- Sort by the grouping key before groupby, and consume each group before advancing
- Compute running totals with accumulate and cut them short with takewhile
- Flatten nested data with chain.from_iterable and dedup with a seen-set recipe
Understanding Essential itertools recipes
The functions in itertools are iterator adapters: each one wraps a source iterator and pulls items from it on demand. What makes them combine so well is that they all agree on a single piece of shared state, the source's current position. If you do `it = iter(data)` once and then call `islice(it, 3)` twice, the second call continues where the first stopped, because islice does not restart anything, it just advances the same cursor. Almost every itertools recipe is built on that fact.
Chunking is the clearest demonstration. `tuple(islice(it, n))` grabs the next n items and returns a short tuple when the source runs dry, so a loop that keeps slicing until it gets an empty tuple splits any iterable into fixed-size batches, including a partial last batch, without knowing the length up front. A sliding window is the mirror image: prime a tuple with the first n items, then for every further item drop the oldest and append the newest. Both work on files, sockets and infinite generators, not just lists.
groupby is the tool people misjudge most often, because it groups only runs of adjacent equal keys, exactly like the Unix `uniq -c`. It does not build a dictionary, so if the same key reappears later in the input you get that key twice. The group it hands you is also a lazy view over the same source position, which means it becomes empty as soon as you ask groupby for the next group. Sort by the key first if you want one group per key, and consume or copy each group before moving on.
from itertools import islice
def chunks(iterable, n):
it = iter(iterable)
while batch := tuple(islice(it, n)):
yield batch
def sliding(iterable, n):
it = iter(iterable)
window = tuple(islice(it, n))
if len(window) == n:
yield window
for item in it:
window = window[1:] + (item,)
yield window
readings = [3, 5, 4, 9, 12, 11, 2]
print("chunks:", list(chunks(readings, 3)))
print("3-point averages:", [round(sum(w) / 3, 2) for w in sliding(readings, 3)])itertools adapters share one cursor into the source iterator, so recipes work by advancing that cursor in different patterns instead of copying data.
Worked examples
groupby needs sorted input
Totalling amounts per region, showing why the data must be sorted by the grouping key first.
from itertools import groupby
rows = [("emea", 120), ("apac", 80), ("emea", 45), ("us", 200), ("apac", 10)]
by_region = lambda r: r[0]
print("unsorted:", [k for k, _ in groupby(rows, key=by_region)])
rows.sort(key=by_region)
for region, group in groupby(rows, key=by_region):
print(region, sum(amount for _, amount in group))Example explained
Line 1The first print shows groupby emitting 'emea' twice: it only breaks runs of adjacent equal keys.
Line 2rows.sort with the same key function brings equal keys together, so each key now appears once.
Line 3sum(...) consumes the group iterator immediately, which is required before the loop asks for the next group.
Line 4Reuse one key function for both sort and groupby; a mismatch silently produces fragmented groups.
Running totals with a cutoff
accumulate produces a running sum lazily, and takewhile stops the scan at the first value over budget.
from itertools import accumulate, takewhile
costs = [4.5, 3.0, 8.25, 6.0, 5.5]
print(list(accumulate(costs)))
within_budget = list(takewhile(lambda total: total <= 16, accumulate(costs)))
print(len(within_budget), "items, spend", within_budget[-1])Example explained
Line 1accumulate yields one partial sum per input item, so its length always matches the source.
Line 2takewhile stops at the first False and never pulls further items, so accumulate never adds 6.0.
Line 3Use takewhile, not a filter: a filter would skip 21.75 but keep testing later values.
Line 4accumulate takes a second argument for other operations, e.g. accumulate(costs, max) for a running maximum.
Flatten then deduplicate
chain.from_iterable turns nested lists into one stream, and a seen-set recipe keeps the first occurrence of each item.
from itertools import chain
def unique_everseen(iterable, key=None):
seen = set()
for item in iterable:
k = item if key is None else key(item)
if k not in seen:
seen.add(k)
yield item
pages = [["a.py", "b.py"], ["b.py", "c.py"], ["a.py", "d.py"]]
flat = chain.from_iterable(pages)
print(list(unique_everseen(flat)))
print(list(unique_everseen(["README", "readme", "Setup"], key=str.lower)))Example explained
Line 1chain.from_iterable takes one iterable of iterables, so it also works on a generator of file handles.
Line 2chain(*pages) would need the outer sequence in memory first; from_iterable pulls sublists one at a time.
Line 3The seen set preserves first-seen order, which sorted(set(...)) would destroy.
Line 4The key argument lets you dedupe on a normalised form while yielding the original item.
Important notes
tee stores every item one branch has read but the other has not; draining one branch completely buffers the whole sequence in memory, and you must never touch the original iterator after tee-ing it.
cycle keeps an internal copy of everything it has seen so it can repeat, so cycling a huge or infinite source is a memory leak in disguise.
Common mistakes
Calling groupby on unsorted data and treating the result as a dictionary of unique keys: keys that reappear later produce extra groups, so per-key totals are silently split.
Doing groups = list(groupby(data)) and reading the groups afterwards: each group is a view on the shared cursor, so all but the last one come back empty.
Passing a list instead of iter(list) to a chunking loop that calls islice repeatedly: islice(a_list, n) starts from index 0 every time, so the loop yields the same first chunk forever.
Try it yourself
Change, predict, then run
Use itertools.groupby to turn the string 'aaabbbbcaa' into the run-length list [('a', 3), ('b', 4), ('c', 1), ('a', 2)]; count each run with sum(1 for _ in group), since a group has no len().
Open the Python workspaceCheck your understanding
You write groups = list(groupby(sorted(data))) and then loop over groups printing list(g) for each. Every list comes out empty except possibly the last. Why?
- Each group is a lazy view over the single shared source position, so building the outer list advanced past every group's items before you read them
- list() copies the keys of the pairs but replaces the group objects with empty placeholders
- sorted() returns an iterator that groupby exhausts, so nothing is left for the groups
- groupby requires its input to be a list, and sorted() defeats the internal buffering it uses to keep groups readable
Show answer
groupby hands out sub-iterators that read from the same underlying cursor; asking for the next (key, group) pair skips any unread items of the previous group, and list(groupby(...)) does exactly that for the entire input before you read anything. The 'list() replaces the groups' option is tempting because the symptom looks like a copying bug, but list() copies the tuples faithfully, the group objects it copies have simply been invalidated. Fix it with [(k, list(g)) for k, g in groupby(sorted(data))], which materialises each group before advancing.