Master the iteration protocol and generators to process data streams lazily, at constant memory.
Open this lesson in KodokonPython's for is nothing more than a protocol: iter(obj) asks for an iterator, then next() is called until the StopIteration exception. Be sure to distinguish the iterable (the list, able to hand out a fresh iterator on each pass) from the iterator (the cursor, which can only be traversed once). Understanding this protocol means understanding why some objects "get consumed."
numbers = [10, 20, 30]
iterator = iter(numbers)
print(next(iterator))
print(next(iterator))
print(next(iterator))
# next(iterator) would raise StopIterationA function that contains yield does not run when called: it returns a generator, an iterator whose body runs on demand. Each next() advances to the next yield, where execution is suspended along with all of its local state. The direct consequence: memory stays constant whatever the size of the stream - you produce the values one by one instead of materializing a full list.
def squares(limit):
n = 0
while n < limit:
yield n * n
n += 1
gen = squares(4)
print(next(gen)) # 0
print(list(gen)) # [1, 4, 9]The main benefit in production: pipelines. Each stage consumes the stream from the previous one and produces its own, lazily; nothing runs before the final consumption (list(), sum(), a for). A trade-off to keep in mind: a list can be traversed several times and sliced, a generator cannot; on the other hand, a generator processes a 10 GB file without saturating RAM. Generator expressions - (f(x) for x in xs if cond) - cover the simple stages without defining a function.
lines = [" 12 ", "x", " 7", "", "30"]
def clean(rows):
for row in rows:
row = row.strip()
if row.isdigit():
yield int(row)
def keep_from(values, floor):
return (v for v in values if v >= floor)
pipeline = keep_from(clean(lines), 10)
print(list(pipeline)) # [12, 30]