Performance: profiling, GIL, generators vs lists, memory pitfalls

Measure before you optimize: timeit and cProfile, what the GIL really blocks, the frugality of generators and the memory pitfalls of strings and slices.

11 min · 3 questions

Never guess, measure. timeit isolates a reliable micro-benchmark: it disables the garbage collector and repeats execution to smooth out the noise. cProfile maps out the calls of a whole program - sort by cumtime (cumulative time, sub-calls included) to identify the real culprits. The golden rule of any performance work: the slow line is almost never the one you suspect.

python -m timeit "sum(x * x for x in range(1000))"
python -m timeit -s "s = 'abc' * 100" "s.upper()"
python -m cProfile -s cumtime -m timeit "min(range(50))"

The GIL (Global Interpreter Lock) guarantees that only one thread executes Python bytecode at a time within a process. Consequence: threads do not speed up pure computation. They remain, however, perfect for I/O, because the GIL is released during network or disk reads, as well as by many C libraries such as NumPy. To saturate several cores with computation, go through multiprocessing or concurrent.futures.ProcessPoolExecutor. Python 3.13 offers an experimental "free-threaded" variant without a GIL, still rare in production.

import sys

squares_list = [x * x for x in range(1_000_000)]
squares_gen = (x * x for x in range(1_000_000))

print(sys.getsizeof(squares_list), "bytes")
print(sys.getsizeof(squares_gen), "bytes")
print(sum(squares_gen))

A generator weighs a few hundred bytes regardless of the length of the sequence: it stores only its execution state and produces each value on demand. Chain them into pipelines - sum(x * x for x in data) - to process streams without ever materializing an intermediate list. A trade-off to be aware of: a generator is single-use. Once exhausted, it produces nothing more, silently - a classic source of bugs where the second iteration seems "empty".

import timeit

def concat(n: int) -> str:
    out = ""
    for i in range(n):
        out += str(i)
    return out

def join(n: int) -> str:
    return "".join(str(i) for i in range(n))

t1 = timeit.timeit(lambda: concat(20_000), number=10)
t2 = timeit.timeit(lambda: join(20_000), number=10)
print(f"concat: {t1:.3f} s / join: {t2:.3f} s")

Knowledge check

Make sure you remember the key points of this lesson.

1. What exactly does the GIL prevent?
   • Any form of parallelism in Python, including multiprocessing
   • The simultaneous execution of Python bytecode by several threads of the same process - I/O and much C code release it
   • The use of several threads in a single program
   • Concurrent writes to the same variable
2. Why does sys.getsizeof of a generator stay tiny compared to the equivalent list?
   • The generator stores no element: it keeps only its execution state and computes each value on demand
   • getsizeof cannot measure generators
   • Generators compress their data internally
3. Why does ''.join(parts) beat out += part repeated in a loop?
   • join uses several threads to assemble the string
   • It's a myth: performance has been identical since Python 3
   • Since str are immutable, each += recopies the whole string (quadratic cost); join computes the final size and allocates only once
   • join avoids creating the intermediate str objects of the generator