Performance profiling generators

Performance work in Python: profiling before you optimise, generators for memory-friendly iteration, and the accidental slowness that creeps into otherwise fine code. The rule has not changed since the intro tutorials – make it work, make it clear, then make it fast if the measurements say so.

Profile first

Guessing where time goes wastes effort. cProfile ships with Python and tells you which functions ate the clock.

import cProfile
import pstats
from io import StringIO

def slow_work() -> int:
    total = 0
    for i in range(500_000):
        total += i ** 2
    return total

profiler = cProfile.Profile()
profiler.enable()
result = slow_work()
profiler.disable()

stream = StringIO()
stats = pstats.Stats(profiler, stream=stream).sort_stats("cumulative")
stats.print_stats(10)
print(stream.getvalue())
print("result", result)

For line-by-line detail inside one function, install line_profiler or use your IDE’s profiler. Run profiles on realistic input sizes – a loop that is fine for ten items may hurt at ten million.

timeit for micro-benchmarks

When comparing two small approaches, timeit repeats the snippet and averages the cost.

import timeit

setup = "data = list(range(100_000))"
loop_append = """
result = []
for x in data:
    result.append(x * 2)
"""
comp = "[x * 2 for x in data]"

print("append loop:", timeit.timeit(loop_append, setup=setup, number=50))
print("comprehension:", timeit.timeit(comp, setup=setup, number=50))

Micro-benchmarks lie easily. A comprehension winning by a microsecond rarely matters in a web request. Use timeit to settle style debates, not architecture ones.

Generators and lazy iteration

A generator yields values one at a time instead of building a full list in memory. That matters when you stream files, parse logs, or walk a large dataset.

from pathlib import Path

def lines_starting_with(path: Path, prefix: str):
    with path.open(encoding="utf-8") as handle:
        for line in handle:
            if line.startswith(prefix):
                yield line.rstrip("n")

for line in lines_starting_with(Path("access.log"), "ERROR"):
    print(line)

Generator expressions look like comprehensions with parentheses. They are lazy; list comprehensions build everything upfront.

total = sum(x * x for x in range(1_000_000))  # generator expression
squares = [x * x for x in range(1_000_000)]   # full list in memory

Accidental slowness

• Repeated work in loops – hoist invariant calculations, cache expensive pure functions with functools.lru_cache
• String concatenation in a loop – collect parts in a list and "".join(...) once
• Membership tests on lists – use a set when you check in many times
• Loading entire files when streaming works – read line by line or in chunks
• N+1 network or database calls – batch requests; fetch related data in one round trip

from functools import lru_cache

@lru_cache(maxsize=256)
def normalise_slug(value: str) -> str:
    return value.strip().lower().replace(" ", "-")

Reach for libraries that do heavy lifting in C – NumPy, pandas, orjson – when the data justifies the dependency. Do not rewrite pandas in pure Python because you read a blog post about speed.

When to stop optimising

If the whole operation finishes in under a hundred milliseconds and runs rarely, your time is better spent elsewhere. Ship readable code, add metrics in production, and revisit hotspots that actually hurt users or your bill.