Compare Python Generators, Decorators, and Context Managers

Overview

In production reliability engineering, processing large log files or stream dumps without exhausting memory or leaving file descriptors dangling is critical. Python provides three distinct abstractions that control execution flow and object lifecycles:

1. Generators: Protocol defined by __iter__() and __next__() (and lifecycle methods close() / throw()), allowing lazy evaluation and cooperative multitasking.
2. Context Managers: Protocol defined by __enter__() and __exit__(), guaranteeing setup and deterministic teardown around a block of code.
3. Decorators: Higher-order callables wrapping functions or classes to extend or monitor behavior without altering core logic.

The Task

You are tasked with building a robust, audited lazy file reader that combines all three concepts.

Implement the following components:

1. track_stream(metrics: dict): A decorator for generator functions. It must track the streaming lifecycle without converting the generator into a list:

   • Increment metrics['active_streams'] when the generator is instantiated.
   • Decrement metrics['active_streams'] when the generator terminates (completed iteration, consumer early exit/break, or exception).
   • Increment metrics['total_yields'] for every item yielded.
   • If the consumer or generator encounters an unhandled exception (excluding GeneratorExit), increment metrics['exceptions'].

2. stream_chunks(file_path: str, chunk_size: int = 1024): A generator function decorated with @track_stream. It opens the specified file and yields chunks of up to chunk_size characters at a time.

   • It must guarantee that the file descriptor is closed immediately upon termination—including when the caller breaks early, encounters an error, or the generator is garbage-collected.

Signatures

from typing import Callable, Generator, Any

def track_stream(metrics: dict) -> Callable:
    ...

def stream_chunks(file_path: str, chunk_size: int = 1024) -> Generator[str, None, None]:
    ...

Example

metrics = {'active_streams': 0, 'total_yields': 0, 'exceptions': 0}

# Assuming 'access.log' has 2500 characters:
g = stream_chunks('access.log', chunk_size=1000)
assert metrics['active_streams'] == 1

first_chunk = next(g)
assert metrics['total_yields'] == 1
assert len(first_chunk) == 1000

# Early exit
g.close()
assert metrics['active_streams'] == 0

Constraints & Edge Cases

• The file must not be read entirely into memory at once.
• Handle consumer-side break in a for loop cleanly (GeneratorExit should not be counted as an exception in metrics['exceptions']).
• File handles must be closed deterministically even if an error is raised mid-consumption.
• Multiple consumers should be able to run concurrently with accurate metric counts.