Wrap a Fixed-Chunk Stream Reader with Arbitrary-Length Reads
Hudson River TradingSoftware Engineer · Technical ScreenLast updated
Implement an arbitrary-length stream reader on top of an underlying reader that only yields data in fixed-size chunks, managing internal buffering and end-of-stream state across successive calls.
Problem Statement
You are working with a low-level I/O library where the underlying stream source provides data exclusively in fixed-size chunks of size CHUNK_SIZE. The underlying interface provides a single method:
read_chunk() -> bytes: Reads and returns the next chunk from the source. It returns abytesobject of length exactlyCHUNK_SIZE, except possibly at the end of the stream where it may return fewer thanCHUNK_SIZEbytes. Once the stream is exhausted, subsequent calls toread_chunk()return an emptybytesobject (b"").
Your task is to implement a wrapper class, ArbitraryReader, that allows callers to request an arbitrary number of bytes on demand.
class ArbitraryReader:
def __init__(self, source: ChunkSource):
...
def read(self, n: int) -> bytes:
"""
Reads up to n bytes from the stream.
Returns fewer than n bytes only if the end of stream is reached.
"""
...
Input & Output Format
nis a non-negative integer indicating the maximum number of bytes requested.read(n)returns abytesobject containing up tonbytes. Ifn == 0or the stream has no remaining bytes, returnb"".
Constraints
CHUNK_SIZE >= 1(known at runtime, accessible assource.chunk_sizeor fixed per source).0 <= n <= 10^7for any individualread(n)call.- Total stream length can be large; you must not load the entire stream into memory at once.
- Buffer operations should minimize unnecessary memory copies.
Examples
Example 1
Underlying stream content: b"ABCDEFGHIJ"
CHUNK_SIZE: 4
Calls:
source.read_chunk() would return b"ABCD", then b"EFGH", then b"IJ", then b""
reader = ArbitraryReader(source)
reader.read(3) -> b"ABC" # 1 byte (b"D") remains buffered internally
reader.read(5) -> b"DEFGH" # uses 1 byte from buffer + fetches chunk b"EFGH"
reader.read(10) -> b"IJ" # fetches b"IJ", hits EOF, returns remaining 2 bytes
reader.read(5) -> b"" # stream is exhausted
Example 2
Underlying stream content: b"HELLO"
CHUNK_SIZE: 16
Calls:
reader = ArbitraryReader(source)
reader.read(0) -> b""
reader.read(2) -> b"HE"
reader.read(3) -> b"LLO"
reader.read(1) -> b""
Edge Cases
- Calling
read(0)should returnb""without discarding buffered data. - Requesting
nlarger than multiple chunks should fetch as many chunks as needed untilnbytes are accumulated or EOF is encountered. - Handling empty source streams where the first
read_chunk()call immediately returnsb"".
Hints
Hint 1
Maintain an internal byte buffer (or deque of slices/bytes) along with an offset to avoid slicing strings or byte arrays with O(k) copy overhead repeatedly.
Hint 2
When servicing a read request, first satisfy as much of n as possible from the internal buffer before invoking the underlying read_chunk.
Hint 3
If the requested n exceeds the current buffer, repeatedly fetch chunks and append or collect them until you have accumulated n bytes or the chunk returned is smaller than CHUNK_SIZE (or empty).
Follow-up questions
- How would you optimize this implementation to achieve zero-copy (or minimize copying) when passing a pre-allocated writable memory buffer (e.g., Python `bytearray` or C `char*`)?
- How should the reader handle a non-blocking underlying source that can return `EAGAIN` / empty data before true EOF?
- If multiple threads need to consume from the same reader concurrently, what synchronization primitives and buffering strategies would minimize lock contention?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.