Katkat

Wrap a Fixed-Chunk Stream Reader with Arbitrary-Length Reads

Hudson River TradingSoftware Engineer · Technical Screen

Last updated

Implement an arbitrary-length stream reader on top of an underlying reader that only yields data in fixed-size chunks, managing internal buffering and end-of-stream state across successive calls.

Problem Statement

You are working with a low-level I/O library where the underlying stream source provides data exclusively in fixed-size chunks of size CHUNK_SIZE. The underlying interface provides a single method:

  • read_chunk() -> bytes: Reads and returns the next chunk from the source. It returns a bytes object of length exactly CHUNK_SIZE, except possibly at the end of the stream where it may return fewer than CHUNK_SIZE bytes. Once the stream is exhausted, subsequent calls to read_chunk() return an empty bytes object (b"").

Your task is to implement a wrapper class, ArbitraryReader, that allows callers to request an arbitrary number of bytes on demand.

class ArbitraryReader:
    def __init__(self, source: ChunkSource):
        ...

    def read(self, n: int) -> bytes:
        """
        Reads up to n bytes from the stream.
        Returns fewer than n bytes only if the end of stream is reached.
        """
        ...

Input & Output Format

  • n is a non-negative integer indicating the maximum number of bytes requested.
  • read(n) returns a bytes object containing up to n bytes. If n == 0 or the stream has no remaining bytes, return b"".

Constraints

  • CHUNK_SIZE >= 1 (known at runtime, accessible as source.chunk_size or fixed per source).
  • 0 <= n <= 10^7 for any individual read(n) call.
  • Total stream length can be large; you must not load the entire stream into memory at once.
  • Buffer operations should minimize unnecessary memory copies.

Examples

Example 1

Underlying stream content: b"ABCDEFGHIJ"
CHUNK_SIZE: 4
Calls:
  source.read_chunk() would return b"ABCD", then b"EFGH", then b"IJ", then b""

reader = ArbitraryReader(source)
reader.read(3)  -> b"ABC"    # 1 byte (b"D") remains buffered internally
reader.read(5)  -> b"DEFGH"  # uses 1 byte from buffer + fetches chunk b"EFGH"
reader.read(10) -> b"IJ"     # fetches b"IJ", hits EOF, returns remaining 2 bytes
reader.read(5)  -> b""       # stream is exhausted

Example 2

Underlying stream content: b"HELLO"
CHUNK_SIZE: 16
Calls:
  reader = ArbitraryReader(source)
  reader.read(0) -> b""
  reader.read(2) -> b"HE"
  reader.read(3) -> b"LLO"
  reader.read(1) -> b""

Edge Cases

  • Calling read(0) should return b"" without discarding buffered data.
  • Requesting n larger than multiple chunks should fetch as many chunks as needed until n bytes are accumulated or EOF is encountered.
  • Handling empty source streams where the first read_chunk() call immediately returns b"".

Hints

Hint 1

Maintain an internal byte buffer (or deque of slices/bytes) along with an offset to avoid slicing strings or byte arrays with O(k) copy overhead repeatedly.

Hint 2

When servicing a read request, first satisfy as much of n as possible from the internal buffer before invoking the underlying read_chunk.

Hint 3

If the requested n exceeds the current buffer, repeatedly fetch chunks and append or collect them until you have accumulated n bytes or the chunk returned is smaller than CHUNK_SIZE (or empty).

Follow-up questions

  • How would you optimize this implementation to achieve zero-copy (or minimize copying) when passing a pre-allocated writable memory buffer (e.g., Python `bytearray` or C `char*`)?
  • How should the reader handle a non-blocking underlying source that can return `EAGAIN` / empty data before true EOF?
  • If multiple threads need to consume from the same reader concurrently, what synchronization primitives and buffering strategies would minimize lock contention?