Paginate an Audit-Log API, Then Compact Entries to a Byte Budget

You're building a log export tool with two stages that feed into each other: pull everything a paginated audit-log API has, then shrink it to fit a size budget before handing it off.

Part 1 — Drain the paginated API

Assume the client looks like this (confirm the shape before coding against it):

def fetch_page(cursor):
    # type: (Optional[str]) -> Tuple[List[dict], Optional[str]]
    # One page of log entries (each a dict carrying at least a "bytes" size),
    # together with whatever cursor value unlocks the page after this one.
    # Pass cursor=None to get the first page. Nothing left to read looks
    # like an entries list with zero items.
    ...

Write fetch_all(fetch_page) -> list[dict] so that it stops exactly when a page comes back empty, and otherwise keeps feeding each response's cursor into the next call, accumulating entries in the order the pages arrived.

Part 2 — Compact to a byte budget

Once every entry is collected, if their total "bytes" exceeds 20_000, they must be compacted down to 10_000 bytes or fewer by keeping the longest contiguous run of entries, in their original order, whose combined "bytes" is at most 10_000. If the total is already 20_000 or less, return the entries unchanged.

def compact_to_budget(entries: list[dict], byte_budget: int = 10_000) -> list[dict]:

Example

entries = [{"bytes": 4000}, {"bytes": 3000}, {"bytes": 9000}, {"bytes": 1000}, {"bytes": 6000}]
# total = 23000 > 20000, so compaction kicks in with byte_budget = 10000
compact_to_budget(entries) -> [{"bytes": 4000}, {"bytes": 3000}]

Checking every length-3 window: indices 0-2 sum to 16000, indices 1-3 sum to 13000, indices 2-4 sum to 16000 — all over budget, so no length-3 run qualifies. Among length-2 windows, indices 0-1 sum to 7000 (fits), 1-2 sums to 12000 (over), 2-3 sums to 10000 (fits, exactly at the cap), 3-4 sums to 7000 (fits). The longest run that fits has length 2, and the earliest such run — indices 0-1 — is the one returned.

Constraints

• Up to 10^5 total entries across all pages; each entry's "bytes" is a non-negative integer.
• The budget check for keeping a run is inclusive: a run summing to exactly 10_000 is acceptable.
• If several runs tie for the longest length, return the earliest one (smallest starting index).
• A single entry whose own "bytes" exceeds the budget can never be part of any valid run; it's simply never included (not an error, not truncated).

Clarifying questions worth raising first

• Does fetch_page track position itself, or does the caller need to thread the returned cursor into the next call — and is an empty list the only possible end-of-data signal, or could a page also come back None?
• Should a failed or timed-out page fetch raise immediately, or does the caller retry with the same cursor?
• Is "longest run" measured in entry count or in total bytes used — this example assumes entry count, with byte total only as the tie-break rule within equal-length runs if that's ever ambiguous.