Extend and optimize an existing CLI tool
OracleSoftware Engineer · OnsiteLast updated
This problem tests your ability to refactor legacy CLI code, implement modular features, and optimize performance for large-scale data processing.
The Task
You are maintaining a CLI tool named log-analyzer written in Python. Currently, the tool reads a local file line-by-line, filters by a single status code, and prints the count of matching entries.
Requirements
- Refactor the existing code to support multiple status code filtering.
- Add a new feature to optionally output results to a JSON file instead of stdout.
- Optimize the tool to handle log files exceeding 10GB without loading them entirely into memory.
- Ensure the solution remains maintainable and unit-testable.
Current Codebase Excerpt
def process_logs(file_path, target_code):
count = 0
with open(file_path, 'r') as f:
for line in f:
# Assuming format: "timestamp status_code message"
if str(target_code) in line.split()[1]:
count += 1
print(f"Total: {count}")
Your Deliverable
Provide a revised version of the process_logs function (or a class-based structure) that implements these requirements. Assume the file format is consistent. Use the following signature:
def process_logs(file_path, target_codes: list[int], output_path: str = None):
# Implementation here
pass
Constraints
- Time complexity: O(N) where N is the number of lines.
- Space complexity: O(K) where K is the number of target codes.
- Edge cases: Handle missing files, empty lines, and malformed lines gracefully.
Hints
Hint 1
Consider using a 'set' for O(1) lookup of target status codes.
Hint 2
Use Python's generators or file object iteration to process the file line-by-line, ensuring memory usage stays constant regardless of file size.
Hint 3
Implement a strategy pattern or a simple conditional check to handle different output streams (stdout vs. file) without duplicating the processing logic.
Follow-up questions
- How would you add unit tests to verify the logic without reading actual 10GB files?
- If the log file contains duplicate entries, how would you modify the current logic to count only unique entries?
- How would you transition this script to support multi-threading or parallel processing for faster execution on multi-core systems?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.