Katkat
Back

Implement web data fetch and storage tool

CoreweaveSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteCoding & Algorithms

Last updated

This question evaluates proficiency in HTTP client communication with token-based authentication, parsing structured responses into data models, relational database interaction including handling duplicate records, configuration management, and robust error handling and testing.

Problem

You are asked to implement a small program that retrieves data from a remote web service protected by token-based authentication, parses the response, and stores the parsed data into a database.

Requirements

  1. Remote request with token-based authentication
  • The program should send an HTTP(S) request to a given URL (e.g., provided via a config file or command-line argument).
  • The remote service uses token-based authentication (e.g., a bearer token).
  • The program should attach the given token to the request in the appropriate HTTP header (for example, Authorization: Bearer <token>).
  1. Parse response content
  • Assume the remote service returns a structured response (e.g., JSON) containing a list of items.
  • Each item has a few fields such as an id, a timestamp, and a message (you can assume reasonable names and types if not specified exactly).
  • The program should parse the response body and extract these fields into in-memory objects/records.
  1. Store into a database
  • The program should connect to a relational database (e.g., PostgreSQL or MySQL) using a connection string or config.
  • Create (or assume the existence of) a table with appropriate columns to store the parsed fields (e.g., id, timestamp, message).
  • Insert the parsed records into the table.
  • Handle duplicates in a sensible way (for example, avoid inserting the same id more than once, or perform an upsert).
  1. Error handling and robustness
  • Handle common errors such as:
  • Network failures or timeouts when calling the remote service.
  • Non-2xx HTTP status codes.
  • Invalid or unexpected response formats.
  • Database connection or insertion errors.
  • Log errors or print meaningful messages so that a user or operator can understand what went wrong.
  1. Execution interface
  • The program can be a command-line tool.
  • It should accept at least:
  • The remote service URL.
  • The authentication token.
  • Database connection information.

Describe how you would design and implement this program, including:

  • How you would structure the code (e.g., separation between HTTP client, parser, and database layer).
  • How you would manage configuration (URL, token, DB credentials).
  • How you would test it (unit tests, integration tests, mocking the remote service and DB).