Write Pseudocode for Embedding Creation and Vector Search

Problem Statement

You are building a service to support a product search feature. You need to implement two core components: a batch ingestion pipeline that generates embeddings for documents and a retrieval function that searches those embeddings with metadata filters.

1. Ingestion Pipeline

Implement a function process_and_index_documents(documents) where each document is a dictionary: {'id': str, 'text': str, 'category': str}.

• Constraints: Assume an embedding model embed_model.encode(texts) which accepts a list of strings and returns a list of floats.
• Requirements:
  • Chunk documents if they exceed 512 tokens (assume a split_text helper exists).
  • Perform vector normalization (L2 norm) for cosine similarity readiness.
  • Ensure the upsert process is idempotent using the id field.
  • Implement basic error handling for failed model inferences.

2. Retrieval Service

Implement search_vector_store(query, category_filter, top_k=5).

• Task: Given a query string, retrieve the top_k relevant documents where the category matches the provided filter.
• Logic: Use the same embedding model for the query. Assume a vector_db interface with query(vector, filter, k).

Example

# Input
docs = [{'id': 'p1', 'text': 'running shoe', 'category': 'footwear'}]
# Vector DB should store: {'id': 'p1', 'vector': [0.1, ...], 'category': 'footwear'}

# Search
results = search_vector_store('athletic gear', 'footwear', top_k=1)
# Expectation: Return document matching the filter with highest cosine similarity.

Edge Cases

• Empty document lists.
• Model timeout or API failure.
• Filter returning zero results.