Katkat
Back

LLM Fundamentals: Tokenization Design and KL-Regularized SFT

AmazonApplied Scientist · Onsite
MediumApplied ScientistOnsiteMachine Learning

Last updated

This problem tests your understanding of subword tokenization strategies and the theoretical motivation behind KL-divergence constraints during supervised fine-tuning.

Part 1: Tokenization

Modern LLMs often utilize Byte Pair Encoding (BPE) or WordPiece. Suppose you are designing a tokenizer for a multilingual model that needs to support both standard English text and high-density source code (e.g., C++, Python).

  1. Explain the primary disadvantage of character-level tokenization compared to subword tokenization in this context.
  2. How does the choice of vocabulary size impact the model's inference latency versus its ability to represent rare tokens (like specific function names or hexadecimal values)?

Part 2: KL-Regularized SFT

When performing Supervised Fine-Tuning (SFT) on a pre-trained base model, we sometimes optimize the objective: Loss = L_SFT(pi_theta) + beta * KL(pi_theta || pi_ref) where pi_theta is the fine-tuned model and pi_ref is the original pre-trained model.

  1. What specific problem does the KL-divergence term address during the fine-tuning process?
  2. What happens to the model's behavior as the hyperparameter beta approaches infinity, and what happens as it approaches zero?

Hints

Hint 1

Consider the sequence length limitations imposed by Transformer-based architectures when evaluating character-level versus subword approaches.

Hint 2

Think about the 'catastrophic forgetting' phenomenon in pre-trained models when updating weights on a small, specific instruction-tuning dataset.

Hint 3

Analyze the KL term as a penalty for deviating from the probability distribution learned during the expensive pre-training phase.

Follow-up questions

  • How does the choice of a specific subword algorithm (e.g., Unigram vs. BPE) affect the model's robustness to noisy input or typos?
  • If we observe that a model becomes overly 'safe' or repetitive after SFT, how might we adjust the KL penalty or the training data distribution to mitigate this?
  • How would you measure the trade-off between instruction-following performance and general-purpose reasoning capabilities when tuning the `beta` parameter?