Transfer Learning
Transfer learning is a technique where a model trained on one task is adapted to improve performance on a different but related task. In practice, it often builds on supervised or self-supervised pretraining.
How It Works
Section titled “How It Works”- Train (or use) a model on a source task with abundant data
- Transfer the learned features/weights to a new target task
- Fine-tune on the target task’s data (usually smaller)
Types of Transfer Learning
Section titled “Types of Transfer Learning”| Type | Description | Example |
|---|---|---|
| Feature Extraction | Use a pre-trained model as a fixed feature extractor | Use ResNet features for a custom image classifier |
| Fine-Tuning | Retrain some or all layers on the new task | Fine-tune BERT for sentiment analysis |
| Domain Adaptation | Adapt a model to a related domain | ImageNet model → medical imaging |
Why Transfer Learning Matters
Section titled “Why Transfer Learning Matters”| Benefit | Description |
|---|---|
| Less data required | Leverage pre-trained knowledge; need fewer labeled examples |
| Faster training | Start from learned weights, not random initialization |
| Better performance | Pre-trained features often generalize well |
Real-World Examples
Section titled “Real-World Examples”- NLP: Fine-tune GPT or BERT for summarization, Q&A, or classification
- Computer Vision: Use ImageNet-trained models for medical imaging or satellite imagery
- Speech: Adapt a general ASR model to a specific accent or domain vocabulary
When It Can Go Wrong
Section titled “When It Can Go Wrong”If the source and target tasks are too different, transfer can hurt performance (called negative transfer). Always validate on the target task.
Exam Tip: When you want to leverage knowledge from a pre-trained model to improve performance on a new, related task with less data and compute, transfer learning is the most suitable approach.