BERT
Definition
BERT, introduced by Google in 2018, is a transformer encoder pre-trained with two self-supervised objectives: Masked Language Modeling (MLM, predicting randomly masked tokens) and Next Sentence Prediction (NSP, classifying whether two sentences follow each other). Training on Wikipedia and BooksCorpus with MLM forces BERT to develop deep bidirectional contextual representations—the representation of each word depends on all surrounding words, not just preceding ones. Fine-tuning BERT on small labeled datasets achieved state-of-the-art results on 11 NLP benchmarks simultaneously, establishing the pre-train-then-fine-tune paradigm that dominates NLP.
Why It Matters
BERT's release democratized high-performance NLP by providing a pre-trained foundation that non-researchers could fine-tune for specific tasks with modest labeled datasets. Before BERT, achieving strong NLP performance required large task-specific datasets and extensive feature engineering. BERT demonstrated that transfer learning from self-supervised pre-training works powerfully for language, paralleling what ImageNet pre-training did for computer vision. Understanding BERT is fundamental to understanding all modern NLP, as virtually every high-performance NLP system uses a BERT-family model or one of its successors.
How It Works
BERT uses a transformer encoder stack (BERT-base has 12 layers, 768 hidden dimensions, 12 attention heads; BERT-large has 24 layers, 1024 dimensions, 16 heads). Tokenization uses WordPiece subword segmentation with a 30,000 token vocabulary. Input sequences include [CLS] (classification) and [SEP] (separator) special tokens. For classification tasks, the [CLS] token representation is fine-tuned with a linear classification head. For sequence labeling (NER), all token representations are used. Fine-tuning updates all model weights on task-specific data, typically for 3-5 epochs on small datasets.
BERT — Bidirectional Transformer Pre-Training
Input tokens (with masking)
Bidirectional self-attention
→ "reads"
Attends to tokens on both sides simultaneously — unlike GPT which is left-to-right only
Pre-training objectives
Masked Language Model (MLM)
Predict masked tokens using left + right context
Next Sentence Prediction (NSP)
Predict whether sentence B follows sentence A
Output: Contextual embeddings
Each token gets a 768-dim (BERT-base) or 1024-dim (BERT-large) vector encoding its meaning in context. Fine-tuned for classification, NER, QA, and more.
Real-World Example
A legal tech company fine-tunes BERT-base on 3,000 labeled contract clauses to classify 18 clause types (limitation of liability, termination, confidentiality, etc.). The fine-tuned BERT model achieves 94% classification accuracy, dramatically outperforming the previous TF-IDF + logistic regression baseline at 76%. Contract review time decreases from 4 hours to 20 minutes per document as BERT automatically locates and labels all clause types. The model processes 100-page contracts in under 30 seconds.
Common Mistakes
- ✕Fine-tuning with too many epochs on small datasets—BERT fine-tuning on small datasets requires early stopping to prevent catastrophic forgetting
- ✕Using base BERT for tasks that require generation—BERT is an encoder-only model and cannot generate text; use T5 or GPT for generation
- ✕Applying BERT to very long documents without chunking—BERT has a 512-token context limit that must be handled by sliding window or hierarchical approaches
Related Terms
Transformer Encoder
The transformer encoder is a neural network architecture that processes entire input sequences bidirectionally using self-attention, producing rich contextual representations of each token that power state-of-the-art NLP models.
Sentence Transformers
Sentence transformers are neural models that produce fixed-size semantic embeddings for entire sentences, enabling efficient semantic similarity search, clustering, and retrieval by representing meaning as comparable vectors.
Word Embeddings
Word embeddings are dense vector representations of words in a continuous numerical space where semantically similar words are positioned close together, enabling machines to understand word meaning through geometry.
Text Classification
Text classification automatically assigns predefined labels to text documents—such as topic, urgency, language, or intent—enabling large-scale categorization of unstructured content without manual review.
Named Entity Recognition (NER)
Named Entity Recognition (NER) is an NLP task that identifies and classifies named entities in text—people, organizations, locations, dates, product names, and other specific items—enabling structured extraction from unstructured text.
Ready to build your AI chatbot?
Put these concepts into practice with 99helpers — no code required.
Start free trial →