The Transformer Era
2018–2021 · 9 milestones
Transformers scaled to billions of parameters. GPT and BERT redefined NLP. AI began generating text, code, and images that stunned researchers.
Milestones
BERT: Bidirectional Language Understanding
Google published BERT (Bidirectional Encoder Representations from Transformers), which could understand language context from both directions simultaneously. BERT shattered records on 11 NLP benchmarks. Google integrated it into Search, affecting 10% of all queries.
GPT-1: Generative Pre-training
OpenAI released GPT-1, demonstrating that a Transformer trained on vast amounts of text using unsupervised pre-training could then be fine-tuned for specific NLP tasks. With 117 million parameters, it showed the potential of scaling language models.
GPT-2: 'Too Dangerous to Release'
OpenAI announced GPT-2 (1.5 billion parameters) but initially refused to release the full model, calling it 'too dangerous' due to its ability to generate convincing fake text. The decision was controversial — some praised the caution, others called it a publicity stunt. The full model was eventually released in November 2019.
AlphaStar Masters StarCraft II
DeepMind's AlphaStar reached Grandmaster level in StarCraft II, a real-time strategy game requiring long-term planning, deception, and split-second tactics with incomplete information — far more complex than Go or chess.
GPT-3: The 175 Billion Parameter Leap
OpenAI released GPT-3 with 175 billion parameters — 100x larger than GPT-2. Without any fine-tuning, GPT-3 could write essays, code, poetry, translate languages, and answer questions through 'few-shot learning' (learning from just a few examples in the prompt). The API launched in beta, enabling thousands of applications.
AlphaFold 2: Protein Folding Solved
DeepMind's AlphaFold 2 solved the 50-year-old protein structure prediction problem, achieving accuracy comparable to experimental methods at CASP14. It could predict how proteins fold from their amino acid sequences — a problem that had stumped biologists for half a century.
DALL-E: Text to Image Generation
OpenAI unveiled DALL-E, a model that could generate images from text descriptions — 'an armchair in the shape of an avocado' became iconic. Built on GPT-3's architecture adapted for images, it showed that language models could bridge the gap between text and visual creativity.
Anthropic Founded
Former OpenAI VP of Research Dario Amodei and his sister Daniela, along with several other OpenAI researchers, founded Anthropic — an AI safety company focused on building reliable, interpretable, and steerable AI systems.
GitHub Copilot: AI Writes Code
GitHub launched Copilot as a technical preview — an AI pair programmer powered by OpenAI Codex that could autocomplete entire functions, write boilerplate, and suggest code from natural language comments. It was trained on billions of lines of public code.