History of Large Language Models

Before models could generate fluent text, they needed a way to represent meaning that a network could work with. In 2013, Google researchers led by Tomas Mikolov published Word2Vec, which showed that relatively small neural networks could learn vector representations of words from large text corpora. The well-known example, king minus man plus woman lands near queen, made the idea concrete: semantic relationships could be captured as geometry in vector space.

Word2Vec mattered because it gave natural language processing a shared representation layer that was cheap to train and easy to reuse across tasks. It helped bridge the older statistical NLP tradition and the neural language-model era that followed. The embeddings were static, one vector per word regardless of context, a limit that later transformer-based models would remove. But the core idea that meaning could live in a learned vector space carried straight into everything that came after.

In June 2017, eight researchers at Google published 'Attention Is All You Need,' introducing the transformer. It replaced the recurrence that older sequence models depended on with self-attention, a mechanism that lets the model weigh every token against every other token and process the whole sequence in parallel. That parallelism is what made training at scale practical, because it mapped cleanly onto modern GPU hardware in a way that recurrent networks never did.

The architecture is the reason the modern LLM era exists. Transformers sit behind GPT, BERT, Claude, Llama, and effectively every frontier system that followed, and several of the paper's co-authors went on to found their own AI companies. The shift was as much about engineering as theory: once attention removed the sequential bottleneck, the question stopped being how to design a better recurrent cell and became how far scale and data could push a single, uniform architecture.

Two transformer-based approaches set the template for what came next, and they read the same text in opposite ways. In June 2018, OpenAI released GPT-1, a 117-million-parameter model that learned by predicting the next token across large amounts of text, then fine-tuned for specific tasks. Four months later, Google published BERT, which read context from both directions at once, broke records on 11 NLP benchmarks, and was folded into Google Search where it affected about 10 percent of queries.

The shared lesson was that pre-training on unlabeled text, then adapting the result, beat building task-specific systems from scratch. GPT-2 pushed the generative side of that idea in 2019 with 1.5 billion parameters, and OpenAI's initial decision to withhold the full model, calling it too dangerous to release, became the first major safety controversy of the LLM era. The full GPT-2 weights shipped later that year, but the episode set the tone for how labs would talk about release decisions going forward.

In June 2020, OpenAI released GPT-3 with 175 billion parameters, about 100 times the size of GPT-2. Without fine-tuning, it could write essays, code, and poetry and translate languages through few-shot learning, picking up a task from a few examples placed in the prompt. GPT-3 gave the scaling hypothesis real credibility: bigger models produced capabilities no one had explicitly trained for, and the API launch spawned an ecosystem of startups built on top of it.

The public turning point came on 30 November 2022, when OpenAI released ChatGPT, built on GPT-3.5 and tuned with reinforcement learning from human feedback. It reached a million users in five days and 100 million in two months, the fastest consumer adoption on record, and it made the technology legible to people who had never read an AI paper. GPT-4 followed in March 2023, adding image understanding and passing professional exams such as the bar at the 90th percentile, which pushed every industry to take the capabilities seriously and intensified the competition between labs.

After ChatGPT, the field stopped being a single-lab story. Anthropic, founded by former OpenAI researchers, released Claude in March 2023, trained with Constitutional AI, an approach where the model follows a written set of principles rather than relying only on human preference ratings. In July 2023, Meta released Llama 2 as open weights with broad commercial use permitted, which made self-hosted, customizable frontier-quality models a mainstream option and produced thousands of fine-tunes within weeks.

The next shift was about how models spend compute rather than how large they are. In September 2024, OpenAI released o1, trained to reason through chain-of-thought at inference time, spending more compute to work through hard math, coding, and science problems step by step. That introduced test-time compute scaling as a second lever alongside model size, and it pointed the field toward systems that deliberate before answering rather than only predicting the next token faster.