History of Neural Networks

Neural networks started as a question about logic, not biology. In 1943, Warren McCulloch and Walter Pitts published a model of an artificial neuron and showed that simple binary neurons wired into networks could, in principle, compute any function a Turing machine could. That paper gave the field its conceptual foundation: intelligence might be assembled from large numbers of simple connected units.

In 1957, Frank Rosenblatt turned the idea into hardware with the Mark I Perceptron, the first physical neural network, which learned to classify simple visual patterns. The press treated it as an electronic brain on the verge of consciousness. That gap between a working classifier and the claims made for it set a pattern the field would repeat: overpromise, underdeliver, retreat.

In 1969, Marvin Minsky and Seymour Papert published Perceptrons, proving that a single-layer perceptron could not solve the XOR problem or other tasks that are not linearly separable. The math was correct. The interpretation that spread beyond it was not: many readers took the book to mean neural networks themselves were a dead end, even though multi-layer networks could handle the very problems the proof ruled out for single layers.

The effect was a funding and research freeze that lasted more than a decade. Money moved to symbolic AI, and the connectionist program went quiet. This is one of the clearest cases in AI history of a narrow technical result being read as a verdict on a whole approach, and it set up the first AI winter for the field's neural-network wing.

The method that would eventually train deep networks existed during the freeze. In 1974, Paul Werbos described backpropagation in his PhD thesis, a way to train multi-layer networks by propagating errors backward through the layers. In the anti-neural-network climate of the 1970s, the work went largely unnoticed, a reminder that an algorithm can be available years before a field is willing to use it.

The idea returned in 1986, when David Rumelhart, Geoffrey Hinton, and Ronald Williams published a Nature paper showing backpropagation could train multi-layer networks effectively, alongside the connectionist Parallel Distributed Processing volumes. Backpropagation became the standard training method and the algorithmic backbone of the deep learning wave that followed.

Even with backpropagation working, neural networks needed an architecture suited to images and enough data and compute to train it. Yann LeCun supplied the architecture in 1989, showing that convolutional neural networks could be trained with backpropagation to read handwritten digits. The refined LeNet-5 later hit over 99 percent accuracy on MNIST and ran in bank check readers and ATMs for years, proving these networks could do real commercial work even during the second AI winter.

The other two ingredients arrived around 2009. Fei-Fei Li's team built ImageNet, a dataset of over 14 million labeled images that gave vision research a benchmark at an unprecedented scale. The same year, researchers including Andrew Ng showed that GPUs could train neural networks tens of times faster than CPUs, removing the compute bottleneck. Hinton's 2006 deep belief networks had already shown deep networks could be trained layer by layer; now the data and hardware were in place.

In 2012, the ingredients came together. AlexNet, a deep convolutional network built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, won the ImageNet competition by a wide margin, cutting the top error rate from 26 percent to 16 percent. Trained on two NVIDIA GTX 580 GPUs, it was deeper than anything before it, and it ended the debate about whether deep learning worked at scale. Traditional computer vision pipelines were obsolete almost overnight.

Depth kept growing. In 2015, Microsoft Research introduced ResNet, whose skip connections let networks reach 152 layers and beyond without breaking during training. ResNet won ImageNet 2015 with a 3.57 percent error rate, passing human-level performance on the benchmark for the first time. Residual connections then became a standard building block across deep architectures, including the Transformers that power today's frontier models.