CNN timeline - Timeline

1754: A French mathematician D'Alembert uses convolutions in a derivation of Taylor's Theorem. This is one of the first known instances of the use of convolutions.

1951: (SGD) A precursor to gradient descent is presented in the paper "A Stochastic Approximation Method." Other sources point to gradient descent being used in control theory as far back as 1940 but I couldn't find actual publications confirming this.

1952: (SGD) "Stochastic Estimation of the Maximum of a Regression Function" is published. This paper contains a complete description of stochastic gradient descent with a batch size of 1.

DNN The Perceptron: A Probabilistic Model For Information Storage And Organization In The Brain Introduction paper to the perceptron (single layer of a DNN)

1971: (DNN) First working multilayer perceptron model introduced in Cybernetics and forecasting techniques. Not trained with SGD.

1970: (Backprop) "The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors," a masters thesis is presented by Seppo Linnainmaa. Commonly said to be the first presentation of backpropagation.

1980: (CNN) Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position is published. First paper to introduce the convolutional layer, based on work related to monkey and cat visual cortex. Not trained with SGD

1986: The term deep learning is first used in the machine learning paper "Learning while searching in constraint-satisfaction problems"

1989: Backpropagation applied to handwritten zip code recognition - Today's paper! LeCun creates one of the first applied, useful deep convolutional neural network.

1998: LeNet is presented in "Gradient-based learning applied to document recognition"

2012: AlexNet is presented in "ImageNet classification with deep convolutional neural networks"

ResNet is presented in "Deep Residual Learning for Image Recognition"

1950-1970: Two doctors, Hubel and Wiesel show that cat and monkey visual cortexes contain neurons that individually respond to small regions of the visual field