LeNet, AlexNet and embeddings

This lecture covers LeNet-5, hierarchical features, AlexNet and image embeddings. It shows learnable parameter counts, output sizes and Euclidean nearest neighbours. After this page you can compute kernel counts and compare image embeddings by Euclidean distance.

CNNs ExplainedComputerphile · Convolutions pooling classification

CNN ArchitecturesStanford · AlexNet VGG GoogLeNet ResNet

Word Embedding Word2VecStatQuest · Embeddings from context windows

LeNet Search on YouTube

LeNet is one of the earliest influential CNNs. It classified grey handwritten digits 0-9 from MNIST data. It proved that convolutions plus subsampling can read images end to end.

A CNN kernel is also called filter. This page uses only kernel after this sentence.

LeNet-5 stacks convolution, subsampling and fully connected layers. Each convolution layer applies several 3D kernels and produces a stack of feature maps. Each pooling layer reduces height and width and controls overfitting.

LeNet-5 pipeline. Convolutions build feature maps. C5 acts as a fully connected layer. The end predicts the digit label.

Layers and activation function Search on YouTube

LeNet-5 used sigmoid activation functions in hidden layers. The sigmoid squashes each sum into the range 0 to 1. Later networks replaced it with ReLU for faster training.

\sigma(z) = \frac{1}{1 + e^{-z}}

$z$ is the weighted sum plus bias at one unit. $\sigma(z)$ is the output between 0 and 1. LeNet-5 applied this activation function after convolution and fully connected layers.

A standard LeNet-5 layout is input $32\times32\times1$ , C1 $28\times28\times6$ , S2 $14\times14\times6$ , C3 $10\times10\times16$ , S4 $5\times5\times16$ , C5 $120$ , F6 $84$ , output $10$ . C means convolution. S means subsampling or pooling. F means fully connected.

When the input has depth $D$ , each kernel has depth $D$ . The depth of the C-layer output equals $K$ , the number of kernels. For RGB input the first depth is 3, but the output depth is still $K$ .

C5 behaves as a fully connected layer Search on YouTube

C5 receives $5\times5\times16$ from S4. It applies $120$ kernels of size $5\times5\times16$ . Each kernel covers the whole input field.

H_{out} = \frac{H_{in} - F + 2P}{S} + 1

$H_{in}$ is the input height. $F$ is the kernel height. $P$ is padding rounds. $S$ is stride. The same rule gives width. Apply it per layer to track each feature map size.

Since input size equals kernel size, the output is $1\times1$ . So each kernel gives one number. The result is $120$ numbers. That is the same as a fully connected layer with $120$ units.

Worked example. LeNet C1 output size and learnable parameter count.

Output size. Input is $32\times32\times1$ . Kernel is $5\times5$ . Stride is 1. Padding is 0.

\begin{aligned} H_{out} &= (32 - 5 + 0)/1 + 1 = 28\\ \text{feature map} &= 28 \times 28 \times 6 \end{aligned}

Learnable parameter count. Each kernel holds $5\times5\times1$ weights plus one bias.

\begin{aligned} \text{per kernel} &= 5 \times 5 \times 1 + 1 = 26\\ \text{C1 total} &= 26 \times 6 = \mathbf{156} \end{aligned}

Without bias the count is $25 \times 6 = 150$ . State the bias rule in the exam.

Worked example. LeNet C5 and total learnable parameter count.

C5 size. Input is $5\times5\times16$ . Kernel is $5\times5$ . Stride is 1.

\begin{aligned} H_{out} &= (5 - 5)/1 + 1 = 1\\ \text{feature map} &= 1 \times 1 \times 120 \end{aligned}

C5 count. Each of the $120$ kernels sees all $16$ input maps.

\begin{aligned} \text{per kernel} &= 5 \times 5 \times 16 + 1 = 401\\ \text{C5 total} &= 401 \times 120 = \mathbf{48120} \end{aligned}

Full net. C1 has $156$ . C3 has $(150+1)\times16=2416$ . F6 has $120\times84+84=10164$ . Output has $84\times10+10=850$ .

\begin{aligned} \text{total} &= 156 + 2416 + 48120 + 10164 + 850\\ &= \mathbf{61706} \end{aligned}

That is about $60$ K learnable parameters.

Write the bias rule first. Then use $(F_h \times F_w \times D_{in} + 1) \times K$ . Examiners accept both $150$ and $156$ for C1 if you state the rule.

Do not call C5 a convolution in the usual sliding sense. It uses convolution kernels, but each kernel sees the full field. Treat it as a fully connected layer.

Hierarchical features Search on YouTube

Early CNN layers learn small-scale features. They detect edges, textures and corners. Middle layers learn medium-scale features. They combine edges into shapes and object parts.

Deeper layers learn global features. They combine parts into whole objects and abstract patterns. The network thus builds combinations across the full image. This hierarchy supports final classification.

Think of bricks, walls and houses. Early layers see bricks. Middle layers see walls. Deep layers see the house.

ImageNet challenge Search on YouTube

ImageNet challenged models to classify RGB images into 1000 classes. The plot of error versus layers showed deeper CNNs with better training beat shallow models. The task needs colour, texture and part cues at many scales.

LeNet suited small grey digits. ImageNet needs far larger capacity and stronger regularization. AlexNet supplied both.

AlexNet Search on YouTube

AlexNet revolutionized computer vision with deep CNNs. It has $8$ learnable layers: $5$ convolutional layers plus $3$ fully connected layers. Depth alone did not explain the gain. Five training tricks improved performance and generalization.

r(z) = \max(0, z)

$z$ is the weighted sum plus bias. $r(z)$ keeps positive inputs and sets negative inputs to 0. AlexNet used this ReLU activation function instead of sigmoid.

The five tricks are ReLU activations, overlapping max pooling, dropout, normalization of pixel values and data augmentation with resize to $256$ , crops and rotations.

R-O-D-N-D: Run Over Drops, Never Dry

ReLU · Overlapping pool · Dropout · Normalization · Data augmentation. AlexNet runs over dropout-regularized maps and never dries out on new images.

ReLU trains faster than sigmoid. Its gradient stays 1 for positive inputs. Sigmoid saturates at both ends and slows learning.

Overlapping pooling uses a window larger than its stride, for example $3\times3$ with stride $2$ . Neighbour windows overlap. The overlap keeps more spatial context than disjoint windows.

Dropout randomly zeroes hidden units during training. It breaks co-adaptation between units. It reduces overfitting on large fully connected layers.

Normalization rescales pixel values and normalizes responses across feature maps. It keeps activations in a stable range. Training becomes faster and more stable.

Data augmentation enlarges the training set. It resizes images to $256$ , takes random crops, flips and rotates them. The network sees new variants each epoch.

Worked example. AlexNet conv1 output size and learnable parameter count.

Output size. Input is $227\times227\times3$ . Kernel is $11\times11$ . Stride is $4$ . Padding is $0$ .

\begin{aligned} H_{out} &= (227 - 11 + 0)/4 + 1 = 216/4 + 1 = 55\\ \text{feature map} &= 55 \times 55 \times 96 \end{aligned}

Learnable parameter count. There are $96$ kernels, each $11\times11\times3$ plus one bias.

\begin{aligned} \text{per kernel} &= 11 \times 11 \times 3 + 1 = 364\\ \text{conv1 total} &= 364 \times 96 = \mathbf{34944} \end{aligned}

One AlexNet layer already holds more learnable parameters than all of LeNet C1 and C3 combined.

For output size, compute the numerator first. Check that it divides by stride. Then add 1. For counts, compute one kernel first, then multiply by $K$ .

Learnable parameter scale Search on YouTube

The slide summary contrasts about $60$ K with about $60$ M learnable parameters. LeNet holds about $60$ K. AlexNet holds about $60$ M. The ratio is about $1000$ .

The growth comes from RGB inputs, larger images, more kernels per layer and wide fully connected layers. The fully connected layers dominate the AlexNet total.

Worked example. Why AlexNet reaches about $60$ M: FC6 count.

Input size. After convolutions and pooling, the map is $6\times6\times256$ .

\begin{aligned} \text{inputs} &= 6 \times 6 \times 256 = 9216\\ \text{outputs} &= 4096 \end{aligned}

Learnable parameter count. Each output has one weight per input plus one bias.

\begin{aligned} \text{weights} &= 9216 \times 4096 = 37748736\\ \text{total} &= 37748736 + 4096 = \mathbf{37752832} \end{aligned}

This one layer holds about $37.8$ M learnable parameters. That alone is about $600$ times the LeNet total.

LeNet	AlexNet
Grey $32\times32$ inputs, $10$ labels	RGB $227\times227$ inputs, $1000$ labels
$5$ convolutions or subsampling plus $2$ dense	$5$ convolutions plus $3$ dense ( $8$ learnable)
Sigmoid activation function	ReLU activation function
No dropout or large augmentation	Dropout, normalization and augmentation
About $60$ K learnable parameters	About $60$ M learnable parameters

Kernels vs tensor outputs Search on YouTube

Each kernel produces one 2D feature map. $K$ kernels produce $K$ stacked maps. The slide shows $64$ learnable kernels giving $64$ outputs.

Each kernel spans the full input depth. For RGB input, each kernel is 3D with depth $3$ . For a $256$ -map input, each kernel has depth $256$ . The model learns every kernel value from the dataset.

P = (F_h \times F_w \times D_{in} + 1) \times K

$F_h \times F_w$ is the spatial kernel size. $D_{in}$ is the input depth. $+1$ is one bias per kernel. $K$ is the kernel count. The output depth equals $K$ .

So output depth is not a free choice after training. It equals the kernel count by construction. More kernels mean more feature maps and more learnable parameters.

Conv3 and conv4 without pooling Search on YouTube

AlexNet pools after conv1, conv2 and conv5. It omits pooling between conv3 and conv4. Both are deeper layers with $3\times3$ kernels. They focus on complex patterns such as object parts.

Pooling reduces height and width. It can destroy fine spatial detail. Skipping pooling here preserves spatial resolution. The network retains detailed part layouts for accurate classification.

Do not place pooling after every convolution by default. Pooling helps invariance and speed, but each pooling step loses location detail. AlexNet keeps conv3 and conv4 dense for that reason.

Embedding space Search on YouTube

An image embedding space is a high-dimensional mathematical space. Each image maps to one vector in that space. These vectors are called embeddings.

AlexNet learned representations in such a space. Similar images mapped to nearby points. Points close together shared more semantic traits. A dog image sat near other dog images, far from truck images.

Embedding scatter. The two dog embeddings lie close together. The truck embedding lies far away.

The output layer then assigns proposed labels. It converts the last embedding into scores over $1000$ labels. The top scores are the proposed labels for the test image.

Similar feature vectors by Euclidean distance Search on YouTube

Given a test image, AlexNet compares its feature vector with training vectors. It ranks training images by Euclidean distance in the output layer. The smallest distances indicate the most similar feature vectors.

The slide illustrates the idea with four feature vectors over traits such as weight, height and ear size. Distance uses all traits together. Close vectors share trait profiles.

d(\mathbf{a}, \mathbf{b}) = \sqrt{\sum_{i=1}^{n}(a_i - b_i)^2}

$\mathbf{a}$ is the test vector. $\mathbf{b}$ is one training vector. $n$ is the embedding size. Small $d$ means high similarity.

Worked example. Nearest neighbour among four 3-dim vectors.

Data. Test vector $\mathbf{q}=[4,2,3]$ . Training vectors $\mathbf{v}_1=[4,2,4]$ , $\mathbf{v}_2=[5,3,3]$ , $\mathbf{v}_3=[1,1,1]$ , $\mathbf{v}_4=[4,5,3]$ .

\begin{aligned} d_1 &= \sqrt{(0)^2+(0)^2+(-1)^2} = \sqrt{1} = 1\\ d_2 &= \sqrt{( -1)^2+( -1)^2+(0)^2} = \sqrt{2} = 1.41\\ d_3 &= \sqrt{(3)^2+(1)^2+(2)^2} = \sqrt{14} = 3.74\\ d_4 &= \sqrt{(0)^2+( -3)^2+(0)^2} = \sqrt{9} = \mathbf{3} \end{aligned}

Rank. $d_1 \lt d_2 \lt d_4 \lt d_3$ .

The nearest neighbour is $\mathbf{v}_1$ with distance $\mathbf{1}$ . The slide reports such neighbours as similar feature vectors to the test image.

Square first, then sum, then root. For ranking only, compare squared distances. The order is the same and you skip four roots.

Quick self-test Search on YouTube

1. State the LeNet task and the standard input and output sizes.

LeNet classified grey handwritten digits 0-9. Standard input is $32\times32\times1$ . Standard output is $10$ labels.

2. Compute the C1 output size for

32\times32

input,

5\times5

kernel, stride

1

and no padding.

$H_{out}=(32-5)/1+1=28$ . The feature map is $28\times28\times6$ for $6$ kernels.

3. Compute the C1 learnable parameter count for

6

kernels of

5\times5\times1

with bias.

Per kernel $25+1=26$ . Total $26\times6=156$ . Without bias the total is $150$ .

4. Why does C5 behave as a fully connected layer?

Its input is $5\times5\times16$ and each kernel is $5\times5\times16$ . Each kernel covers the full field and gives one number. So $120$ kernels give $120$ numbers like a dense layer.

5. Differentiate early, middle and deep CNN features.

Early layers learn edges, textures and corners. Middle layers learn shapes and object parts. Deep layers learn whole objects and abstract patterns.

6. List the five AlexNet tricks and state the layer count.

AlexNet has $5$ convolutions plus $3$ dense layers ( $8$ learnable). Tricks are ReLU, overlapping pooling, dropout, normalization and data augmentation.

7. Compute the AlexNet conv1 output width for

227

input,

11

kernel and stride

4

$W_{out}=(227-11)/4+1=55$ . The feature map is $55\times55\times96$ .

8. Why do conv3 and conv4 skip pooling?

They capture complex object parts and need spatial detail. Pooling would shrink the maps and destroy fine location cues. Skipping it preserves resolution for classification.

9. Compare

60

K and

60

M. Which layer type drives the AlexNet total?

LeNet holds about $60$ K learnable parameters. AlexNet holds about $60$ M, about $1000$ times more. Wide fully connected layers drive it: FC6 alone holds about $37.8$ M.

10. Rank these training vectors for test

\mathbf{q}=[4,2,3]

[4,2,4]

[5,3,3]

[4,5,3]

Distances are $1$ , $1.41$ and $3$ . Rank is $[4,2,4] \lt [5,3,3] \lt [4,5,3]$ . The first is the nearest neighbour.

LeNet classified grey digits with stacked convolution, pooling and dense layers. It used sigmoid activation functions.
Standard sizes run $32\times32\times1$ to C1 $28\times28\times6$ , S2 $14\times14\times6$ , C3 $10\times10\times16$ , S4 $5\times5\times16$ , C5 $120$ , F6 $84$ , output $10$ .
C5 behaves as a fully connected layer: $5\times5\times16$ input with $5\times5$ kernels gives $1\times1\times120$ . Its count is $48120$ .
Early layers learn edges and textures. Middle layers learn shapes and parts. Deep layers learn objects.
ImageNet needs $1000$ RGB labels. It rewards deeper nets with strong regularization.
AlexNet has $5$ convolutions plus $3$ dense layers. Tricks are R-O-D-N-D: ReLU, Overlapping pool, Dropout, Normalization, Data augmentation.
Output size is $(H_{in}-F+2P)/S+1$ . AlexNet conv1 gives $55\times55\times96$ with $34944$ learnable parameters.
Learnable count is $(F_h\times F_w\times D_{in}+1)\times K$ . LeNet totals about $60$ K. AlexNet totals about $60$ M.
Conv3 and conv4 skip pooling to preserve spatial detail for object parts.
An embedding maps each image to a vector. Close vectors share semantic traits. Nearest neighbours use $d(\mathbf{a},\mathbf{b})=\sqrt{\sum(a_i-b_i)^2}$ .

LeNet, AlexNet and embeddings

LeNetSearch on YouTubeBookmarkSaved

Layers and activation functionSearch on YouTubeBookmarkSaved

C5 behaves as a fully connected layerSearch on YouTubeBookmarkSaved

Hierarchical featuresSearch on YouTubeBookmarkSaved

ImageNet challengeSearch on YouTubeBookmarkSaved

AlexNetSearch on YouTubeBookmarkSaved

Learnable parameter scaleSearch on YouTubeBookmarkSaved

Kernels vs tensor outputsSearch on YouTubeBookmarkSaved

Conv3 and conv4 without poolingSearch on YouTubeBookmarkSaved

Embedding spaceSearch on YouTubeBookmarkSaved

Similar feature vectors by Euclidean distanceSearch on YouTubeBookmarkSaved

Quick self-testSearch on YouTubeBookmarkSaved

On this page