Convolution & CNN architecture

This lecture builds convolution from addition and sliding kernels to CNN blocks. It covers size formulas, RGB channels, sparsity and pooling. After this page you can hand-compute a 1D convolution, apply the output-size formula and compare dense and CNN counts.

CNN Image ClassificationStatQuest · Convolution pooling image classification

Inside a Neural NetworkComputerphile · CNN internals visualized

CNNs ExplainedDataMListic · Conv math pooling stride

Addition vs convolution Search on YouTube

Addition works element by element. Convolution slides one array over the other and sums products. The two operations give very different outputs from the same inputs.

Take input $x = [1, 5, 4]$ and kernel $h = [1, 2, -2]$ . Addition pairs equal positions. Convolution combines neighbourhoods.

y[i] = x[i] + h[i]

$y[i]$ is the output at position $i$ . $x[i]$ is the input value. $h[i]$ is the kernel value at the same position.

Worked example. Add $x = [1, 5, 4]$ and $h = [1, 2, -2]$ .

-2

h

\begin{aligned} y[0] &= 1 + 1 = \mathbf{2}\\ y[1] &= 5 + 2 = \mathbf{7}\\ y[2] &= 4 + (-2) = \mathbf{2} \end{aligned}

So $y = [2, 7, 2]$ . Length stays 3.

Mathematical convolution Search on YouTube

Mathematical convolution reverses the kernel first. Then it slides the reversed kernel over the input. Partial overlaps at the edges count, so the output is longer than the input.

The kernel here is a small weight array. The term kernel (also called filter) means the same sliding array in this lecture. After this line the page uses only "kernel".

Flip the kernel end for end. Then drag it across the input. Each stop gives one output value.

y = x \ast h, \qquad L = N + M - 1

$y$ is the full convolution output. $x$ has length $N$ . $h$ has length $M$ . $L$ is the output length.

Reversal for the slide values gives $[-2, 2, 1]$ . The slide trace below keeps all five overlap positions. Length is $3 + 3 - 1 = 5$ .

Valid overlap: kernel sits fully on the input. The dot sum gives one feature-map value.

Worked example. Full trace with $x = [1, 5, 4]$ and $h = [1, 2, -2]$ . The slide order gives $[1, -8, 3, -2, -8]$ .

Step 1. First overlap.

\begin{aligned} y[0] &= 1 \times 1 = \mathbf{1} \end{aligned}

Step 2. Second overlap.

\begin{aligned} y[1] &= 1 \times 2 + 5 \times (-2)\\ &= 2 - 10 = \mathbf{-8} \end{aligned}

Step 3. Full overlap.

\begin{aligned} y[2] &= 1 \times 1 + 5 \times 2 + 4 \times (-2)\\ &= 1 + 10 - 8 = \mathbf{3} \end{aligned}

Step 4. Fourth overlap.

\begin{aligned} y[3] &= 5 \times (-2) + 4 \times 2\\ &= -10 + 8 = \mathbf{-2} \end{aligned}

Step 5. Last overlap.

\begin{aligned} y[4] &= 4 \times (-2) = \mathbf{-8} \end{aligned}

Full output is $[1, -8, 3, -2, -8]$ . Check: $1 + 10 - 8 = 3$ and $-10 + 8 = -2$ .

Do not mix full mode and valid mode in one answer. Full mode keeps edge overlaps and returns 5 values here. Valid mode keeps only the full overlap and returns one value.

Convolution in neural networks Search on YouTube

Neural networks skip the reversal. They slide the kernel as stored and keep only full overlaps. This operation is cross-correlation, but the slides call it convolution in NN. The output is shorter than the input.

O = \frac{N - M + 2P}{S} + 1

$O$ is the output length. $N$ is the input size. $M$ is the kernel size. $P$ is padding (zeroes added symmetrically at the ends). $S$ is stride (step size of the kernel).

With $P = 0$ and $S = 1$ , the formula gives $N - M + 1$ . For $N = 3$ and $M = 3$ , the output has one value.

Worked example. Valid-mode output of $x = [1, 5, 4]$ with $h = [1, 2, -2]$ .

\begin{aligned} O &= (3 - 3 + 0)/1 + 1 = \mathbf{1}\\ y[0] &= 1 \times 1 + 5 \times 2 + 4 \times (-2)\\ &= 1 + 10 - 8 = \mathbf{3} \end{aligned}

The single valid value is 3. It matches the middle value of the full trace.

In the exam write the valid dot product in one line. Then write the size formula with numbers. Examiners accept $1 + 10 - 8 = 3$ .

1D convolution Search on YouTube

1D convolution slides a short kernel along a vector. It detects local patterns such as rises, falls or bumps. The output is a feature map. A feature map is the convolution output.

The diagram labels from the slides are kernel, input vector and feature map. Each output position uses one window of the input.

S_t = \sum_{a=0}^{k-1} X_{t+a} \, W_a

$S_t$ is the output at position $t$ . $X_{t+a}$ is the input at $t + a$ . $W_a$ is the kernel weight at position $a$ . $k$ is the kernel length.

\text{Linear dim} = \frac{N - M + 2P}{S} + 1

$N$ is the input length. $M$ is the kernel length. $P$ is padding. $S$ is stride.

Stride in 1D Search on YouTube

Stride is the step size. Stride 1 moves the kernel by one position. Stride 2 skips one position between stops. Larger stride gives a shorter output.

2D convolution Search on YouTube

2D convolution slides a small matrix over an image. It sums element-wise products inside each window. The same cross-correlation rule holds in two axes.

The slides show a $3 \times 4$ target with a $2 \times 2$ kernel. Only full overlaps count in valid mode.

S_{i,j} = \sum_{m} \sum_{n} I_{i+m, j+n} \, K_{m,n}

$S_{i,j}$ is the output at row $i$ and column $j$ . $I$ is the input. $K$ is the kernel with $m \times n$ size.

\text{Width} = \frac{C_1 - C_2 + 2P}{S} + 1, \qquad \text{Height} = \frac{R_1 - R_2 + 2P}{S} + 1

$C_1$ and $R_1$ are input columns and rows. $C_2$ and $R_2$ are kernel columns and rows. $P$ is padding. $S$ is stride.

Output size, padding and stride Search on YouTube

Padding adds zeroes around the borders. It lets the kernel see edge pixels. It also controls the output size. Stride skips positions. It reduces compute and downsamples the map.

Padding: add zeroes symmetrically at the borders. It preserves edge data. It can keep the output size equal to the input size.
Stride: move the kernel by $S$ steps. It controls output size. It cuts compute and builds spatial hierarchy in CNNs.

In 2D, padding adds a zero ring around all borders. Stride can move along rows and columns together. The slides use the same stride on both axes.

Worked example. Input length $N = 5$ , kernel $M = 3$ .

\begin{aligned} P = 0, S = 1: \quad O &= (5 - 3 + 0)/1 + 1 = \mathbf{3}\\ P = 1, S = 1: \quad O &= (5 - 3 + 2)/1 + 1 = \mathbf{5}\\ P = 0, S = 2: \quad O &= (5 - 3 + 0)/2 + 1 = \mathbf{2} \end{aligned}

Padding 1 keeps size 5. Stride 2 cuts the output to 2. For images, $5 \times 5$ input with $3 \times 3$ kernel, $P = 0$ and $S = 1$ gives $3 \times 3$ .

Do not forget the final $+1$ in the size formula. Without it a $5$ -input with a $3$ -kernel gives 2. The right valid count is 3.

Drag the kernel across the input and watch each dot product form. Change size, channels and pooling and check each result against the page numbers.

1. Slide the kernel over [1, 5, 4]

Drag the kernel across the input. The readout shows the live dot product.

Position2

-8

-2

-8

y[2] = 1 x 1 + 5 x 2 + 4 x (-2) = 1 + 10 - 8 = 3

2. Output size O = (N - M + 2P) / S + 1

Input N5

Kernel M3

Padding P0

Stride S1

O = (5 - 3 + 2x0) / 1 + 1 = 3

3. RGB channels sum to one value

R patch

-1

R kernel

Bias b0

R sum = 1 + 0 - 2 + 1 = 0. O = 0 + 2 + 3 + 0 = 5

4. Pool a 4x4 input with a 2x2 window

1	3	2	0
5	6	7	8
4	2	1	0
3	9	5	6

top-left: max(1, 3, 5, 6) = 6

Convolution on images Search on YouTube

Each window gives one output value. The kernel values decide the effect. Different kernels extract different structure from the same patch.

Averaging neighbours smooths the image. It blurs detail.
Subtracting neighbours responds to change. It marks edges.

So the same slide, convolve and slide operation stays fixed. Only the kernel numbers change the meaning of the feature map.

RGB channels and multiple kernels Search on YouTube

An RGB image is a 3D array. It has height, width and depth 3. The kernel depth must match the input depth. Each channel has its own $m \times n$ kernel slice.

For each spot, take one patch per channel. Multiply each patch by its kernel slice. Sum all products across all channels. The sum is one output value. The result is a 2D feature map.

O_{i,j} = \sum_{c} \sum_{m} \sum_{n} I_{i+m, j+n, c} \, K_{m,n,c} + b

$O_{i,j}$ is the output value. $c$ runs over R, G and B. $m$ and $n$ run over kernel rows and columns. $b$ is the bias for that kernel.

Zero padding can keep the height and width. Each colour channel carries distinct data. Separate kernel slices let the model learn colour-specific edges and textures.

Worked example. One $2 \times 2$ spot across three channels. Compute one output value, no bias.

R patch

G patch

B patch

Kernels: R $= \begin{bmatrix}1&0\\-1&1\end{bmatrix}$ , G $= \begin{bmatrix}0&1\\1&0\end{bmatrix}$ , B $= \begin{bmatrix}1&1\\0&-1\end{bmatrix}$ .

\begin{aligned} \text{R sum} &= 1\cdot1 + 0\cdot0 + 2\cdot(-1) + 1\cdot1 = 1 + 0 - 2 + 1 = 0\\ \text{G sum} &= 0\cdot0 + 1\cdot1 + 1\cdot1 + 2\cdot0 = 0 + 1 + 1 + 0 = 2\\ \text{B sum} &= 2\cdot1 + 2\cdot1 + 0\cdot0 + 1\cdot(-1) = 2 + 2 + 0 - 1 = 3\\ O &= 0 + 2 + 3 = \mathbf{5} \end{aligned}

One spot gives value 5. Check: R gives 0, G gives 2, B gives 3, total 5.

Multiple kernels and depth Search on YouTube

One 3D kernel gives one 2D feature map. $N$ kernels give $N$ feature maps. Stack the maps to add depth. Output depth equals kernel count.

The slides show kernel 1, kernel 2 and kernel 3 on one RGB input. A later slide shows $F_1$ to $F_{96}$ . With 96 kernels the output depth is 96.

Why convolution Search on YouTube

Classical vision uses fixed kernels. Sobel and Prewitt mark vertical or horizontal edges. Their numbers stay frozen.

Deep learning learns the kernels. Treat kernel values as learnable parameters. Update them with backpropagation together with classifier weights. The data decides the features.

Fixed kernels are ready-made glasses. Learned kernels grind new glasses for each dataset.

Fixed kernels	Learned kernels
Handcrafted numbers (Sobel, Prewitt)	Numbers learned by backpropagation
Detect chosen edges	Detect task-useful patterns
Same for all datasets	Adapt to the input dataset

CNN vs dense network Search on YouTube

CNNs copy the visual cortex plan. Cortical cells watch small visual regions. Each cell fires for a distinct edge direction. Some cells like vertical lines. Others like horizontal or diagonal lines.

A dense network flattens the image first. It loses spatial layout. A CNN keeps the image as a grid. It slides kernels over local patches.

The count of units in one conv layer is $W_{out} \times H_{out} \times K$ . $W_{out}$ and $H_{out}$ are output width and height. $K$ is the kernel count.

Sparse connectivity Search on YouTube

Each output sees only a small patch. Distant inputs do not meet in one layer. In the slide example, inputs $x_1$ and $x_5$ do not interact in hidden layer 1. They meet later through $g_3$ . So links are sparse but indirect paths remain.

Sparsity cuts learnable parameters. The model stores fewer weights. It also overfits less.

Weight sharing reuses one kernel at all spots. A vertical-edge kernel should fire anywhere in the image. So the same weights scan all patches. Each kernel thus detects one pattern across the whole image. Learning gets easier because the model learns each pattern once.

\text{Dense: } (9 \times 4) + 4 = 40, \qquad \text{CNN: } 1 \times 4 = 4

Dense uses 9 inputs to 4 units plus 4 biases. CNN shares one kernel shape across spots and needs 4 weights. Add one bias per kernel if used, so $4 + 1 = 5$ .

Worked example. 9 inputs to 4 units vs shared kernels.

\begin{aligned} \text{Dense weights} &= 9 \times 4 = 36\\ \text{Dense biases} &= 4\\ \text{Dense total} &= 36 + 4 = \mathbf{40}\\ \text{CNN weights} &= 1 \times 4 = \mathbf{4}\\ \text{CNN with bias} &= 4 + 1 = \mathbf{5} \end{aligned}

The CNN uses about one-eighth of the count. Check: $9 \times 4 = 36$ , $36 + 4 = 40$ .

CNN building blocks Search on YouTube

The slides list six parts. Each part has one role. Data flows in this order.

Input layer: the input image.
Convolutional layer: kernels extract edges and textures into feature maps.
Activation function: usually ReLU.
Pooling layer: downsamples maps with max or average pooling.
Fully connected layer: flattens features for classification or regression.
Output layer: final result, often softmax for classification.

Early layers learn small-scale edges, textures and corners. Middle layers learn shapes and object parts. Deep layers learn whole-object patterns. This ladder supports final classification.

C-R-P-F-O keeps Full Valid straight

Conv, ReLU, Pool, FC, Output is the block order. Full adds ( $N+M-1$ ). Valid subtracts ( $(N-M+2P)/S+1$ ).

Pooling Search on YouTube

Pooling shrinks height and width. It keeps key data. It cuts compute and controls overfitting. The two slide types are max pooling and average pooling.

y = \max(x_1, x_2, x_3, x_4), \qquad y = \frac{x_1 + x_2 + x_3 + x_4}{4}

The max form picks the largest value in a $2 \times 2$ window. The average form takes the mean of the four values. Both use stride $S$ between windows.

Stride 2 gives disjoint windows. Stride 1 gives overlapping windows. The slides show both for max pooling. Replace max by mean for average pooling.

One 2x2 window: max picks 6, average gives 3.75. Stride 2 moves to a fresh window.

Worked example. $4 \times 4$ input, $2 \times 2$ window, stride 2. Outputs are $2 \times 2$ .

1	3	2	0
5	6	7	8
4	2	1	0
3	9	5	6

Max pooling. Pick the max of each window.

\begin{aligned} \text{top-left} &= \max(1, 3, 5, 6) = \mathbf{6}\\ \text{top-right} &= \max(2, 0, 7, 8) = \mathbf{8}\\ \text{bottom-left} &= \max(4, 2, 3, 9) = \mathbf{9}\\ \text{bottom-right} &= \max(1, 0, 5, 6) = \mathbf{6} \end{aligned}

Average pooling. Take the mean of each window.

\begin{aligned} \text{top-left} &= (1 + 3 + 5 + 6)/4 = 15/4 = \mathbf{3.75}\\ \text{top-right} &= (2 + 0 + 7 + 8)/4 = 17/4 = \mathbf{4.25}\\ \text{bottom-left} &= (4 + 2 + 3 + 9)/4 = 18/4 = \mathbf{4.5}\\ \text{bottom-right} &= (1 + 0 + 5 + 6)/4 = 12/4 = \mathbf{3} \end{aligned}

Max keeps strong responses. Average smooths the window.

Quick self-test Search on YouTube

1. Add

x = [1, 5, 4]

and

h = [1, 2, -2]

. What is the result?

$[2, 7, 2]$ . Add equal positions: $1 + 1 = 2$ , $5 + 2 = 7$ , $4 + (-2) = 2$ .

2. State the full-mode length for

N = 3

and

M = 3

. List the slide trace values.

$L = 3 + 3 - 1 = 5$ . Values are $[1, -8, 3, -2, -8]$ . Middle value $1 + 10 - 8 = 3$ .

3. Compute the valid-mode output of

[1, 5, 4]

with

[1, 2, -2]

$1 \times 1 + 5 \times 2 + 4 \times (-2) = 1 + 10 - 8 = 3$ . One value only.

4. Input size is 7, kernel is 3, padding is 0, stride is 2. What is the output size?

$O = (7 - 3 + 0)/2 + 1 = 4/2 + 1 = 3$ . The output has 3 positions.

5. Input is

6 \times 6

, kernel is

3 \times 3

, padding is 1, stride is 1. What is the output size?

Width $= (6 - 3 + 2)/1 + 1 = 6$ . Height is also 6. Output is $6 \times 6$ .

6. Differentiate mathematical convolution and NN convolution.

Math convolution reverses the kernel and keeps edge overlaps. NN convolution skips reversal and keeps only full overlaps. NN form is cross-correlation.

7. R, G and B sums at one spot are 0, 2 and 3. Bias is 1. What is the output?

$0 + 2 + 3 + 1 = 6$ . Sum across channels first, then add the bias of that kernel.

8. Differentiate learned kernels and Sobel kernels.

Sobel kernels have fixed numbers for chosen edges. Learned kernels start as weights and change by backpropagation. Learned kernels adapt to the dataset.

9. Compare 9 inputs to 4 dense units with a 4-weight shared kernel. Give both counts.

Dense needs $(9 \times 4) + 4 = 40$ . CNN needs $1 \times 4 = 4$ , or 5 with one bias. Sharing removes repeat weights.

10. Pool the window

\begin{bmatrix}1&3\\5&6\end{bmatrix}

by max and by average.

Max is 6. Average is $(1 + 3 + 5 + 6)/4 = 15/4 = 3.75$ . Max keeps the peak. Average reports the level.

Addition pairs equal positions: $[1, 5, 4] + [1, 2, -2] = [2, 7, 2]$ .
Full convolution length is $L = N + M - 1$ . Slide trace gives $[1, -8, 3, -2, -8]$ .
NN mode skips reversal and keeps full overlaps only. Valid value here is 3.
1D rule is $S_t = \sum X_{t+a} W_a$ . 2D rule is $S_{i,j} = \sum\sum I_{i+m,j+n} K_{m,n}$ .
Output size is $O = (N - M + 2P)/S + 1$ . Width and height use the same form.
Padding adds zeroes to save edges and set size. Stride sets step size and downsamples.
RGB output sums all channels: $O_{i,j} = \sum_c\sum_m\sum_n I K + b$ . Output depth equals kernel count.
Kernels are learned by backpropagation, unlike fixed Sobel and Prewitt kernels.
CNN uses sparsity and weight sharing. Dense needs 40 here; shared kernel needs 4 (5 with bias).
Blocks run Conv, ReLU, Pool, FC, Output. Pooling is max or average over a window.

Convolution & CNN architecture

Addition vs convolutionSearch on YouTubeBookmarkSaved

Mathematical convolutionSearch on YouTubeBookmarkSaved

Convolution in neural networksSearch on YouTubeBookmarkSaved

1D convolutionSearch on YouTubeBookmarkSaved

Stride in 1DSearch on YouTubeBookmarkSaved

2D convolutionSearch on YouTubeBookmarkSaved

Output size, padding and strideSearch on YouTubeBookmarkSaved

Convolution on imagesSearch on YouTubeBookmarkSaved

RGB channels and multiple kernelsSearch on YouTubeBookmarkSaved

Multiple kernels and depthSearch on YouTubeBookmarkSaved

Why convolutionSearch on YouTubeBookmarkSaved

CNN vs dense networkSearch on YouTubeBookmarkSaved

Sparse connectivitySearch on YouTubeBookmarkSaved

Weight sharingSearch on YouTubeBookmarkSaved

CNN building blocksSearch on YouTubeBookmarkSaved

PoolingSearch on YouTubeBookmarkSaved

Quick self-testSearch on YouTubeBookmarkSaved

On this page