Regularization and deep networks

This lecture controls overfitting with penalties, dropout, early stopping and new samples. It then scales shallow nets to deep nets with pretraining and batch normalization. After this page you can compute bias, variance and batch-norm outputs by hand.

Bias and VarianceStatQuest · Bias-variance tradeoff fundamentals

Ridge L2 RegularizationStatQuest · L2 penalty overfitting control

Dropout ExplainedMindframe Labs · Dropout prevents overfitting

Regularization Search on YouTube

Regularization prevents overfitting with a penalty on complexity. Underfitting comes from a model that is too simple. It misses the true pattern in the data. Overfitting comes from a model that is too complex. It learns noise as well as the true pattern.

Regularization adds a penalty term to the training objective. The penalty keeps weights small or sparse. It improves results on unseen data.

Think of packing for a trip. A small bag forces a few useful items. A huge bag collects clutter too.

R_{\text{reg}}(\theta) = R(\theta) + \lambda \, \Omega(\theta)

$R(\theta)$ is the empirical risk on training samples. $\Omega(\theta)$ is the penalty on weights. $\lambda$ controls the trade between fit and simplicity.

A large $\lambda$ forces a simple model. A small $\lambda$ lets the model fit the data more closely.

Bias and variance Search on YouTube

Bias is the error from wrong assumptions. It measures the gap between the average prediction and the true function. High bias means the model is too simple. It underfits the data.

Variance measures the spread of predictions across training sets. It shows how much the output moves when the data change. High variance means the model is too complex. It overfits noise in the data.

\text{Bias}(x) = E[\hat{f}(x)] - f(x)

$f(x)$ is the true function. $\hat{f}(x)$ is the learned model. $E[\hat{f}(x)]$ is the mean prediction over many training sets.

\text{Var}(x) = E\left[(\hat{f}(x) - E[\hat{f}(x)])^2\right]

$\text{Var}(x)$ averages squared gaps from the mean prediction. Large gaps mean high variance.

Expectation $E[\hat{f}(x)]$ is the mean over training sets. Fix one input such as $x = (1,2)$ . Train one net on each set $D_1, D_2, \dots, D_n$ . Each net gives one output for $x$ . The mean of those outputs is the expectation.

One training set cannot give this mean. Cross-validation splits the data into $k$ fixed folds. It trains on $k-1$ folds and tests on the held-out fold. Bootstrapping resamples with replacement. It makes more varied training sets than fixed folds.

Worked example. True output is $5$ . Model A predicts $10, 10, 10, 10$ in four runs. Model B predicts $2, 4, 6, 8$ in four runs.

Model A mean and bias.

\begin{aligned} \text{mean}_A &= (10 + 10 + 10 + 10)/4 = 40/4 = 10\\ \text{Bias}_A &= 10 - 5 = \mathbf{5} \end{aligned}

Model A variance. Each run equals the mean, so each gap is $0$ .

\begin{aligned} \text{Var}_A &= (0^2 + 0^2 + 0^2 + 0^2)/4 = \mathbf{0} \end{aligned}

Model A has high bias and low variance.

Model B mean and bias.

\begin{aligned} \text{mean}_B &= (2 + 4 + 6 + 8)/4 = 20/4 = 5\\ \text{Bias}_B &= 5 - 5 = \mathbf{0} \end{aligned}

Model B variance.

\begin{aligned} \text{Var}_B &= ((2-5)^2 + (4-5)^2 + (6-5)^2 + (8-5)^2)/4\\ &= (9 + 1 + 1 + 9)/4 = 20/4 = \mathbf{5} \end{aligned}

Model B has low bias and high variance.

Read the two cases as extremes. Constant $10$ misses the target by $5$ each time. Spread $2, 4, 6, 8$ centres on $5$ but swings widely.

Training error and validation error Search on YouTube

Training error is the loss on the training data. It shows how well the model fits seen samples. It usually falls as training runs.

Validation error is the loss on a held-out set. The model never trains on this set. It estimates results on unseen data. It may fall first and then rise again.

Training error alone misleads. A net can memorise the training set. Then training error is near zero but validation error is high. Track both curves during training.

L1 and L2 regularization Search on YouTube

L2 regularization is also called ridge or weight decay. It adds the sum of squared weights to the loss. It pushes weights toward small values. It keeps all inputs but shrinks their effect.

L1 regularization is also called lasso or sparsity. It adds the sum of absolute weights to the loss. It pushes weak weights to exact zero. It thus selects a subset of inputs.

\Omega_{L2}(w) = \sum_i w_i^2 = \Vert w \Vert_2^2

$w_i$ is one weight. Large weights pay a steep squared cost.

\Omega_{L1}(w) = \sum_i \vert w_i \vert = \Vert w \Vert_1

Small weights face steady pressure toward zero under L1.

R_{\text{reg}}(\theta) = R(\theta) + \lambda \, \Omega(w)

$\lambda$ is the regularization strength. Use L1 or L2 for $\Omega(w)$ .

Method	Penalty	Effect	Keeps
L2 ridge	sum of squares	shrinks all weights	all inputs with small weights
L1 lasso	sum of absolute values	forces sparsity	few inputs, rest at zero

Worked example. Weights are $w = [0.8, -0.4, 0.1]$ . Strength is $\lambda = 0.1$ .

L2 penalty.

\begin{aligned} \sum w_i^2 &= 0.8^2 + (-0.4)^2 + 0.1^2\\ &= 0.64 + 0.16 + 0.01 = 0.81\\ \lambda \, \Omega_{L2} &= 0.1 \times 0.81 = \mathbf{0.081} \end{aligned}

L1 penalty.

\begin{aligned} \sum \vert w_i \vert &= 0.8 + 0.4 + 0.1 = 1.3\\ \lambda \, \Omega_{L1} &= 0.1 \times 1.3 = \mathbf{0.13} \end{aligned}

L1 charges more here. It will kill the $0.1$ weight first.

Dropout and DropConnect Search on YouTube

Dropout drops random neurons during each training pass. Their outputs become zero for that pass. Each pass thus trains a different thin net. This stops neurons from co-adapting to noise.

At test time keep all neurons. Scale each output by its keep rate. If a unit stayed on half the time, halve its output at test. This average mimics the ensemble of thin nets.

a_{\text{test}} = p \, a_{\text{train}}

$a_{\text{train}}$ is the unit output in training. $p$ is the fraction of passes where the unit stayed on. $a_{\text{test}}$ is the scaled test output.

Dropout trains an efficient ensemble. One net with $n$ units gives $2^n$ dropout masks. Training tries a new mask each pass. Testing uses the scaled average instead of all masks.

DropConnect drops weights instead of neurons. It sets random links to zero during training. The unit stays active but loses some inputs. At test time keep all links.

Dropout	DropConnect
Sets random neuron outputs to zero	Sets random weights to zero
Removes a unit with all its links	Keeps the unit with fewer links
Ensemble over thin nets	Ensemble over sparse links

Keep all units at test time. Do not keep dropping them. Forgetting the scale $p$ inflates test outputs.

Early stopping Search on YouTube

Early stopping halts training before full convergence. Training error keeps falling with more epochs. Validation error falls first and then climbs. Stop where validation error starts to rise.

Training loss falls throughout. Validation loss forms a U. Stop at the low point.

Convergence watches the training loss. Stop when it barely moves between epochs. Weights have then settled. Early stopping watches the validation loss. Stop when it stops improving and starts to worsen.

Convergence	Early stopping
Signal is training loss	Signal is validation loss
Stop when change is tiny	Stop when validation loss rises
Gives full fit to training data	Blocks memorisation of noise

Data augmentation Search on YouTube

Data augmentation creates new samples from old ones. It applies label-preserving transforms to inputs. The label stays the same but the form varies. The model then learns invariance to those shifts.

Typical image transforms are flips, crops, rotations and colour shifts. Typical text moves are synonym swaps or small deletions. Choose transforms that keep the true label valid.

Augmentation suits small data sets. It is cheap because no new labelling is needed. It pairs well with other regularizers.

Lazy Lions Doze Every Afternoon

L1 penalty · L2 penalty · Dropout (and DropConnect) · Early stopping · Augmentation. Five tools against overfitting.

Deep neural networks Search on YouTube

ANN is the broad term for neural nets. It often means a shallow net with few layers. DNN means a deep net with many layers. A net with at least three hidden layers counts as a DNN.

Depth builds hierarchical features. Lower layers learn edges or parts. Higher layers combine them into objects or meanings. This automatic hierarchy is the main strength of depth.

Depth has costs. It needs more data and more compute. It also makes training less stable.

ANN	DNN
Few layers, shallow shape	At least three hidden layers
Learns simple patterns	Learns layered patterns
Needs less data and compute	Needs more data and compute

Greedy layer-wise pretraining Search on YouTube

Greedy pretraining trains one layer at a time. It served as smart initialization for deep nets. Early deep nets failed from bad random starts. This method gave each layer a useful start before fine-tuning.

Deep nets then suffered vanishing or exploding gradients. Vanishing means gradients shrink toward zero as they move back. Lower layers then get almost no update. Exploding means gradients grow huge as they move back. Weights then jump and training turns unstable.

Each layer learns to reconstruct its own input. It maps input $x$ to code $h$ and back. No labels take part, so the goal is unsupervised. Repeat this for each hidden layer in turn.

Train layer 1 to reconstruct the raw input.
Freeze layer 1 and feed its code forward.
Train layer 2 to reconstruct that code.
Repeat up to the last hidden layer.
Add the output layer and fine-tune all weights.

Fine-tuning uses the task labels. It starts from the pretrained weights. It adjusts the whole net with supervision.

State the order in the exam. Pretrain each hidden layer alone with reconstruction. Then add the output layer and fine-tune everything.

Batch normalization Search on YouTube

Internal covariate shift means layer inputs keep moving. Each update changes early weights. Later layers then see a new input spread. Learning must chase a moving target.

The slide analogy uses a changing syllabus. Students cannot build on past lessons. Later layers face the same trouble with shifting inputs.

Saturation is one result. Sigmoid or tanh units hit flat zones. Gradients then shrink and learning stalls. Training also slows because each layer adapts to drift.

Elliptic bowls force zigzag steps. Circular bowls allow direct steps from any side.

Shape matters for speed. An elliptic bowl has steep and flat sides. Steps from one start differ sharply from another start. Close contour lines mark the steep side. A circular bowl looks the same from each side. Steps from the same level reach the centre in like time.

Batch normalization fixes the spread per minibatch. It takes one minibatch of activations. It subtracts the batch mean. It divides by the batch spread. Each layer then sees steady zero-mean inputs.

\mu_B = \frac{1}{m}\sum_{i=1}^{m} x_i

$m$ is the minibatch size. $x_i$ is one activation in the batch. $\mu_B$ is the batch mean.

\sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m}(x_i - \mu_B)^2

$\sigma_B^2$ is the batch variance. It averages squared gaps from $\mu_B$ .

\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}

$\hat{x}_i$ is the normalized value. $\epsilon$ is a small constant for stability. Output has zero mean and unit variance.

A minibatch size is a hyperparameter. It sets how many samples precede each update. Normalization uses batch stats, not single samples.

Worked example. One neuron outputs $2, 4, 4, 6$ on four samples. Use $\epsilon = 0$ for hand work.

Batch mean.

\begin{aligned} \mu_B &= (2 + 4 + 4 + 6)/4\\ &= 16/4 = \mathbf{4} \end{aligned}

Batch variance.

\begin{aligned} \sigma_B^2 &= ((2-4)^2 + (4-4)^2 + (4-4)^2 + (6-4)^2)/4\\ &= (4 + 0 + 0 + 4)/4 = 8/4 = \mathbf{2} \end{aligned}

Normalized values. Spread is $\sqrt{2} = 1.414$ .

\begin{aligned} \hat{x}_1 &= (2-4)/1.414 = -2/1.414 = \mathbf{-1.414}\\ \hat{x}_2 &= (4-4)/1.414 = \mathbf{0}\\ \hat{x}_3 &= (4-4)/1.414 = \mathbf{0}\\ \hat{x}_4 &= (6-4)/1.414 = 2/1.414 = \mathbf{1.414} \end{aligned}

Check. Mean is $(-1.414 + 0 + 0 + 1.414)/4 = 0$ . Variance is $(2 + 0 + 0 + 2)/4 = 1$ .

Batch normalization differs from input normalization. Input normalization standardizes raw inputs once before training. Batch normalization standardizes layer outputs during training. It repeats on every minibatch.

Ensemble and boosting Search on YouTube

An ensemble trains several base learners. Each base learner is one model. Their predictions then combine into one final output. Errors differ across models, so the average cancels single faults.

f_{\text{ens}}(x) = \frac{1}{M}\sum_{m=1}^{M} f_m(x)

$M$ is the count of base learners. $f_m(x)$ is the output of learner $m$ . The mean is the final prediction for regression or votes.

Boosting is a sequential ensemble. It trains the same shape many times on the same data. Each round stresses the samples that prior rounds missed. Later models thus fix earlier faults.

Dropout links to ensembles. Each mask is one thin member. Test scaling averages them without storing them all.

Hyperparameter tuning Search on YouTube

Hyperparameters control learning itself. They are set before training starts. Weights are learned, but hyperparameters are chosen.

Hyperparameter	Role
Learning rate	Step size for each weight update
Batch size	Samples before each update
Epochs	Full passes over the training set
Optimizer type	Rule for weight updates
Activation function	Nonlinearity such as ReLU, sigmoid, tanh
Dropout rate	Share of units dropped in training
Weight init	Start values such as Xavier or He
Regularization strength	Penalty weight $\lambda$ for L1 or L2

w_{\text{new}} = w - \eta \, g

$w$ is one weight. $g$ is its loss gradient. $\eta$ is the learning rate. Small $\eta$ moves slowly and steadily. Large $\eta$ moves fast but may skip minima.

The learning rate often lies between $0.0001$ and $0.1$ . A tiny rate needs many epochs. A huge rate overshoots good minima. Adaptive rates change $\eta$ during training. They decay fast-moving weights more. They suit vision, language and other large tasks.

Grid search tries each combination in the search space. It is thorough but costly. Random search tries random combinations. It is cheaper and often finds a strong setting fast.

Grid search	Random search
Tests all listed combinations	Tests random combinations
Cost grows fast with options	Cost stays bounded by trial count
Fits small search spaces	Fits large search spaces

Lazy Bears Eat Oranges And Dates In Rows

Learning rate · Batch size · Epochs · Optimizer.

Activation · Dropout rate · Init · Regularization strength. Eight knobs shape each run.

Quick self-test Search on YouTube

1. State the bias and variance for predictions 10, 10, 10, 10 with true value 5.

Mean is $10$ . Bias is $10 - 5 = 5$ . Variance is $0$ . This is high bias with low variance.

2. State the bias and variance for predictions 2, 4, 6, 8 with true value 5.

Mean is $(2+4+6+8)/4 = 5$ . Bias is $0$ . Variance is $(9+1+1+9)/4 = 5$ . This is low bias with high variance.

3. Define expectation and name two ways to approximate it.

Expectation is the mean output for one input over many training sets. Approximate it with $k$ -fold cross-validation or bootstrapping with replacement.

4. Differentiate training error and validation error. Why track both?

Training error uses seen data and falls with epochs. Validation error uses held-out data and may rise after a point. Both reveal memorisation.

5. Weights are 0.5 and minus 0.5 with lambda 0.2. Give the L1 and L2 penalties.

L1 is $0.2 \times (0.5+0.5) = 0.2$ . L2 is $0.2 \times (0.25+0.25) = 0.1$ . L1 charges more and favours sparsity.

6. Differentiate dropout and DropConnect. What happens at test time?

Dropout zeroes random neuron outputs. DropConnect zeroes random weights. At test time keep all units and weights, with dropout scaled by $p$ .

7. Differentiate convergence and early stopping as stop rules.

Convergence stops when training loss barely moves. Early stopping stops when validation loss starts to rise. The first fits fully, the second blocks overfitting.

8. Batch values are 2, 4, 4, 6. Compute the normalized outputs.

Mean is $4$ and variance is $2$ . Outputs are $-1.414, 0, 0, 1.414$ . Their mean is $0$ and variance is $1$ .

9. Differentiate batch normalization and input normalization.

Input normalization standardizes raw inputs once before training. Batch normalization standardizes layer outputs on each minibatch during training.

10. Differentiate grid search and random search. When does each suit?

Grid search tests all combinations and suits small spaces. Random search tests random picks and suits large spaces at lower cost.

Regularization adds $\lambda \, \Omega(\theta)$ to the objective. It trades fit against simplicity.
High bias means underfit with steady wrong outputs. High variance means overfit with swinging outputs.
Expectation is the mean over training sets. Cross-validation and bootstrapping approximate it.
Training error falls with epochs. Validation error may rise again after its low point.
L2 uses squared weights and shrinks all weights. L1 uses absolute weights and forces zeros.
Dropout zeroes random neurons with test scale $p$ . DropConnect zeroes random weights.
Early stopping halts at rising validation loss. Convergence halts at settled training loss.
Data augmentation adds label-preserving variants for invariance.
A DNN has at least three hidden layers. It learns layered features at high data and compute cost.
Greedy pretraining rebuilds each layer input in turn. Fine-tuning then trains all weights with labels.
Batch norm maps each minibatch to zero mean and unit variance with $\hat{x}_i = (x_i - \mu_B)/\sqrt{\sigma_B^2 + \epsilon}$ .
Ensembles average base learners. Boosting retrains in sequence on missed samples.
Key knobs are learning rate, batch size, epochs, optimizer, activation, dropout rate, init and $\lambda$ . Grid search is thorough, random search is cheaper.

Regularization and deep networks

RegularizationSearch on YouTubeBookmarkSaved

Bias and varianceSearch on YouTubeBookmarkSaved

Training error and validation errorSearch on YouTubeBookmarkSaved

L1 and L2 regularizationSearch on YouTubeBookmarkSaved

Dropout and DropConnectSearch on YouTubeBookmarkSaved

Early stoppingSearch on YouTubeBookmarkSaved

Data augmentationSearch on YouTubeBookmarkSaved

Deep neural networksSearch on YouTubeBookmarkSaved

Greedy layer-wise pretrainingSearch on YouTubeBookmarkSaved

Batch normalizationSearch on YouTubeBookmarkSaved

Ensemble and boostingSearch on YouTubeBookmarkSaved

Hyperparameter tuningSearch on YouTubeBookmarkSaved

Quick self-testSearch on YouTubeBookmarkSaved

On this page