Optimizers from momentum to Adam

This lecture compares paths down the error surface from gradient descent to Adam. It covers momentum, Nesterov lookahead, batch and minibatch updates, Adagrad, RMSprop and second-order methods. After this page you can run one update step of each optimizer by hand.

SGD Clearly ExplainedStatQuest · Stochastic GD mini-batch

Adam Optimization AlgorithmDeepLearningAI · Adam RMSprop momentum combined

Adam is Momentum+RMSPropDataMListic · Momentum RMSprop Adam race

Paths down the error surface Search on YouTube

Optimization moves the parameter vector from high loss to low loss. The slides use Zoey at height A. Zoey must reach point B below. Four routes exist: inclined slide, spiral slide, direct jump and staircase. Each route gives a different arrival time and cost. Training behaves the same way. Different update rules trace different paths to the minimum.

Think of Zoey on a hill. A steep slide is fast but rough. A spiral slide is longer but smoother. Small stairs are safe but slow.

The route decides convergence speed and stability. A good optimizer reaches low loss in fewer updates. It also avoids wild oscillations and long stalls.

\theta_{t+1} = \theta_t - \eta \nabla J(\theta_t)

$\theta_t$ is the parameter vector at iteration $t$ . $\eta$ is the learning rate. $\nabla J(\theta_t)$ is the gradient of the loss at iteration $t$ . The step size scales with the slope. Steep regions give large steps. Gentle regions give small steps.

Gradient sign and size Search on YouTube

The gradient sign shows the direction of rise. The gradient size shows the rate of change. A positive slope means the function rises as $x$ rises. A negative slope means the function falls as $x$ rises. To descend, move opposite to the gradient.

Worked example. Check $f(x) = x^2 + 1$ with $f^\prime(x) = 2x$ at $x = 1$ , $x = 2$ and $x = -1$ .

Function values.

\begin{aligned} f(1) &= 1^2 + 1 = \mathbf{2}\\ f(2) &= 2^2 + 1 = \mathbf{5}\\ f(-1) &= (-1)^2 + 1 = \mathbf{2}\\ f(0) &= 0^2 + 1 = \mathbf{1} \end{aligned}

Slope sign check.

\begin{aligned} f^\prime(1) &= 2 \cdot 1 = \mathbf{2 \gt 0} \;\Rightarrow\; \text{rises as } x \text{ rises}\\ f^\prime(2) &= 2 \cdot 2 = \mathbf{4 \gt 0} \;\Rightarrow\; \text{rises faster, steeper}\\ f^\prime(-1) &= 2 \cdot (-1) = \mathbf{-2 \lt 0} \;\Rightarrow\; \text{falls as } x \text{ rises} \end{aligned}

Descent sizes. Move $x$ from 2 to 1, then from 1 to 0.

\begin{aligned} f(2) - f(1) &= 5 - 2 = \mathbf{3} \;\; \text{large drop on steep part}\\ f(1) - f(0) &= 2 - 1 = \mathbf{1} \;\; \text{small drop on gentle part} \end{aligned}

One gradient descent step. Start at $x = 2$ with $\eta = 0.1$ .

\begin{aligned} x_{\text{new}} &= 2 - 0.1 \cdot 4 = 2 - 0.4 = \mathbf{1.6}\\ f(1.6) &= 1.6^2 + 1 = 2.56 + 1 = \mathbf{3.56} \end{aligned}

Loss falls from 5 to 3.56 in one step.

Error surface and start position Search on YouTube

A 2D error surface needs a top view. Take a horizontal slice of the bowl. View the contours from above. The slide marks two paths from level J1 to level J2. Path BD crosses contours in a short distance. Path AC runs a longer way across the same levels. BD is steeper than AC. The steeper direction gives faster descent.

Top view of contours. BD is short and steep. AC is long and gentle. Both reach J2.

Gentle slopes cause slow progress with plain gradient descent. The start position then matters a lot. Start S1 may sit on a steep face and converge fast. Start S2 may sit on a flat plain and crawl. The same rule with the same learning rate gives very different times.

Momentum Search on YouTube

Momentum accelerates gradient descent with memory of past steps. It adds a velocity term that accumulates past gradients. It gains speed when gradients point in one direction. It damps oscillations when gradients flip sign. It helps on steep valleys, small local minima and noisy gradients.

v_t = \beta v_{t-1} + \nabla J(\theta_t)

\theta_{t+1} = \theta_t - \eta v_t

$v_t$ is the velocity at iteration $t$ . $\beta$ is the momentum coefficient, often 0.9. $\eta$ is the learning rate. $\nabla J(\theta_t)$ is the current gradient. $\theta_t$ is the parameter vector.

Overshoot and valley oscillations Search on YouTube

Momentum can run past the goal. The stored velocity pushes the parameters forward. The profile view on the slides shows this overshoot risk. The top view shows oscillations across a narrow valley. Plain descent creeps down the valley floor. Momentum swings from side to side while moving forward. It still reaches the minimum faster despite larger swings.

The slides fit a sigmoid with weight $w$ and bias $b$ from start $(w, b) = (2, 6)$ . Each update favours the downhill direction of the loss. The $(w, b)$ point shifts toward the valley floor. After several swings it settles at the best fitting pair.

Do not set $\beta$ near 1 without care. The stored velocity grows large. The updates then overshoot the minimum and oscillate hard.

Examiners accept $\beta = 0.9$ as the standard momentum value. State the velocity rule first. Then state that consistent directions speed up and flips cancel out.

Nesterov accelerated gradient Search on YouTube

Nesterov accelerated gradient (NAG) improves momentum with a lookahead. It does not use the gradient at the current point. It first jumps along the stored velocity. It then measures the gradient at that lookahead point. This peek corrects the course before a bad leap. It cuts overshoot and oscillation.

\tilde{\theta}_t = \theta_t - \beta v_{t-1}

g_t = \nabla J(\tilde{\theta}_t)

v_t = \beta v_{t-1} + \eta g_t

\theta_{t+1} = \theta_t - v_t

$\tilde{\theta}_t$ is the lookahead point. $g_t$ is the gradient at the lookahead point. $v_t$ is the corrected velocity. $\theta_{t+1}$ is the new parameter vector. The four steps are lookahead, gradient, velocity correction and parameter update.

Look, Grade, Velocity, Drop: L-G-V-D

Lookahead point $\tilde{\theta}_t$ · Gradient at lookahead $g_t$ · Velocity correction $v_t$ · Drop to $\theta_{t+1} = \theta_t - v_t$ .

Momentum	NAG
Uses the gradient at the current point	Uses the gradient at the lookahead point
Takes a big leap, then sees the slope	Peeks ahead, then adjusts the leap
Faster than plain descent but can overshoot	Less overshoot and less oscillation
One gradient per step at $\theta_t$	One gradient per step at $\tilde{\theta}_t$

Worked example. NAG step on $J(\theta) = \theta^2$ with $\nabla J(\theta) = 2\theta$ . Set $\theta_t = 1.0$ , $v_{t-1} = 0.2$ , $\beta = 0.9$ and $\eta = 0.1$ .

Step 1. Lookahead point.

\begin{aligned} \tilde{\theta}_t &= 1.0 - 0.9 \cdot 0.2\\ &= 1.0 - 0.18 = \mathbf{0.82} \end{aligned}

Step 2. Gradient at lookahead.

\begin{aligned} g_t &= 2 \cdot 0.82 = \mathbf{1.64} \end{aligned}

The gradient at the current point would be $2.0$ . The lookahead sees the gentler slope ahead.

Step 3. Corrected velocity.

\begin{aligned} v_t &= 0.9 \cdot 0.2 + 0.1 \cdot 1.64\\ &= 0.18 + 0.164 = \mathbf{0.344} \end{aligned}

Step 4. Parameter update.

\begin{aligned} \theta_{t+1} &= 1.0 - 0.344 = \mathbf{0.656} \end{aligned}

Plain momentum would use gradient 2.0 and give $v_t = 0.38$ and $\theta_{t+1} = 0.62$ . NAG takes the shorter corrected step.

Batch, stochastic and minibatch updates Search on YouTube

The three methods differ only in data per update. The per-epoch update count decides speed and noise.

Batch gradient descent Search on YouTube

Batch gradient descent uses all $N$ training points for one update. It averages the $N$ gradients. It then applies the mean gradient once. One epoch gives one update. The path is smooth and direct. The cost per update grows with $N$ . Large datasets make each step costly.

Stochastic gradient descent Search on YouTube

Stochastic gradient descent (SGD) updates after each single point. One epoch gives $N$ updates. Learning starts at once without waiting for the full pass. Early progress is fast. The path is noisy and less stable. The noise helps escape small local minima and saddle points. SGD suits large datasets with frequent updates.

Minibatch gradient descent Search on YouTube

Minibatch gradient descent sits between the two. Fix a minibatch size $B$ . Update after each block of $B$ points. One epoch gives $N/B$ updates. It keeps the stability of batch updates. It keeps much of the speed of SGD.

\text{batch updates per epoch} = 1 \qquad \text{SGD updates per epoch} = N \qquad \text{minibatch updates per epoch} = N/B

$N$ is the training set size. $B$ is the minibatch size. One epoch means one full pass over the data. $B = 1$ gives SGD. $B = N$ gives batch descent.

Algorithm	Updates per epoch	Character
Batch gradient descent	1	Smooth, costly per step
Stochastic gradient descent	$N$	Fast start, noisy path
Minibatch gradient descent	$N/B$	Stable and fast, the common choice

Worked example. Count updates for $N = 10000$ points with minibatch size $B = 100$ .

\begin{aligned} \text{batch} &= \mathbf{1} \text{ update per epoch}\\ \text{SGD} &= \mathbf{10000} \text{ updates per epoch}\\ \text{minibatch} &= 10000/100 = \mathbf{100} \text{ updates per epoch} \end{aligned}

Over 5 epochs.

\begin{aligned} \text{batch total} &= 5 \cdot 1 = \mathbf{5}\\ \text{SGD total} &= 5 \cdot 10000 = \mathbf{50000}\\ \text{minibatch total} &= 5 \cdot 100 = \mathbf{500} \end{aligned}

A second case: $N = 50000$ with $B = 50$ gives $50000/50 = \mathbf{1000}$ minibatch updates per epoch.

Write $N/B$ for minibatch updates per epoch in the exam. Add the two ends: $B = 1$ gives $N$ SGD updates, $B = N$ gives 1 batch update.

Adagrad Search on YouTube

Different weights need different step sizes. Some weights see large gradients often. They need small steps to avoid overshoot. Other weights see tiny or rare gradients. They need large steps to move at all. One global learning rate cannot serve both groups. Uneven feature scales make the gap worse.

Adaptive gradient (Adagrad) gives each weight its own rate. It tracks the sum of squared gradients per weight. Weights with large past gradients get smaller rates. Weights with small past gradients keep larger rates.

G_t = G_{t-1} + g_t^2

\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_t + \epsilon}} g_t

$G_t$ is the accumulated sum of squared gradients. $g_t$ is the current gradient. $\eta$ is the base learning rate. $\epsilon$ is a small constant that avoids division by zero. All operations act per weight. The fraction is the effective learning rate for that weight.

Worked example. Two weights share $\eta = 0.1$ . Weight $w_1$ sees large gradients. Weight $w_2$ sees small gradients. Start with $G_0 = 0$ for both. Use $\epsilon = 0$ for hand calculation.

Iteration 1. Gradients are $g = 2.0$ for $w_1$ and $g = 0.5$ for $w_2$ .

\begin{aligned} G_{1,w_1} &= 0 + 2.0^2 = \mathbf{4.0}\\ \text{effective rate}_{w_1} &= 0.1/\sqrt{4.0} = 0.1/2 = \mathbf{0.05}\\ \Delta w_1 &= 0.05 \cdot 2.0 = \mathbf{0.10}\\ G_{1,w_2} &= 0 + 0.5^2 = \mathbf{0.25}\\ \text{effective rate}_{w_2} &= 0.1/\sqrt{0.25} = 0.1/0.5 = \mathbf{0.20}\\ \Delta w_2 &= 0.20 \cdot 0.5 = \mathbf{0.10} \end{aligned}

So $w_1$ gets the smaller effective rate and $w_2$ gets the larger one.

Iteration 2. Same gradients again.

\begin{aligned} G_{2,w_1} &= 4.0 + 4.0 = \mathbf{8.0}\\ \text{effective rate}_{w_1} &= 0.1/\sqrt{8.0} = 0.1/2.8284 = \mathbf{0.0354}\\ \Delta w_1 &= 0.0354 \cdot 2.0 = \mathbf{0.0707}\\ G_{2,w_2} &= 0.25 + 0.25 = \mathbf{0.50}\\ \text{effective rate}_{w_2} &= 0.1/\sqrt{0.50} = 0.1/0.7071 = \mathbf{0.1414}\\ \Delta w_2 &= 0.1414 \cdot 0.5 = \mathbf{0.0707} \end{aligned}

Both rates shrink by $1/\sqrt{2}$ . The denominator only grows. Learning stalls with time.

Do not expect Adagrad to train to the end. The sum $G_t$ grows without bound. The effective rate falls toward zero and updates freeze.

RMSprop Search on YouTube

Root mean square propagation (RMSprop) fixes the decay of Adagrad. It replaces the full sum with a moving average. Recent squared gradients matter more. Old ones fade away. The denominator then stays bounded.

v_t = \beta v_{t-1} + (1 - \beta) g_t^2

\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t + \epsilon}} g_t

$v_t$ is the moving average of squared gradients. $\beta$ is the decay rate, often 0.9. $g_t$ is the current gradient. $\eta$ is the base learning rate. $\epsilon$ avoids division by zero. Each weight keeps its own $v_t$ .

A short check shows the effect. With $\beta = 0.9$ , each new squared gradient adds only 10 percent weight. A long run of large gradients still leaves room for new steps. The method keeps adaptive rates without the aggressive freeze.

Adam Search on YouTube

Adaptive moment estimation (Adam) joins both ideas. Momentum tracks the mean gradient. RMSprop tracks the mean squared gradient. Adam keeps both moving averages. It corrects their early bias. It then applies an adaptive momentum step.

m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t

v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2

$m_t$ is the first moment, the momentum average. $v_t$ is the second moment, the adaptive average. $\beta_1 = 0.9$ and $\beta_2 = 0.999$ are the decay rates. $g_t$ is the current gradient. Both averages start at zero.

\hat{m}_t = \frac{m_t}{1 - \beta_1^t} \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}

\theta_{t+1} = \theta_t - \frac{\eta \hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

$\hat{m}_t$ and $\hat{v}_t$ are the bias-corrected moments. $t$ is the iteration index starting at 1. $\eta$ is the learning rate. $\epsilon$ is a small constant. The correction matters most in early steps when the averages still lean toward zero.

The four Adam steps are first moment, second moment, bias correction and parameter update. A variant called Nadam swaps plain momentum for Nesterov lookahead momentum. It applies the lookahead idea inside the Adam first moment.

G-M-A-R-D-A-N: Good Momentum And Rates Defeat All Noise

Gradient descent base · Momentum velocity · Nesterov Ahead lookahead. SGD and minibatch Rate is $N/B$ . Divided rates mark Adagrad · Averaged squares mark RMSprop · Adam and Nadam join both.

Adam in four: M-V-C-U. Mean gradient $m_t$ · Variance term $v_t$ · Corrected $\hat{m}_t$ , $\hat{v}_t$ · Update $\theta$ .

Worked example. One Adam step with $g_1 = 0.6$ , $\beta_1 = 0.9$ , $\beta_2 = 0.999$ , $\eta = 0.001$ and $\epsilon = 10^{-8}$ .

Moments at $t = 1$ from zero start.

\begin{aligned} m_1 &= 0.9 \cdot 0 + 0.1 \cdot 0.6 = \mathbf{0.06}\\ v_1 &= 0.999 \cdot 0 + 0.001 \cdot 0.36 = \mathbf{0.00036} \end{aligned}

Bias correction at $t = 1$ .

\begin{aligned} \hat{m}_1 &= 0.06/(1 - 0.9) = 0.06/0.1 = \mathbf{0.6}\\ \hat{v}_1 &= 0.00036/(1 - 0.999) = 0.00036/0.001 = \mathbf{0.36} \end{aligned}

Parameter change.

\begin{aligned} \Delta \theta &= 0.001 \cdot 0.6/(\sqrt{0.36} + 10^{-8})\\ &= 0.001 \cdot 0.6/0.6 = \mathbf{0.001} \end{aligned}

Without correction the raw ratio $m_1/\sqrt{v_1}$ would give $0.06/0.019 = 3.16$ times $\eta$ . The correction restores the true gradient scale of 0.6.

Second-order methods Search on YouTube

First-order methods use only the gradient $\nabla f(\theta)$ . This group holds plain descent, SGD, RMSprop and Adam. The gradient gives the slope. It does not give the curvature. Second-order methods also use the Hessian $H = \nabla^2 f(\theta)$ . The Hessian shows how the gradient itself changes. Curvature guides a more exact step.

\theta_{t+1} = \theta_t - H^{-1} \nabla f(\theta_t)

$H = \nabla^2 f(\theta_t)$ is the Hessian matrix of second derivatives. $H^{-1}$ is its inverse. $\nabla f(\theta_t)$ is the gradient. This is Newton's method. It follows curvature in each direction.

The rule is simple. For a sharply curved direction, move a short way. For a flat direction, move a farther way. Convergence needs fewer iterations in theory. It also routes around tricky saddle zones better. Two limits block its use in deep nets. The Hessian has squared size in the parameter count. It is huge for large networks. Its inverse costs even more to compute and store.

Saddle points Search on YouTube

A saddle point has zero gradient without being a minimum or maximum. It slopes up in some directions. It slopes down in others. Gradient methods rely on slope to move. At zero slope they see no direction. They may halt and report a false minimum. Training then stalls or slows hard.

Saddle shape. One slice curves up. The other curves down. The centre has zero slope.

Deep loss surfaces have high dimension. Saddle points there outnumber true local minima by far. Methods with memory or curvature escape more often. Momentum rolls through on stored velocity. Adam adapts each rate and keeps motion. Second-order methods read curvature and step away.

Race four optimizers down one loss surface. Change steps, surface and paths to test each rule.

Valley surface, start (w,b)=(2,6)

→

— GD

— Momentum

— NAG

— Adam

η 0.12, Adam 0.35

NAG looks ahead to 0.82. It sees slope 1.64, not 2.0. It steps to 0.656. Momentum steps to 0.62.

Adagrad gives 0.05 to w1 and 0.20 to w2. RMSprop holds both rates alive.

Adagrad w1 0.05

Adagrad w2 0.20

RMSprop w1 0.05

RMSprop w2 0.20

Steps 28

Step 28. GD loss 0.00. Momentum loss 4.57. NAG loss 0.00. Adam loss 2.90.

Valley view: GD creeps down the floor. Momentum swings but runs fast. NAG cuts the swing. Adam steadies each direction. Adagrad iter 2 falls to 0.0354 and 0.1414. Both shrink by 1/√2.

Quick self-test Search on YouTube

1. State the Zoey analogy and the plain gradient descent rule.

Zoey at height A must reach B below by slide, spiral, jump or stairs. Each path gives a different time and cost. Training matches this: $\theta_{t+1} = \theta_t - \eta \nabla J(\theta_t)$ . Steep slopes give large steps and gentle slopes give small steps.

2. For

f(x) = x^2 + 1

, find the slope sign at

x = 2

and

x = -1

. Which change drops more,

2 \to 1

1 \to 0

$f^\prime(x) = 2x$ . At $x = 2$ the slope is 4, positive and steep. At $x = -1$ the slope is $-2$ , negative. The drop $2 \to 1$ is $5 - 2 = 3$ . The drop $1 \to 0$ is $2 - 1 = 1$ . The steep part drops more.

3. Differentiate momentum and NAG in one line each. Write the NAG four steps.

Momentum uses the gradient at the current point. NAG uses the gradient at the lookahead point. The four steps are lookahead $\tilde{\theta}_t$ , gradient $g_t$ , velocity $v_t$ and update $\theta_{t+1} = \theta_t - v_t$ .

4. Run one NAG step:

\theta = 1.0

v = 0.2

\beta = 0.9

\eta = 0.1

J = \theta^2

Lookahead is $1.0 - 0.18 = 0.82$ . Gradient is $1.64$ . Velocity is $0.18 + 0.164 = 0.344$ . New $\theta = 1.0 - 0.344 = 0.656$ .

5. Differentiate batch, SGD and minibatch by updates per epoch for

N = 10000

and

B = 100

Batch gives 1 update. SGD gives $N = 10000$ updates. Minibatch gives $N/B = 100$ updates. Over 5 epochs the totals are 5, 50000 and 500.

6. In Adagrad with

\eta = 0.1

G_{w_1} = 4.0

and

G_{w_2} = 0.25

, which weight gets the larger effective rate?

Effective rate is $0.1/\sqrt{G}$ . Weight $w_1$ gets $0.05$ . Weight $w_2$ gets $0.20$ . The weight with small past gradients gets the larger rate.

7. Why does Adagrad stall, and how does RMSprop fix it?

Adagrad sums all squared gradients in $G_t$ . The denominator grows without bound and the rate falls toward zero. RMSprop uses a moving average $v_t = \beta v_{t-1} + (1-\beta)g_t^2$ with $\beta$ near 0.9. Old gradients fade, so the rate stays alive.

8. Write the four Adam steps and the bias correction at

t = 1

for

g_1 = 0.6

Steps are first moment $m_t$ , second moment $v_t$ , bias correction and update. Here $m_1 = 0.06$ and $v_1 = 0.00036$ . Correction gives $\hat{m}_1 = 0.6$ and $\hat{v}_1 = 0.36$ . The step is $\eta \cdot 0.6/0.6 = \eta$ .

9. Differentiate first-order and second-order methods. State two limits of Newton updates.

First-order methods use only the gradient $\nabla f$ . Second-order methods add the Hessian $H = \nabla^2 f$ and use $\theta - H^{-1}\nabla f$ . The Hessian is huge for large nets. Its inverse is costly to compute and store.

10. What is a saddle point, and which methods escape it more often?

A saddle point has zero gradient but is neither minimum nor maximum. Plain descent stalls there. Momentum, Adam and second-order methods escape more often through stored velocity, adaptive rates or curvature data.

Plain gradient descent uses $\theta_{t+1} = \theta_t - \eta \nabla J(\theta_t)$ . Steep slopes step far. Gentle slopes step short.
For $f(x) = x^2 + 1$ , $f^\prime = 2x$ . Slopes are 2 at $x = 1$ , 4 at $x = 2$ and $-2$ at $x = -1$ .
Momentum adds velocity $v_t = \beta v_{t-1} + \nabla J$ . It speeds consistent directions and damps flips. Standard $\beta = 0.9$ .
NAG runs four steps: lookahead, gradient at lookahead, velocity correction and update. It cuts overshoot.
Batch gives 1 update per epoch. SGD gives $N$ . Minibatch gives $N/B$ . $B = 1$ is SGD and $B = N$ is batch.
Adagrad scales each weight by $\eta/\sqrt{G_t + \epsilon}$ . $G_t$ grows forever, so the rate decays to zero.
RMSprop uses $v_t = \beta v_{t-1} + (1-\beta)g_t^2$ with $\beta$ near 0.9. It keeps adaptive rates alive.
Adam keeps $m_t$ and $v_t$ , corrects bias with $1 - \beta^t$ terms and updates with $\eta \hat{m}_t/(\sqrt{\hat{v}_t} + \epsilon)$ . Nadam uses Nesterov momentum inside Adam.
Newton uses $\theta - H^{-1}\nabla f$ . It moves short on sharp curves and far on flat ones. Hessian size and inversion block large nets.
Saddle points have zero gradient but are not minima. They dominate deep loss surfaces. Memory and curvature methods escape them.

Optimizers from momentum to Adam

Paths down the error surfaceSearch on YouTubeBookmarkSaved

Gradient sign and sizeSearch on YouTubeBookmarkSaved

Error surface and start positionSearch on YouTubeBookmarkSaved

MomentumSearch on YouTubeBookmarkSaved

Overshoot and valley oscillationsSearch on YouTubeBookmarkSaved

Nesterov accelerated gradientSearch on YouTubeBookmarkSaved

Batch, stochastic and minibatch updatesSearch on YouTubeBookmarkSaved

Batch gradient descentSearch on YouTubeBookmarkSaved

Stochastic gradient descentSearch on YouTubeBookmarkSaved

Minibatch gradient descentSearch on YouTubeBookmarkSaved

AdagradSearch on YouTubeBookmarkSaved

RMSpropSearch on YouTubeBookmarkSaved

AdamSearch on YouTubeBookmarkSaved

Second-order methodsSearch on YouTubeBookmarkSaved

Saddle pointsSearch on YouTubeBookmarkSaved

Quick self-testSearch on YouTubeBookmarkSaved

On this page