/

Formula cheatsheet

Every key formula from the course, one table per lecture. Click a heading to open the full lecture.

Perceptron & linear separability Search on YouTube

Topic	Formula	Meaning
MP sum	$f(x) = \sum_{i=1}^{2} w_i x_i$	Fixed-weight sum before threshold
AND threshold	$a = \begin{cases} 1 & \text{if } \sum_{i=1}^{2} w_i x_i \ge 2 \\ 0 & \text{if } \sum_{i=1}^{2} w_i x_i \lt 2 \end{cases}$	AND fires only at sum two
Supervised map	$\hat{y} = f(X)$	Learned map predicts output
Single loss	$\text{loss} = Y - \hat{Y}$	Single prediction error signal
Sigmoid	$a = \sigma(z) = \frac{1}{1 + e^{-z}}$	Graded activity between zero one
Threshold form	$a = \begin{cases} 1 & \text{if } \sum_{i=1}^{n} w_i x_i \ge t \\ 0 & \text{if } \sum_{i=1}^{n} w_i x_i \lt t \end{cases}$	Fire when sum reaches threshold
Bias net	$z = \sum_{i=1}^{n} w_i x_i + w_0$	Bias replaces threshold, fire at zero
Boundary forms	$\sum_{i=1}^{n} w_i x_i + w_0 = 0 \qquad\Longleftrightarrow\qquad \sum_{i=1}^{n} w_i x_i - t = 0 \qquad\Longleftrightarrow\qquad \sum_{i=1}^{n} w_i x_i = t$	Same boundary in three guises
Line form	$w_1 x_1 + w_2 x_2 + w_0 = 0 \qquad\Longleftrightarrow\qquad y = mx + c \;\; \text{means} \;\; mx - y + c = 0$	Two-input boundary is line
OR constraints	$w_0 \lt 0;\; w_0 + w_1 \ge 0;\; w_0 + w_2 \ge 0;\; w_0 + w_1 + w_2 \ge 0$	Four inequalities define OR
PLA update	$w \leftarrow w + x$ (miss); $w \leftarrow w - x$ (false alarm)	Miss adds, false alarm subtracts
Separability	$\sum w_i x_i \gt -w_0$ for every point in $P$ ; $\lt -w_0$ for every point in $N$	All P positive, all N negative

ANN building blocks & activation functions Search on YouTube

Topic	Formula	Meaning
Neuron sum	$z = \sum_{i=1}^{m} w_i x_i + b \qquad a = f(z)$	Weighted sum plus activation level
Parameter step	$\theta \leftarrow \theta - \eta\,\frac{d\,\text{Loss}}{d\theta}$	Step against gradient scaled by rate
Identity / step / sign	$f(z) = z$ ; $0$ if $z \lt 0$ else $1$ ; $-1/0/+1$ by sign of $z$	Piecewise activation trio
Sigmoid / tanh	$\sigma(z) = \frac{1}{1 + e^{-z}}$ ; $\tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}$	Ranges (0,1) and (−1,1)
ReLU / SiLU	$f(z) = \max(0, z)$ ; $f(z) = z\,\sigma(z)$	Sparse rectifier and smooth version
Softmax	$f_i(z_i) = \frac{\exp(z_i)}{\sum_{j=1}^{k}\exp(z_j)} \qquad \sum_{i=1}^{k} f_i = 1$	Exponentiate normalize to probabilities
Slopes	$\sigma'(z) = \sigma(z)\,(1 - \sigma(z)) \le 0.25$ ; $\tanh'(z) = 1 - \tanh^{2}(z)$ ; ReLU slope $0$ or $1$	Peak slopes govern vanishing flow
GeLU	$\text{GeLU}(z) \approx 0.5\,z\left(1 + \tanh\left(\sqrt{2/\pi}\,\left(z + 0.044715\,z^{3}\right)\right)\right)$	Smooth NLP activation approximation

Feedforward networks & gradient descent Search on YouTube

Topic	Formula	Meaning
Hidden detector	$h_k = 1 \text{ if } a_k x_1 + b_k x_2 \gt 1, \text{ else } 0$	One unit fires per pattern
Output OR	$y = 1 \text{ if } \sum_{i=1}^{4} w_i h_i \gt w_0, \text{ else } 0$	OR over hidden winners
Sigmoid neuron	$f(z) = \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad z = wx + b$	Neuron output between zero one
Squared error	$L = \frac{1}{n}\sum_{i=1}^{n}(\hat{y}_i - y_i)^2, \qquad n = 2$	Average squared prediction error
Descent rule	$\theta_{\text{new}} = \theta - \eta\,\nabla L(\theta)$	Step opposite gradient downhill
Taylor reason	$L(\theta + \Delta\theta) \approx L(\theta) + \nabla L(\theta)\cdot\Delta\theta$	Predict loss change from slope
Gradient shape	$f'(x) = \frac{df}{dx}, \qquad \nabla f(x, y) = \begin{bmatrix} df/dx \\ df/dy \end{bmatrix}$	Slope scalar versus vector gradient
Sigmoid tweak	$\Delta w = (f(x) - y)\,f(x)\,(1 - f(x))\,x, \qquad \Delta b = (f(x) - y)\,f(x)\,(1 - f(x))$	Per-sample tweak, average over points
Binary loss	$L = -\big[y\log p + (1 - y)\log(1 - p)\big]$	Punish confident wrong binary bets
Multiclass loss	$L = -\sum_{c=1}^{C} y_c\,\log p_c$	Only true class contributes
Approximation	One hidden layer approximates any continuous $f$ to any $\epsilon$	Universal approximation guarantee
Curse	$\text{neurons needed} \;\propto\; k^n$	Width explodes exponentially with dimension

Backpropagation & empirical risk Search on YouTube

Topic	Formula	Meaning
Forward pass	$z = \sum_i w_i x_i + b \qquad a = f(z) \qquad h = a \qquad O = F(z)$	Forward sums then activations
Sample loss	$L(y, O) = \frac{1}{n}\sum_{i=1}^{n}(y_i - O_i)^2$	Mean squared gap to label
Sigmoid slope	$\sigma^{\prime}(z) = \sigma(z)(1 - \sigma(z))$	Slope needed for chain rule
Output signal	$\delta_o = -2(y - O)\,O(1 - O) \qquad w := w - \eta\,\frac{\partial L}{\partial w}$	Output error times slope, descend
Hidden signal	$\delta_{h1} = \delta_o\,v_1\,h_1(1 - h_1) \qquad \delta_{h2} = \delta_o\,v_2\,h_2(1 - h_2)$	Backprop splits error by weights
Chain rule	$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial O}\,\frac{\partial O}{\partial z}\,\frac{\partial z}{\partial w} \qquad \frac{\partial z}{\partial w} = x$	Multiply local slopes for gradient
Risk	$R(\theta) = \frac{1}{n}\sum_{i=1}^{n}L(y_i, f(X_i, \theta))$	Average training loss to minimize

Regularization & deep networks Search on YouTube

Topic	Formula	Meaning
Penalized objective	$R_{\text{reg}}(\theta) = R(\theta) + \lambda \, \Omega(\theta)$	Fit plus complexity penalty
Bias variance	$\text{Bias}(x) = E[\hat{f}(x)] - f(x)$ , $\text{Var}(x) = E\left[(\hat{f}(x) - E[\hat{f}(x)])^2\right]$	Mean error versus spread
Sparse penalties	$\Omega_{L2}(w) = \sum_i w_i^2 = \Vert w \Vert_2^2$ , $\Omega_{L1}(w) = \sum_i \vert w_i \vert = \Vert w \Vert_1$	Squares shrink, absolutes sparsify
Dropout test	$a_{\text{test}} = p \, a_{\text{train}}$	Scale test output by keep rate
Batch norm	$\mu_B = \frac{1}{m}\sum_{i=1}^{m} x_i$ , $\sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m}(x_i - \mu_B)^2$ , $\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$	Standardize batch to zero mean
Ensemble mean	$f_{\text{ens}}(x) = \frac{1}{M}\sum_{m=1}^{M} f_m(x)$	Average models cancels faults
Rate step	$w_{\text{new}} = w - \eta \, g$	Learning rate scales gradient step

Optimizers: momentum to Adam Search on YouTube

Topic	Formula	Meaning
Base descent	$\theta_{t+1} = \theta_t - \eta \nabla J(\theta_t)$	Slope-scaled downhill step
Momentum	$v_t = \beta v_{t-1} + \nabla J(\theta_t)$ , $\theta_{t+1} = \theta_t - \eta v_t$	Accumulate past gradients for speed
Lookahead	$\tilde{\theta}_t = \theta_t - \beta v_{t-1}$ , $g_t = \nabla J(\tilde{\theta}_t)$ , $v_t = \beta v_{t-1} + \eta g_t$ , $\theta_{t+1} = \theta_t - v_t$	Peek ahead before leaping
Update counts	Batch $1$ ; SGD $N$ ; minibatch $N/B$ updates per epoch	One, N, N over B updates
Adagrad	$G_t = G_{t-1} + g_t^2$ , $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_t + \epsilon}} g_t$	Shrink rate for frequent gradients
RMSprop	$v_t = \beta v_{t-1} + (1 - \beta) g_t^2$ , $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t + \epsilon}} g_t$	Moving average keeps rates alive
Adam moments	$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$ , $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$	Mean and variance trackers
Adam correction	$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$ , $\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$ , $\theta_{t+1} = \theta_t - \frac{\eta \hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$	Debias early then adaptive step
Newton	$\theta_{t+1} = \theta_t - H^{-1} \nabla f(\theta_t)$	Curvature-guided exact step

Convolution & CNN architecture Search on YouTube

Topic	Formula	Meaning
Addition	$y[i] = x[i] + h[i]$	Pairwise sum, same length
Full length	$y = x \ast h, \qquad L = N + M - 1$	Flip slide, longer output
NN size	$O = \frac{N - M + 2P}{S} + 1$	Valid size from padding stride
1D slide	$S_t = \sum_{a=0}^{k-1} X_{t+a} \, W_a$	Slide dot product along vector
2D slide	$S_{i,j} = \sum_{m} \sum_{n} I_{i+m, j+n} \, K_{m,n}$	Patch sum over rows columns
Image size	$\text{Width} = \frac{C_1 - C_2 + 2P}{S} + 1, \qquad \text{Height} = \frac{R_1 - R_2 + 2P}{S} + 1$	Apply size rule twice
RGB sum	$O_{i,j} = \sum_{c} \sum_{m} \sum_{n} I_{i+m, j+n, c} \, K_{m,n,c} + b$	Sum channels plus bias
Sharing gain	$\text{Dense } (9 \times 4) + 4 = 40, \qquad \text{CNN } 1 \times 4 = 4$	Sharing cuts forty to four
Pooling	$y = \max(x_1, x_2, x_3, x_4), \qquad y = \frac{x_1 + x_2 + x_3 + x_4}{4}$	Max keeps peak, average smooths

LeNet, AlexNet & embeddings Search on YouTube

Topic	Formula	Meaning
Sigmoid	$\sigma(z) = \frac{1}{1 + e^{-z}}$	Squashes unit sum to 0-1
Output size	$H_{out} = \frac{H_{in} - F + 2P}{S} + 1$	Tracks conv map height per layer
ReLU	$r(z) = \max(0, z)$	Keeps positives, zeroes negatives
Param count	$P = (F_h \times F_w \times D_{in} + 1) \times K$	Counts kernels plus biases
Euclidean distance	$d(\mathbf{a}, \mathbf{b}) = \sqrt{\sum_{i=1}^{n}(a_i - b_i)^2}$	Ranks image similarity by distance

RNN & backprop through time Search on YouTube

Topic	Formula	Meaning
RNN hidden	$h_t = \sigma(U x_t + W h_{t-1} + b)$ , $\hat{y}_t = O_t(V h_t + c)$	Shared weights update state, predict output
Total loss	$L = \sum_{t=1}^{T} L_t$ ; squared or cross-entropy per step	Adds step losses over sequence
BPTT sum	$\frac{\partial L}{\partial W} = \sum_{t=1}^{T} \frac{\partial L_t}{\partial W}$	Sums shared gradients over time
Gradient chain	$\frac{\partial h_t}{\partial h_{t-k}} = \prod_{j=0}^{k-1} \sigma^\prime(a_{t-j})\,W$	Long product vanishes or explodes
Gradient clipping	$g \leftarrow g \cdot \min\left(1,\, \tau / \Vert g \Vert\right)$	Caps exploding gradients, keeps direction
Embedding lookup	$e = x_{\text{one-hot}}\,E \qquad (1 \times V)\,(V \times D) = (1 \times D)$	Copies embedding row, skips sparse multiply
Char prediction	$P(y = i \vert z) = e^{z_i}/\sum_j e^{z_j} \qquad L_t = -\log P(y = \text{label} \vert z)$	Softmax next char, negative log loss

LSTM, GRU & bidirectional RNN Search on YouTube

Topic	Formula	Meaning
Forget gate	$f_t = \sigma(W_f x_t + U_f h_{t-1} + b_f)$	Erases stale cell memory selectively
Cell update	$i_t = \sigma(W_i x_t + U_i h_{t-1} + b_i)$ , $\tilde{C}_t = \tanh(W_g x_t + U_g h_{t-1} + b_g)$ , $C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$	Adds gated candidate to gated memory
Output gate	$o_t = \sigma(W_o x_t + U_o h_{t-1} + b_o)$ , $h_t = o_t \odot \tanh(C_t)$	Filters cell into hidden output
GRU update	$r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$ , $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$ , $\tilde{h}_t = \tanh(W_g x_t + U_g (r_t \odot h_{t-1}) + b_g)$ , $h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$	Mixes past and candidate by update
Bi-LSTM merge	$y_t = [\overrightarrow{h}_t ; \overleftarrow{h}_t]$ , $y_t = W_y [\overrightarrow{h}_t ; \overleftarrow{h}_t] + b_y$	Concats forward-backward contexts, projects
Bi-RNN merges	Concat $[\overrightarrow{h}_t ; \overleftarrow{h}_t]$ ; sum; average; element max	Concat doubles size, others keep size

Probability & Bayesian networks Search on YouTube

Topic	Formula	Meaning
Joint table	$P(S, R) = P(S = s, R = r)$ for each cell	Full table holds all information
Marginals	$P(S = s) = \sum_{r} P(S = s, R = r) \qquad P(R = r) = \sum_{s} P(S = s, R = r)$	Sums joint cells along rows or columns
Conditionals	$P(S \vert R) = \frac{P(S, R)}{P(R)} \qquad P(R \vert S) = \frac{P(S, R)}{P(S)}$	Selects slice, normalizes to one
Chain rule	$P(X, Y) = P(X)\,P(Y \vert X) \qquad P(X, Y) = P(Y)\,P(X \vert Y)$	Rebuilds joint from marginal plus conditional
Independence test	$P(X \vert Y) = P(X) \quad \text{or} \quad P(Y \vert X) = P(Y)$	Knowing other changes nothing
Independent joint	$P(X, Y) = P(X)\,P(Y)$	Joint equals product when independent
BN factorization	$P(X_1, X_2, \ldots, X_n) = \prod_{i=1}^{n} P(X_i \vert \text{parents}(X_i))$	Each node conditions on parents only
Loan profile	$P(a, \neg b, c, \neg d, e) = P(a)\,P(\neg b \vert a)\,P(c \vert a, \neg b)\,P(\neg d \vert c, e)\,P(e)$	Five-variable default factorization example
Bayes posterior	$P(D \vert F, C) = \frac{P(D, F, C)}{P(F, C)}$	Posterior equals joint over symptom marginal
Disease joint	$P(D, F, C) = P(D)\,P(F \vert D)\,P(C \vert D)$	Symptoms independent given disease
Word sequence	$P(W_1 \ldots W_5) = P(W_1)\,P(W_2 \vert W_1)\,P(W_3 \vert W_1, W_2)\,P(W_4 \vert W_2, W_3)\,P(W_5 \vert W_3, W_4)$	Two-parent chain models review probability

Markov chains & Gibbs sampling Search on YouTube

Topic	Formula	Meaning
Markov property	$P(X_{t+1} = j \vert X_t = i, \ldots) = P(X_{t+1} = j \vert X_t = i)$	Future depends only on present
Transition matrix	$T_{ij} = P(X_{t+1} = j \vert X_t = i), \qquad \sum_j T_{ij} = 1$	Rows sum to one, same each step
State count	$N_{\text{states}} = 2^n$	Binary seats give power-of-two states
Distribution update	$\mu_{t+1} = \mu_t T$	Row vector times matrix evolves distribution
Stationary distribution	$\mu^* = \mu^* T$	Long-run distribution reproduces itself
Cover bound	$f(x) \le c \cdot g(x) \quad \text{for all } x$	Proposal scaled covers target everywhere
Accept chance	$P(\text{accept} \vert x) = \frac{f(x)}{c \cdot g(x)}$	Keep candidate with ratio chance
MCMC chain	$X_0 \to X_1 \to X_2 \to \cdots \to X_m \to \cdots$	Dependent draws converge to target
Gibbs X	$P(x \vert y) = \frac{P(x, y)}{P(y)}$	Sample X fixing Y, no normalizer
Gibbs Y	$P(y \vert x) = \frac{P(x, y)}{P(x)}$	Sample Y fixing X, alternate steps

RBM & generative models Search on YouTube

Topic	Formula	Meaning
Hidden activation	$a_j = c_j + \sum_i v_i\,W_{ij}, \qquad p_j = \sigma(a_j)$	Weighted sum to firing probability
Reconstruction error	$\text{error} = \frac{1}{n}\sum_{i=1}^{n}(v_i - r_i)^2$	Mean squared gap to minimum
Energy	$E(v,h) = -\sum_i b_i v_i - \sum_j c_j h_j - \sum_i\sum_j v_i\,W_{ij}\,h_j$	Low energy means high probability
Joint distribution	$P(v,h) = \frac{\exp(-E(v,h))}{Z}, \qquad Z = \sum_v\sum_h \exp(-E(v,h))$	Boltzmann distribution over all states
Conditionals	$P(h_j = 1 \vert v) = \sigma\left(c_j + \sum_i W_{ij}\,v_i\right), \qquad P(v_i = 1 \vert h) = \sigma\left(b_i + \sum_j W_{ij}\,h_j\right)$	Same weights both directions via sigmoid
Sigmoid	$\sigma(x) = \frac{1}{1 + \exp(-x)}$	Maps total input to probability
CD update	$\Delta W_{ij} = \eta\,\left(\langle v_i h_j\rangle_{\text{data}} - \langle v_i h_j\rangle_{\text{model}}\right)$	Data association minus model association

Autoencoders, VAE & GAN Search on YouTube

Topic	Formula	Meaning
Autoencoder map	$h = f(\mathbf{x}) = \sigma(W\mathbf{x} + b), \qquad \mathbf{x}' = g(h) = \sigma(W'h + c)$	Encode to bottleneck, decode back
MSE loss	$L_{\text{MSE}} = \frac{1}{n}\sum_{i=1}^{n}(x_i - x'_i)^2$	Averages squared gaps for continuous values
BCE loss	$L_{\text{BCE}} = -\frac{1}{n}\sum_{i=1}^{n}\left[y_i\log y'_i + (1 - y_i)\log(1 - y'_i)\right]$	Averages log loss for binary outputs
Latent prior	$q(z \vert \mathbf{x}) = \mathcal{N}(\mu, \sigma^2), \qquad p(z) = \mathcal{N}(0, 1)$	Learned Gaussian versus standard normal
VAE loss	$L = L_{\text{recon}} + L_{\text{KL}}$	Reconstruction plus regularization toward prior
KL term	$L_{\text{KL}} = -\frac{1}{2}\left(1 + \log\sigma^2 - \mu^2 - \sigma^2\right)$	Penalizes deviation from standard normal
Reparameterization	$z = \mu + \sigma\,\varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, 1)$	Outside noise lets gradients flow through
Bayes link	$P(Y \vert X) = \frac{P(X, Y)}{P(X)}$	Links joint, conditional, data distributions
GAN value	$\min_G \max_D V(G, D) = \mathbb{E}_{\mathbf{x}}[\log D(\mathbf{x})] + \mathbb{E}_z[\log(1 - D(G(z)))]$	Discriminator maximizes, generator minimizes value
BCE single	$L = -\left[y\log y' + (1 - y)\log(1 - y')\right]$	Single label log loss form
Convergence	$P_g(X) \approx P_{\text{data}}(X)$	Fake distribution matches real distribution

On this page