Autoencoders, VAE and GAN

This lecture compresses high-dimensional inputs with autoencoders and then builds generative models. It covers bottleneck coding, reconstruction loss, VAE sampling with KL regularization, and the GAN minimax game. After this page you can compute MSE, reparameterization, KL, and GAN value numbers by hand.

Variational Autoencoders AnimatedDeepia · VAE latent KL reparameterization

Variational Autoencoder ExplainedDataMListic · Bottleneck probabilistic encoding decoding

GANsComputerphile · Generator discriminator minimax game

GANs ExplainedDataMListic · Counterfeiter detective adversarial training

Curse of dimensionality Search on YouTube

High-dimensional data often contains redundant features. Redundant features carry almost the same information. It also contains irrelevant features. Irrelevant features add noise and carry no signal. It also contains random variation from measurement errors.

As the feature count grows, the volume of the feature space grows fast. Data points become sparse in that huge volume. Sparse data shows few examples per dimension. Models that use distance or density then struggle. kNN, clustering, and neural networks all suffer.

Fewer dimensions give smaller matrices and faster distance computations. They need less memory and less storage. They also work with simpler models. The gain is large for images, text embeddings, and genomics.

Autoencoder structure Search on YouTube

An autoencoder is a neural network that copies its input to its output through a narrow middle. It has three parts: encoder, bottleneck layer, and decoder. The bottleneck layer (also called code) holds the latent vector. After this point the page uses only "latent vector" and "bottleneck layer".

The encoder maps the input to the latent vector. The bottleneck layer stores that compressed form. The decoder rebuilds the input from it.

Think of a summary note. You compress a long chapter into half a page. Then a friend rebuilds the chapter from your note alone.

Encoder maps n inputs to a narrow latent vector. Decoder rebuilds n outputs from it.

Encoder, bottleneck, and decoder Search on YouTube

The encoder transforms input $\mathbf{x}$ of $n$ dimensions into $h = f(\mathbf{x})$ . The vector $h$ lies in a reduced latent space. The bottleneck layer is that compressed form. It keeps the most salient features in fewer dimensions. The decoder maps it back as $\mathbf{x}' = g(h)$ . Encoding layers can be dense or convolutional. The choice depends on the input type and the task.

h = f(\mathbf{x}) = \sigma(W\mathbf{x} + b), \qquad \mathbf{x}' = g(h) = \sigma(W'h + c)

$W$ = encoder weight matrix. $b$ = encoder bias vector. $W'$ = decoder weight matrix. $c$ = decoder bias vector. $\sigma$ = activation function (sigmoid, ReLU, or tanh). $h$ = latent vector at the bottleneck layer.

Method	Mapping	Handles nonlinear data
PCA	Linear projection only	No
Autoencoder	Weighted sums plus activation functions	Yes

PCA applies only linear transformations. It fails on complex nonlinear distributions. Autoencoders apply activation functions to weighted inputs. They learn nonlinear mappings and richer low-dimensional codes.

Worked example. Image input $28 \times 28 = 784$ dims. Bottleneck layer has $32$ units. Weights use float32 (4 bytes each).

Compression ratio.

\begin{aligned} \text{input dims} &= 28 \times 28 = \mathbf{784}\\ \text{latent dims} &= \mathbf{32}\\ \text{ratio} &= 784/32 = \mathbf{24.5} \end{aligned}

Memory per sample.

\begin{aligned} \text{input bytes} &= 784 \times 4 = \mathbf{3136}\\ \text{latent bytes} &= 32 \times 4 = \mathbf{128} \end{aligned}

Encoder size.

\begin{aligned} W &: 32 \times 784 = \mathbf{25088} \text{ weights}\\ b &: \mathbf{32} \text{ biases} \end{aligned}

The bottleneck layer stores 24.5 times fewer numbers than the input.

Reconstruction loss Search on YouTube

The decoder aims to make $\mathbf{x}'$ match $\mathbf{x}$ . Reconstruction loss measures that gap. Training minimizes it over the training set. The slides name two loss functions. The choice depends on the data type.

L_{\text{MSE}} = \frac{1}{n}\sum_{i=1}^{n}(x_i - x'_i)^2

$x_i$ = true value of feature $i$ . $x'_i$ = reconstructed value. $n$ = input dimension. Use it for continuous values such as pixel intensities.

L_{\text{BCE}} = -\frac{1}{n}\sum_{i=1}^{n}\left[y_i\log y'_i + (1 - y_i)\log(1 - y'_i)\right]

$y_i$ = true binary value (0 or 1). $y'_i$ = predicted probability in $(0, 1)$ . Use it for binary or probability outputs.

Compression, inpainting, and denoising Search on YouTube

Autoencoders compress data for transmission over a network. The encoder sends only the latent vector. That step cuts bandwidth needs. The decoder at the receiving end rebuilds the input, for example a downscaled image. They also restore damaged inputs. In inpainting, the model fills missing patches. In denoising, it removes noise from corrupted inputs.

Worked example. True vector $\mathbf{x} = [0.8, 0.2, 0.5, 0.9]$ . Reconstruction $\mathbf{x}' = [0.7, 0.3, 0.4, 1.0]$ .

Squared gaps.

\begin{aligned} (0.8 - 0.7)^2 &= 0.01\\ (0.2 - 0.3)^2 &= 0.01\\ (0.5 - 0.4)^2 &= 0.01\\ (0.9 - 1.0)^2 &= 0.01 \end{aligned}

Mean.

\begin{aligned} L_{\text{MSE}} &= (0.01 + 0.01 + 0.01 + 0.01)/4\\ &= 0.04/4 = \mathbf{0.01} \end{aligned}

Exam shortcut: for MSE, square each gap first and keep four decimals. Then add and divide by $n$ . For BCE with $y = 1$ , the term is $-\log y'$ . For $y = 0$ , the term is $-\log(1 - y')$ .

Limit of deterministic autoencoders Search on YouTube

A standard autoencoder maps each input to one fixed point $h = f(\mathbf{x})$ . Decoding that point always gives the same output. It cannot create diverse new samples. It also learns no smooth latent space. Nearby latent vectors need not decode to similar outputs. It does not model the probability distribution of the data. Its deterministic nature gives poor generative ability.

Variational autoencoders Search on YouTube

A variational autoencoder (VAE) maps each input to a distribution. For each latent dimension, the encoder outputs two parts: a mean $\mu$ part and a standard deviation $\sigma$ part. They define a Gaussian distribution. A sample $z$ from that distribution feeds the decoder.

Replace a fixed address with a delivery zone. Each input marks a zone (mean) with a size (spread). Each visit picks a random door inside the zone.

Because codes are probabilistic, nearby latent vectors can decode to similar outputs. Two close face codes may show visually similar faces. A VAE can shift a neutral face toward a smile. But it does not know the smile direction by itself. That direction must be learned or found by search.

VAE encoder outputs Search on YouTube

q(z \vert \mathbf{x}) = \mathcal{N}(\mu, \sigma^2), \qquad p(z) = \mathcal{N}(0, 1)

$q(z \vert \mathbf{x})$ = learned latent distribution for input $\mathbf{x}$ . $\mu$ = encoder mean output per latent dimension. $\sigma$ = encoder standard deviation output per latent dimension. $p(z)$ = prior, usually standard normal.

Do not confuse the two uses of $\sigma$ . In $h = \sigma(W\mathbf{x} + b)$ it is the activation function. In $\mathcal{N}(\mu, \sigma^2)$ it is the standard deviation. Read the symbol from context.

VAE loss Search on YouTube

The VAE loss has two terms. The first rebuilds the input. The second pulls the learned distribution toward the prior.

L = L_{\text{recon}} + L_{\text{KL}}

$L_{\text{recon}}$ = reconstruction loss (MSE or BCE between $\mathbf{x}$ and $\mathbf{x}'$ ). $L_{\text{KL}}$ = KL divergence loss between $q(z \vert \mathbf{x})$ and $p(z)$ .

The KL term measures the gap between $q(z \vert \mathbf{x}) = \mathcal{N}(\mu, \sigma^2)$ and the prior $p(z)$ . It regularizes the latent space. The space then stays close to standard normal. That property allows smooth interpolation and sampling of new faces. For one Gaussian dimension against $\mathcal{N}(0,1)$ :

L_{\text{KL}} = -\frac{1}{2}\left(1 + \log\sigma^2 - \mu^2 - \sigma^2\right)

$\mu$ = learned mean. $\sigma$ = learned standard deviation. $\log$ = natural log. Sum over latent dimensions for a vector code.

R plus K: Reconstruction plus KL

Reconstruction rebuilds the input. KL pulls the code toward the prior. Memorize loss = R $+$ K.

Worked example. One latent dimension with $\mu = 0.5$ , $\sigma = 0.3$ .

Squares and log.

\begin{aligned} \sigma^2 &= 0.3^2 = 0.09\\ \mu^2 &= 0.5^2 = 0.25\\ \log\sigma^2 &= \log 0.09 = -2.4079 \end{aligned}

KL value.

\begin{aligned} L_{\text{KL}} &= -\tfrac{1}{2}(1 + (-2.4079) - 0.25 - 0.09)\\ &= -\tfrac{1}{2}(1 - 2.4079 - 0.34)\\ &= -\tfrac{1}{2}(-1.7479) = \mathbf{0.8740} \end{aligned}

Recheck: $1 - 2.4079 = -1.4079$ . Then $-1.4079 - 0.25 - 0.09 = -1.7479$ . Half of $1.7479$ is $0.8740$ .

Reparameterization trick Search on YouTube

Sampling $z \sim \mathcal{N}(\mu, \sigma^2)$ blocks gradients. The sampling node is stochastic. Backpropagation cannot pass through it. The reparameterization trick rewrites sampling with outside noise. Gradients then flow through $\mu$ and $\sigma$ .

z = \mu + \sigma\,\varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, 1)

$z$ = latent sample fed to the decoder. $\mu, \sigma$ = encoder outputs (learned). $\varepsilon$ = noise from standard normal (not learned). Gradients flow through $\mu$ and $\sigma$ around the noise.

Worked example. Encoder gives $\mu = 0.5$ , $\sigma = 0.3$ . Draw gives $\varepsilon = 1.2$ .

Sample.

\begin{aligned} z &= \mu + \sigma\,\varepsilon\\ &= 0.5 + 0.3 \times 1.2\\ &= 0.5 + 0.36 = \mathbf{0.86} \end{aligned}

Two-dimensional check. $\mu = [0.5, -0.4]$ , $\sigma = [0.3, 0.2]$ , $\varepsilon = [1.2, -0.5]$ .

\begin{aligned} z_1 &= 0.5 + 0.3 \times 1.2 = \mathbf{0.86}\\ z_2 &= -0.4 + 0.2 \times (-0.5) = -0.4 - 0.10 = \mathbf{-0.50} \end{aligned}

Recheck: $0.3 \times 1.2 = 0.36$ , plus $0.5$ gives $0.86$ . And $0.2 \times 0.5 = 0.10$ , so $-0.4 - 0.10 = -0.50$ .

Exam wording: "gradients flow through $\mu$ and $\sigma$ " gets full credit. Add that $\varepsilon$ needs no training. Then write $z = \mu + \sigma\varepsilon$ with $\varepsilon \sim \mathcal{N}(0,1)$ .

Generative models and GAN roles Search on YouTube

A discriminative model learns $P(Y \vert X)$ . It gives the label $Y$ for input $X$ . It learns the decision boundary. Logistic regression, support vector machines, and neural network classifiers are examples. They serve classification and regression.

A generative model learns the joint distribution $P(X, Y)$ . It can also learn the data distribution $P(X)$ . It can then create new points like the training data. Naive Bayes, hidden Markov models, VAEs, and GANs are examples.

P(Y \vert X) = \frac{P(X, Y)}{P(X)}

$P(Y \vert X)$ = conditional probability of label $Y$ given input $X$ . $P(X, Y)$ = joint distribution learned by generative models. $P(X)$ = data distribution. Bayes' theorem links the two views.

Generator and discriminator Search on YouTube

A GAN holds a generator ( $G$ ) and a discriminator ( $D$ ). Both are neural networks. They train together in a minimax game. The generator ( $G$ ) takes random noise and makes fake data. The discriminator ( $D$ ) is a binary classifier. It separates real training data from fake generator data.

The true distribution $P_{\text{data}}(X)$ is high-dimensional and complex. The generator takes noise $Z$ from a simple known $P_Z(Z)$ . Its weights $\theta_g$ map noise to samples $G(Z, \theta_g)$ . Those outputs follow $P_g(X)$ , the fake distribution. The goal is $P_g(X) \approx P_{\text{data}}(X)$ .

The generator never sees real data directly. It learns only through the discriminator feedback signal. Over many epochs, its noise-to-sample map improves. At the end, its samples look real.

G fakes, D judges

Generator fakes data from noise. Discriminator judges real against fake. Training ends when the judge guesses at chance.

Noise enters G. Fake samples and real samples enter D. D outputs a real-or-fake score.

GAN value function Search on YouTube

The discriminator outputs a probability. The value is the chance that the input is real. $D$ maximizes correct real/fake calls. $G$ minimizes the ability of $D$ to detect fakes. That conflict gives the value function $V(G, D)$ .

\min_G \max_D V(G, D) = \mathbb{E}_{\mathbf{x}}[\log D(\mathbf{x})] + \mathbb{E}_z[\log(1 - D(G(z)))]

$\mathbf{x} \sim P_{\text{data}}$ = real sample. $z \sim P_Z$ = noise sample. $D(\mathbf{x})$ = score that real input is real. $D(G(z))$ = score that fake input is real. $\mathbb{E}$ = mean over the named distribution.

It matches binary cross-entropy. For one data point:

L = -\left[y\log y' + (1 - y)\log(1 - y')\right]

$y$ = true label ( $1$ = real, $0$ = fake). $y'$ = discriminator score. For real data $y = 1$ and $y' = D(\mathbf{x})$ . For fake data $y = 0$ and $y' = D(G(z))$ .

Averaging both groups gives the same form as $-V(G, D)$ up to the factor of two. $D$ wants large $V$ . $G$ wants small $V$ .

Worked example. One real sample scores $D(\mathbf{x}) = 0.8$ . One fake sample scores $D(G(z)) = 0.3$ .

GAN value.

\begin{aligned} \log D(\mathbf{x}) &= \log 0.8 = -0.2231\\ \log(1 - D(G(z))) &= \log 0.7 = -0.3567\\ V &= -0.2231 + (-0.3567) = \mathbf{-0.5798} \end{aligned}

BCE link.

\begin{aligned} L_{\text{real}} &= -\log 0.8 = 0.2231\\ L_{\text{fake}} &= -\log(1 - 0.3) = -\log 0.7 = 0.3567\\ \text{mean} &= (0.2231 + 0.3567)/2 = \mathbf{0.2899} = -V/2 \end{aligned}

Recheck: $-\log 0.8 = 0.2231$ and $-\log 0.7 = 0.3567$ . Sum is $0.5798$ . Half is $0.2899$ . And $-(-0.5798)/2 = 0.2899$ .

Alternating training Search on YouTube

Training alternates two steps. Each step freezes one network.

Freeze $G$ . Update the discriminator with gradient ascent on $V$ . Use a minibatch of $m$ real and $m$ fake samples.
Freeze $D$ . Update the generator with gradient descent on $V$ . This step raises $D(G(z))$ for fresh fakes.
Repeat until $P_g(X) \approx P_{\text{data}}(X)$ . Then $D$ cannot separate real from fake.

P_g(X) \approx P_{\text{data}}(X)

$P_g(X)$ = distribution of generator outputs. $P_{\text{data}}(X)$ = true training distribution. Training stops when the two match and $D$ guesses at chance.

Worked example. Trace two updates from $V = -0.5798$ above.

D update (freeze G). $D(\mathbf{x})$ rises $0.8 \to 0.9$ . Fake score falls $0.3 \to 0.2$ .

\begin{aligned} V_{\text{new}} &= \log 0.9 + \log(1 - 0.2)\\ &= \log 0.9 + \log 0.8\\ &= -0.1054 + (-0.2231) = \mathbf{-0.3285} \end{aligned}

G update (freeze D). Generator improves. Fake score rises $0.2 \to 0.6$ .

\begin{aligned} V_{\text{new}} &= \log 0.9 + \log(1 - 0.6)\\ &= \log 0.9 + \log 0.4\\ &= -0.1054 + (-0.9163) = \mathbf{-1.0217} \end{aligned}

$V$ rose under the $D$ step ( $-0.5798 \to -0.3285$ ). It fell under the $G$ step ( $-0.3285 \to -1.0217$ ). That rise-then-fall is the minimax pattern.

Do not swap the update directions. $D$ uses gradient ascent because it maximizes $V$ . $G$ uses gradient descent because it minimizes $V$ . Keep $G$ fixed during the $D$ step.

Train one GAN round by hand in the lab. Freeze G to raise V, then freeze D to lower it. Then move to the VAE tab and push the code toward the prior.

Bars show discriminator scores for real and fake samples. The dashed line marks the judge threshold.

V = log 0.8 + log 0.7 = -0.2231 + (-0.3567) = -0.5798 | judge accuracy 100%

Quick self-test Search on YouTube

1. An input has

48 \times 48

pixels and a bottleneck layer of

64

units. Find the compression ratio and the latent memory in float32.

Input has $48 \times 48 = 2304$ dims. Ratio is $2304/64 = 36$ . Latent memory is $64 \times 4 = 256$ bytes against $2304 \times 4 = 9216$ bytes.

2. True values are

[0.6, 0.4]

and reconstructions are

[0.5, 0.6]

. Compute the MSE.

Squared gaps are $0.01$ and $0.04$ . MSE is $(0.01 + 0.04)/2 = 0.025$ .

3. Differentiate PCA and autoencoders for dimensionality reduction.

PCA uses only linear projections. Autoencoders add activation functions to weighted sums. So autoencoders capture nonlinear structure and PCA does not.

4. Encoder gives

\mu = -0.2

\sigma = 0.5

, and draw

\varepsilon = 0.8

. Find the latent sample

z

$z = -0.2 + 0.5 \times 0.8 = -0.2 + 0.4 = 0.2$ .

5. Encoder gives

\mu = 0

\sigma = 1

. Compute the single-dimension KL against standard normal.

$\sigma^2 = 1$ , $\mu^2 = 0$ , $\log 1 = 0$ . KL is $-0.5(1 + 0 - 0 - 1) = 0$ . The learned code already matches the prior.

6. Why does VAE training need the reparameterization trick?

Direct sampling blocks backpropagation at the stochastic node. The trick writes $z = \mu + \sigma\varepsilon$ . Gradients then flow through $\mu$ and $\sigma$ while $\varepsilon$ stays fixed noise.

7. Real score

D(\mathbf{x}) = 0.9

and fake score

D(G(z)) = 0.1

. Compute

V

for this pair.

$V = \log 0.9 + \log 0.9 = -0.1054 + (-0.1054) = -0.2108$ . Both terms agree here because $1 - 0.1 = 0.9$ .

8. Differentiate the generator and the discriminator. Which one sees real data?

The generator fakes data from noise and never sees real samples. The discriminator judges real against fake and sees both. Only the discriminator touches real data directly.

9. Why does a standard autoencoder generate poorly while a VAE can generate?

A standard autoencoder stores one fixed point per input with no smooth space. The VAE regularizes codes toward $\mathcal{N}(0,1)$ with the KL term. Sampling that smooth space gives new valid outputs.

10. In one GAN round,

V

moves from

-0.58

-0.33

and then to

-1.02

. Which step belongs to

D

and which to

G

The rise to $-0.33$ belongs to the $D$ step (gradient ascent on $V$ ). The fall to $-1.02$ belongs to the $G$ step (gradient descent on $V$ ).

High dimensions cause sparsity: volume grows fast and distance-based models struggle.
PCA is linear only. Autoencoders add activation functions and learn nonlinear codes.
Encoder maps $\mathbf{x}$ to $h = f(\mathbf{x})$ . Bottleneck layer stores the latent vector. Decoder rebuilds $\mathbf{x}' = g(h)$ .
Example shrinks $784$ dims to $32$ (ratio $24.5$ , $3136$ bytes to $128$ bytes).
Reconstruction loss is MSE for continuous values and BCE for binary or probability values.
MSE example $[0.8, 0.2, 0.5, 0.9]$ against $[0.7, 0.3, 0.4, 1.0]$ gives $\mathbf{0.01}$ .
Uses: compression for transmission, inpainting of missing parts, denoising of corrupted inputs.
Standard autoencoders are deterministic and generate poorly.
VAE outputs $\mu$ and $\sigma$ , samples $z$ , and keeps a smooth latent space.
VAE loss = R $+$ K: reconstruction plus KL against the prior.
Reparameterization $z = \mu + \sigma\varepsilon$ lets gradients flow through $\mu$ and $\sigma$ .
KL example $\mu = 0.5$ , $\sigma = 0.3$ gives $\mathbf{0.8740}$ .
GAN: G fakes from noise, D judges real against fake, with $P_g(X) \approx P_{\text{data}}(X)$ at convergence.
Value $\min_G \max_D V = \mathbb{E}[\log D(\mathbf{x})] + \mathbb{E}[\log(1 - D(G(z)))]$ matches BCE.
GAN pair $0.8$ and $0.3$ gives $V = \mathbf{-0.5798}$ and mean BCE $\mathbf{0.2899}$ .
Train alternately: $D$ ascends $V$ with $G$ fixed, $G$ descends $V$ with $D$ fixed.

Curse of dimensionalitySearch on YouTubeBookmarkSaved

Autoencoder structureSearch on YouTubeBookmarkSaved

Encoder, bottleneck, and decoderSearch on YouTubeBookmarkSaved

Reconstruction lossSearch on YouTubeBookmarkSaved

Compression, inpainting, and denoisingSearch on YouTubeBookmarkSaved

Limit of deterministic autoencodersSearch on YouTubeBookmarkSaved

Variational autoencodersSearch on YouTubeBookmarkSaved

VAE encoder outputsSearch on YouTubeBookmarkSaved

VAE lossSearch on YouTubeBookmarkSaved

Reparameterization trickSearch on YouTubeBookmarkSaved

Generative models and GAN rolesSearch on YouTubeBookmarkSaved

Generator and discriminatorSearch on YouTubeBookmarkSaved

GAN value functionSearch on YouTubeBookmarkSaved

Alternating trainingSearch on YouTubeBookmarkSaved

Quick self-testSearch on YouTubeBookmarkSaved

On this page