RBM and generative models

This lecture shows how generative models learn the data distribution and create new samples. It then builds the Restricted Boltzmann Machine from the full Boltzmann Machine. After this page you can compute RBM energies, hidden probabilities and one contrastive-divergence update by hand.

Restricted Boltzmann MachinesJeff Orchard · Energy Gibbs generative model

Contrastive Divergence RBMsSuper Data Science · Contrastive divergence weight updates

Generative models Search on YouTube

A generative model learns the underlying distribution of the data. After training, it creates new samples that resemble the training data. The slides name four examples: GANs, autoencoders, variational encoders and RBMs. Typical uses are realistic images, text generation, music or speech synthesis, data augmentation and image-to-image translation.

A discriminative model learns the boundary between classes. It maps an input to a label. A generative model learns how the data is distributed. An RBM is generative. It learns a probability distribution over the inputs. Then it reconstructs inputs instead of predicting a label.

A judge sorts photos into boxes. That is discriminative. An artist studies photos and draws new ones. That is generative.

Type	Learns	Answers	Example
Generative	distribution of the data	what a new real sample looks like	RBM, GAN, variational encoder
Discriminative	boundary between classes	which label fits this input	feedforward classifier

Boltzmann machines Search on YouTube

A Boltzmann Machine (BM) is a stochastic neural network. Stochastic means probabilistic: units fire with a probability. It learns complex probability distributions over its inputs. It has visible units $v$ and hidden units $h$ . Visible units hold the data. Hidden units capture latent variables.

Every unit connects to every other unit. There are no self-connections. All connections are symmetric. The network gives low energy to training patterns. Low energy means high probability. Other patterns get high energy and low probability.

Why full Boltzmann machines fail Search on YouTube

Training a full BM is computationally expensive. Every neuron connects to every other neuron. Gradients need sums over all states of visible and hidden units. The state count grows exponentially. With $n$ binary units there are $2^n$ states. So 10 units need 1024 sums, while 100 units need $2^{100}$ sums. Full BMs are impractical for real datasets. This problem led to the Restricted Boltzmann Machine.

Restricted Boltzmann machines Search on YouTube

An RBM is a generative stochastic network with two layers. The first layer is the visible (input) layer. The second layer is the hidden layer. RBMs are shallow two-layer nets. They form the building blocks of deep-belief networks.

Each node is a locus of computation. It processes its input and makes a stochastic decision about transmission. The restriction is strict. No two nodes of the same layer share a link. There is no intra-layer communication. Inputs from all visible nodes reach all hidden nodes. So an RBM is a symmetric bipartite graph. Symmetric means each visible node links to each hidden node. Bipartite means the graph has two parts (layers). The same weight $w_{ij}$ works in both directions.

RBMs help with dimensionality reduction, classification, regression, collaborative filtering, feature learning and topic modelling.

Images hide information that sits outside the pixels. A blue-heavy photo may come from a clear daytime sky. The sky is hidden, yet it explains the pixels. Hidden units model such latent variables. Visible units hold the observed pixel intensities. Joint modelling finds latent features for reconstruction, generation and feature extraction.

RBM as a symmetric bipartite graph. Every visible unit links to every hidden unit. No link sits inside one layer.

B-S-G: Bipartite, Symmetric, Generative

Bipartite: two layers with no intra-layer links. Symmetric: one weight $w_{ij}$ serves both directions. Generative: the net models the joint distribution and reconstructs inputs.

Do not add links inside one layer. Intra-layer links destroy the conditional independence. Without it, Gibbs sampling turns hard.

Forward and backward passes Search on YouTube

The forward pass finds the hidden activation pattern. Each hidden node combines several inputs. Each input is multiplied by a separate weight. The products are summed and added to a bias. An activation function turns the sum into the output.

The backward pass reconstructs the input. It starts from a hidden pattern and rebuilds a visible pattern. Symmetric weights carry the signal back.

a_j = c_j + \sum_i v_i\,W_{ij}, \qquad p_j = \sigma(a_j)

$a_j$ is the total input to hidden unit $j$ . $c_j$ is the hidden bias. $v_i$ is visible unit $i$ . $W_{ij}$ is the weight between them. $\sigma$ is the sigmoid activation function. $p_j$ is the activation probability of hidden unit $j$ .

Random initial weights make early reconstructions poor. The gap between the reconstruction $r$ and the input $v$ is the training signal. Each iteration pushes the weights against this gap. The error falls until it reaches a minimum.

\text{error} = \frac{1}{n}\sum_{i=1}^{n}(v_i - r_i)^2

$v_i$ is input unit $i$ . $r_i$ is its reconstruction. $n$ is the count of visible units. Training repeats until this reconstruction loss reaches a minimum.

Energy and joint probability Search on YouTube

Notation first. Visible units are $v = (v_1, v_2, \dots, v_n)$ . Hidden units are $h = (h_1, h_2, \dots, h_m)$ . Weights $W$ link the layers, and $W_{ij}$ links $v_i$ with $h_j$ . Visible bias $b_i$ sits on $v_i$ . Hidden bias $c_j$ sits on $h_j$ .

Picture a ball on a hilly surface. Valleys trap the ball, peaks repel it. Training digs valleys (low energy) where real data sits.

E(v,h) = -\sum_i b_i v_i - \sum_j c_j h_j - \sum_i\sum_j v_i\,W_{ij}\,h_j

$E(v,h)$ is the energy of the joint state. Aligned visible and hidden units lower it through $-v_i W_{ij} h_j$ . Matching biases lower it further. Low energy means high probability.

P(v,h) = \frac{\exp(-E(v,h))}{Z}, \qquad Z = \sum_v\sum_h \exp(-E(v,h))

$P(v,h)$ is the joint probability of the state. $Z$ is the partition function. It sums over all visible and hidden states. It normalises the distribution. Its size is why exact training is expensive.

Worked example. One tiny RBM: two visible units, one hidden unit. Weights $W_{11} = 0.6$ and $W_{21} = -0.4$ . Biases $b = (0.2, -0.2)$ and $c_1 = -0.1$ . Compare a data-like state $A$ with $v = (1,0)$ , $h = (1)$ against an unlikely state $B$ with $v = (0,1)$ , $h = (1)$ .

Energy of state A.

\begin{aligned} E_A &= -(0.2\cdot1 + (-0.2)\cdot0) - (-0.1\cdot1) - (1\cdot0.6\cdot1 + 0\cdot(-0.4)\cdot1)\\ &= -0.2 + 0.1 - 0.6\\ &= \mathbf{-0.7} \end{aligned}

Energy of state B.

\begin{aligned} E_B &= -(0.2\cdot0 + (-0.2)\cdot1) - (-0.1\cdot1) - (0\cdot0.6\cdot1 + 1\cdot(-0.4)\cdot1)\\ &= 0.2 + 0.1 + 0.4\\ &= \mathbf{0.7} \end{aligned}

Probability ratio. $Z$ cancels in the ratio.

\begin{aligned} \frac{P_A}{P_B} &= \frac{\exp(0.7)}{\exp(-0.7)} = \exp(1.4)\\ &\approx \mathbf{4.06} \end{aligned}

State $A$ carries low energy and state $B$ carries high energy. So the data-like state is about four times more likely.

Conditional probabilities Search on YouTube

With no intra-layer links, hidden units turn independent given the visible layer. The reverse holds too. Each conditional is a sigmoid of the weighted sum. These two formulas drive both passes and all sampling.

P(h_j = 1 \vert v) = \sigma\left(c_j + \sum_i W_{ij}\,v_i\right), \qquad P(v_i = 1 \vert h) = \sigma\left(b_i + \sum_j W_{ij}\,h_j\right)

The hidden conditional uses the visible state $v$ . The visible conditional uses the hidden state $h$ . Both share the same weights $W_{ij}$ .

\sigma(x) = \frac{1}{1 + \exp(-x)}

$\sigma$ is the sigmoid activation function. It maps any total input to a probability between 0 and 1.

Compute the weighted sum once per direction. The forward pass reads the sum bottom-up. The backward pass reads the same weights top-down.

Worked example. Same tiny RBM: $W_{11} = 0.6$ , $W_{21} = -0.4$ , $b = (0.2, -0.2)$ , $c_1 = -0.1$ . Input $v = (1, 0)$ . First find the hidden probability, then reconstruct the visible layer from a sampled $h_1 = 1$ .

Forward pass. Total input to the hidden unit:

\begin{aligned} a_1 &= -0.1 + 0.6\cdot1 + (-0.4)\cdot0 = 0.5\\ P(h_1 = 1 \vert v) &= \sigma(0.5) = \frac{1}{1 + \exp(-0.5)}\\ &= \frac{1}{1 + 0.6065} = \frac{1}{1.6065}\\ &\approx \mathbf{0.622} \end{aligned}

Backward pass. Clamp $h_1 = 1$ and rebuild each visible unit:

\begin{aligned} P(v_1 = 1 \vert h) &= \sigma(0.2 + 0.6\cdot1) = \sigma(0.8)\\ &= \frac{1}{1 + 0.4493} \approx \mathbf{0.690}\\ P(v_2 = 1 \vert h) &= \sigma(-0.2 + (-0.4)\cdot1) = \sigma(-0.6)\\ &= 1 - \sigma(0.6) = 1 - 0.6457 \approx \mathbf{0.354} \end{aligned}

The reconstruction $(0.690, 0.354)$ resembles the input $(1, 0)$ . The first unit is likely on and the second is likely off.

Positive and negative phases Search on YouTube

The log-likelihood gradient has two parts. The positive phase raises the probability of observed data. The model notes which visible and hidden units fire together on real data. It strengthens weights between co-active patterns. The negative phase lowers the probability of false configurations. These are patterns the model likes but the data lacks. It weakens their weights.

\Delta W_{ij} = \eta\,\left(\langle v_i h_j\rangle_{\text{data}} - \langle v_i h_j\rangle_{\text{model}}\right)

$\eta$ is the learning rate. $\langle v_i h_j\rangle_{\text{data}}$ is the co-activation measured on real data. $\langle v_i h_j\rangle_{\text{model}}$ is the co-activation measured on reconstructions. Weights move toward the data and away from fantasy.

Worked example. Same tiny RBM with learning rate $\eta = 0.1$ . Training vector $v = (1, 0)$ . The hidden sample from the data is $h_1 = 1$ (drawn with probability 0.622). One Gibbs round gives reconstruction $v' = (1, 0)$ and a fresh hidden sample $h'_1 = 0$ (possible since $1 - 0.622 = 0.378$ ).

Data association (positive phase).

\begin{aligned} \langle v_1 h_1\rangle_{\text{data}} &= 1\cdot1 = \mathbf{1}\\ \langle v_2 h_1\rangle_{\text{data}} &= 0\cdot1 = \mathbf{0} \end{aligned}

Model association (negative phase).

\begin{aligned} \langle v_1 h_1\rangle_{\text{model}} &= 1\cdot0 = \mathbf{0}\\ \langle v_2 h_1\rangle_{\text{model}} &= 0\cdot0 = \mathbf{0} \end{aligned}

Weight update.

\begin{aligned} \Delta W_{11} &= 0.1\cdot(1 - 0) = \mathbf{0.1}, &\quad W_{11}: 0.6 \to \mathbf{0.7}\\ \Delta W_{21} &= 0.1\cdot(0 - 0) = \mathbf{0}, &\quad W_{21} \text{ stays at } \mathbf{-0.4} \end{aligned}

Check: with $W_{11} = 0.7$ the forward total is $-0.1 + 0.7 = 0.6$ . Then $\sigma(0.6) \approx 0.646$ , above 0.622. The data state gains probability, as wanted.

Contrastive divergence and Gibbs sampling Search on YouTube

Exact gradients need sums over all states. Large models make the negative phase too complex. Contrastive divergence (CD) approximates the log-likelihood gradient. It uses Markov Chain Monte Carlo (MCMC) sampling. Gibbs sampling is the MCMC method used here. It samples a complex joint through its conditionals. Direct sampling is hard, but one-sided conditionals are easy. Once trained, the net generates data by sampling the learned distribution.

Follow these steps for CD with $k$ Gibbs rounds (CD- $k$ ):

Clamp a training vector on the visible layer.
Sample the hidden layer from $P(h \vert v)$ .
Reconstruct the visible layer from $P(v \vert h)$ , then resample hidden. Repeat $k$ times.
Update weights with data association minus model association.

D-R-U: Data, Reconstruction, Update

Clamp the Data and read hidden activations. Reconstruct by Gibbs sampling. Update with data association minus model association.

CD with one Gibbs round (CD-1) already trains well in practice. Start chains from the data, not from noise. Few rounds keep each update cheap.

Do not stop at a large initial error. Random weights always start far from the input. The reconstruction loss falls over iterations.

RBM versus feedforward networks Search on YouTube

Aspect	RBM	Feedforward network
Graph	bipartite and undirected	layered and feedforward
Signal flow	both directions ( $v$ to $h$ and $h$ to $v$ )	one direction (input to output)
Goal	model the joint $P(v,h)$	map input to output
Type	generative	discriminative
Learns	reconstruct inputs	predict labels
Training	contrastive divergence with Gibbs sampling	gradient descent with backpropagation

Stacked RBMs and pretraining Search on YouTube

Two RBM layers can sit inside a deeper network. Hidden layer 1 outputs feed hidden layer 2. The chain continues to a final classifying layer. Each RBM learns one layer at a time:

Train RBM1 on the raw input with contrastive divergence.
Freeze the weights of RBM1.
Pass the data through RBM1 and read hidden activations.
Train RBM2 on these activations as its input.
Repeat for each deeper layer.

Freezing matters. It locks learned features before the next layer trains. Each stage then models its own input distribution. Stacked RBMs pretrain deep-belief networks this way.

Quick self-test Search on YouTube

1. Visible vector

v = (1, 1)

, weights

(0.4, 0.4)

, hidden bias

0

. Find

P(h = 1 \vert v)

Total $a = 0 + 0.4 + 0.4 = 0.8$ . Then $\sigma(0.8) = 1/(1 + 0.4493) \approx \mathbf{0.690}$ .

2. Hidden state

h = 1

, weight

0.4

, visible bias

0.1

. Find

P(v_1 = 1 \vert h)

Total $a = 0.1 + 0.4 = 0.5$ . Then $\sigma(0.5) = 1/(1 + 0.6065) \approx \mathbf{0.622}$ .

3. State A has energy

-0.7

and state B has energy

0.7

. How much more likely is A than B?

The ratio is $\exp(0.7)/\exp(-0.7) = \exp(1.4) \approx \mathbf{4.06}$ . State A is about four times more likely. $Z$ cancels in the ratio.

4. Learning rate

\eta = 0.5

, data association

1

, model association

0.25

, old weight

0.6

. Find the update and the new weight.

$\Delta W = 0.5 \cdot (1 - 0.25) = 0.5 \cdot 0.75 = \mathbf{0.375}$ . New weight $= 0.6 + 0.375 = \mathbf{0.975}$ .

5. Differentiate generative and discriminative models.

A generative model learns the data distribution and creates new samples. A discriminative model learns class boundaries and maps inputs to labels. An RBM is generative: it reconstructs inputs.

6. Differentiate a Boltzmann Machine and a Restricted Boltzmann Machine.

A BM links every unit to every other unit, including links inside a layer. An RBM links only across layers, with no intra-layer links. The RBM is a symmetric bipartite graph, so training stays tractable.

7. Why does training a full Boltzmann Machine fail on real datasets?

Gradients need sums over all visible and hidden states. The count grows as $2^n$ for $n$ binary units. Full BMs are therefore impractical beyond tiny sizes.

8. Why does training need both a positive and a negative phase?

The positive phase raises the probability of real data. The negative phase lowers the probability of false configurations. Without the negative phase, the model would spread high probability over unreal states too.

9. Differentiate the forward and backward passes of an RBM.

The forward pass maps visible input to a hidden activation pattern. The backward pass rebuilds visible input from a hidden pattern. Both use the same symmetric weights.

10. State the stacked RBM pretraining order. Why freeze each RBM?

Train RBM1 on raw input, freeze it, pass data through it, then train RBM2 on the activations. Repeat deeper. Freezing locks learned features, so each layer models a fixed input distribution.

A generative model learns the data distribution and creates new samples. A discriminative model maps inputs to labels.
A BM has visible units (data) and hidden units (latent variables) with symmetric all-to-all links and no self-connections.
Full BMs fail since gradients sum over $2^n$ states, which grows exponentially.
An RBM is a symmetric bipartite graph: no intra-layer links, one weight $w_{ij}$ per direction pair.
The forward pass finds hidden activations; the backward pass reconstructs inputs with the same weights.
Energy $E(v,h) = -\sum b_i v_i - \sum c_j h_j - \sum\sum v_i W_{ij} h_j$ ; joint $P(v,h) = \exp(-E)/Z$ .
Conditionals $P(h_j = 1 \vert v)$ and $P(v_i = 1 \vert h)$ are sigmoids of the weighted sums.
CD update: $\Delta W_{ij} = \eta(\langle v_i h_j\rangle_{\text{data}} - \langle v_i h_j\rangle_{\text{model}})$ .
Gibbs sampling alternates the two conditionals; CD- $k$ runs $k$ rounds from the data.
Stacked RBMs pretrain layer by layer: train, freeze, feed activations upward.

Generative modelsSearch on YouTubeBookmarkSaved

Boltzmann machinesSearch on YouTubeBookmarkSaved

Why full Boltzmann machines failSearch on YouTubeBookmarkSaved

Restricted Boltzmann machinesSearch on YouTubeBookmarkSaved

Forward and backward passesSearch on YouTubeBookmarkSaved

Energy and joint probabilitySearch on YouTubeBookmarkSaved

Conditional probabilitiesSearch on YouTubeBookmarkSaved

Positive and negative phasesSearch on YouTubeBookmarkSaved

Contrastive divergence and Gibbs samplingSearch on YouTubeBookmarkSaved

RBM versus feedforward networksSearch on YouTubeBookmarkSaved

Stacked RBMs and pretrainingSearch on YouTubeBookmarkSaved

Quick self-testSearch on YouTubeBookmarkSaved

On this page