LSTM, GRU and bidirectional RNN

Why long sequences break plain RNNsSearch on YouTubeBookmarkSaved

A plain RNN keeps one hidden state. It rewrites this state at each step with the new input. Old information gets morphed at each step. After many steps the first inputs leave almost no trace.

The slides show this with a short gap. The sky is blank. The answer is blue. A short context is enough. A plain RNN predicts this well.

The slides then show a long gap. Harit lives in India for 13 years. He loves movies and works as an artist. He is very fluent in blank. The answer is Hindi. The clue (India) sits far from the blank. A plain RNN struggles here. It has morphed the early clue away.

The slides show a subject switch. Mark is a good singer and lived in LA for three years. Jacob is also a good singer. His debut was a great success. Congrats blank. The answer is Jacob. The model must forget Mark and keep Jacob. A plain RNN has no clear forget path.

Cell state and hidden stateSearch on YouTubeBookmarkSaved

An RNN has one vector: the hidden state $h_t$. It stores memory and it also drives the output. These two roles clash. Storage needs stability. Output needs change.

LSTM splits the roles into two vectors. The cell state $C_t$ stores information (internal memory). The hidden state $h_t$ produces the output and prediction. Gates control the flow between them.

• Cell state $C_t$: a slow conveyor. It carries facts across many steps. Changes to it are small and gated.
• Hidden state $h_t$: a filtered view of $C_t$. It feeds the classifier and the next step.

LSTM gatesSearch on YouTubeBookmarkSaved

LSTM uses three gates. Each gate is a sigmoid in $[0, 1]$. A value $0$ blocks flow. A value $1$ passes all flow.

The three gates act in order. The forget gate drops stale memory. The input gate adds fresh memory. The output gate filters what leaves.

Shapes and batch-first layoutSearch on YouTubeBookmarkSaved

The slides use $B$ for batch size (number of words in the batch). They use $D$ for feature size (size of each word vector). They use $H$ for hidden size (number of hidden units).

In batch-first layout each input batch has shape $(B, T, D)$. Here $T$ is sequence length. Each hidden and cell state has shape $(B, H)$. A $D \times H$ matrix maps input to hidden. A $H \times H$ matrix maps hidden to hidden. A $B \times H$ result holds one row per sequence.

Practical frameworks favour batch-first layout. Without batching each vector is a column. With batching each vector is a row block.

Forget gateSearch on YouTubeBookmarkSaved

The forget gate decides which part of $C_{t-1}$ to keep. It reads $x_t$ and $h_{t-1}$. It outputs $f_t$ in $[0, 1]$.

A value near $1$ keeps the old cell entry. A value near $0$ erases it. For the Mark-Jacob story the gate learns to erase Mark when Jacob arrives.

Input gate and candidateSearch on YouTubeBookmarkSaved

The input gate has two parts. The gate $i_t$ decides where to write. The candidate $\tilde{C}_t$ holds the new values. The value $i_t = 0$ stores nothing. The value $i_t = 1$ stores the full candidate.

The cell update is a sum. The old term keeps gated memory. The new term adds gated input. This sum path carries gradients far. It avoids the fast decay of plain RNNs.

Output gate and hidden stateSearch on YouTubeBookmarkSaved

The output gate decides which part of $C_t$ to show. A closed gate ($o_t = 0$) blocks the cell. An open gate ($o_t = 1$) passes all of it.

The slides stress two computation styles. Separate computation learns one matrix per gate. Unified computation stacks the four matrices and learns once. Both learn the same count of parameters. Unified form runs faster.

Move the gate sliders and watch the cell state update. The readout repeats the lecture numbers at each step.

GRUSearch on YouTubeBookmarkSaved

LSTM trains slowly. It learns four gate matrices per step. GRU is a simpler LSTM with two gates and one state. It merges cell and hidden state into one hidden state $h_t$. It is faster and uses fewer parameters.

GRU gates:

• Reset gate $r_t$: controls how much of $h_{t-1}$ enters the candidate.
• Update gate $z_t$: decides how much of the candidate replaces $h_{t-1}$.

The last line can copy the past directly. When $z_t$ is near $0$, $h_t$ stays near $h_{t-1}$. This direct copy path cuts the vanishing effect.

Bidirectional LSTMSearch on YouTubeBookmarkSaved

A plain LSTM reads past and present only. State at time $t$ sees $x_1 \dots x_t$. Some tasks need the future too. A missing word may depend on words after it.

The slides show fill-in (interpolation). I am blank. Add one clue: I am blank hungry. Add more: I am blank hungry, I can eat a whole pig. Each future word narrows the blank. One direction alone cannot use the later words.

The slides show NER with apple. Sentence A: Dhaval loves apple, it keeps him healthy. Here apple is fruit. Sentence B: Dhaval loves apple, he bought their latest phone. Here apple is a brand. The word after apple (it versus he, healthy versus phone) fixes the label. A forward-only reader tags too early.

Bi-LSTM fixes this. It runs two LSTM layers. One reads forward from step $1$ to $T$. One reads backward from step $T$ to $1$. Each layer keeps its own hidden and cell states. At each step it combines the two hidden states.

Forward passSearch on YouTubeBookmarkSaved

The forward LSTM reads $x_1$ first and $x_T$ last. At step $t$ it uses $x_t$, forward $h_{t-1}$ and forward $C_{t-1}$. It stores past context.

Backward passSearch on YouTubeBookmarkSaved

The backward LSTM reads $x_T$ first and $x_1$ last. At step $t$ it uses $x_t$, backward $h_{t+1}$ and backward $C_{t+1}$. It stores future context.

Combine stepSearch on YouTubeBookmarkSaved

After both passes each step has two vectors. The model merges them into one output.

Benefits and drawbacksSearch on YouTubeBookmarkSaved

Benefits:

• It sees past and future context at each step.
• It captures long-range ties in both directions.
• It scores higher on speech, translation and sentiment tasks.
• It accepts extra layers such as attention on top.

Drawbacks:

• It costs about twice the compute of one LSTM.
• It needs large data to avoid overfitting.
• It is hard to interpret (a black box).

WorkflowSearch on YouTubeBookmarkSaved

1. Take the input sequence (words or characters as vectors).
2. Map tokens to dense embeddings (semantic vectors).
3. Run the Bi-LSTM layer over past and future context.
4. Combine forward and backward states at each step (concat or transform).

Bidirectional RNNSearch on YouTubeBookmarkSaved

A Bi-RNN applies the same two-direction idea with plain RNN cells. It passes the input through two RNNs. One reads forward and one reads in reverse. It then merges the two outputs into one final output.

Bi-RNN suits NLP and speech tasks. Plain RNN suits time-series prediction. Time series has no future at test time. Text tagging can use both sides.

Dual BPTTSearch on YouTubeBookmarkSaved

Training uses two backprop passes. The forward RNN reads the sequence in order. Its error updates the forward weights. The backward RNN reads the sequence in reverse. Its error updates the backward weights. The loop repeats until the model converges. Both sides then shape each prediction.

Merge modesSearch on YouTubeBookmarkSaved

The slides list four merges. The choice depends on the task. Concat is the default. It works well in many cases.

Text to vectorsSearch on YouTubeBookmarkSaved

A network needs numbers, not letters. The pipeline turns text into vectors. It has four stages: text, tokenize, token IDs and embedding lookup. Each word then enters the net as an $N$-dim vector at time $t$.

1. Split text into tokens (words or sub-words).
2. Map each token to an integer ID from the vocabulary.
3. Look up each ID in the embedding matrix (one row per token).
4. Feed the $N$-dim rows in order (pad short sentences to equal length).

One-hot form is $1 \times V$ (one $1$ in a long sparse row). The embedding matrix is $V \times D$ (one dense row per word). Lookup skips the sparse product. It copies rows by index. This cuts sparse compute.

Vector options:

• One-hot: a sparse vector per token.
• Bag of words: counts per word (order lost).
• TF-IDF: counts weighted by rarity across docs.
• Word embeddings: dense vectors (Word2Vec, GloVe, FastText).
• Contextual embeddings: vectors that shift with context (BERT, RoBERTa).

Static versus contextual embeddingsSearch on YouTubeBookmarkSaved

Static vectors give one vector per word. The slides use bank. Sentence 1: The bank approved my loan. Sentence 2: I sat on the bank of the river. The static vector is about $[0.21, -0.43, 0.17, \dots, 0.62]$ in both. It cannot shift sense.

Contextual models read the full context (e.g. with a Transformer). The same word then gets two vectors. Loan-bank gives about $[0.18, -0.72, 0.31, \dots, 0.44]$. River-bank gives about $[0.61, 0.12, -0.27, \dots, 0.83]$. The vectors differ. The model can tag finance versus nature.

Embedding matrix, OOV and fine-tuneSearch on YouTubeBookmarkSaved

Two ways to get the matrix:

• Train it with the task (random start, learn rows with the model).
• Use pretrained rows (Word2Vec, GloVe, FastText).

Out-of-vocabulary (OOV) words miss the table. Two fixes exist. Map them to one unknown vector. Or build them from sub-word parts (FastText style).

Fine-tune means keep training the rows on task data. Do not train a fresh Word2Vec only on new words. Its space can sit in a new coordinate frame. Then old and new rows clash. Train jointly or learn a map between spaces.

Quick self-testSearch on YouTubeBookmarkSaved

1. Why does a plain RNN fail on the Harit-Hindi sentence?

It rewrites one state at each step. The early clue (India) fades over the long gap. LSTM keeps it on the cell path.

2. State the roles of cell state and hidden state in one line each.

Cell state $C_t$ stores facts across steps. Hidden state $h_t$ is the filtered output shown to the next layer.

3. Pre-activations are 0.4 (forget), -0.4 (input), 0.5 (candidate), 0.8 (output), with prior cell 0.4. Find the new cell and hidden state.

$f_t = 0.5987$, $i_t = 0.4013$, candidate $0.4621$, $o_t = 0.69$. $C_t = 0.4249$ and $h_t = 0.2767$.

4. Differentiate reset gate and update gate of a GRU.

Reset $r_t$ scales the past inside the candidate. Update $z_t$ mixes the candidate with the past. Small $z_t$ keeps $h_{t-1}$.

5. GRU has prior state 0.6, reset pre-activation -0.2, update pre-activation 0.4 and candidate 0.1974. Find the new state.

$r_t = 0.4502$ and $z_t = 0.5987$. $h_t = 0.4013 \times 0.6 + 0.5987 \times 0.1974 = 0.3590$.

6. Why does the apple NER case need a backward pass?

The label of apple depends on later words (it/healthy versus he/phone). A forward-only pass tags before it sees them.

7. Each Bi-LSTM direction gives shape (32, 10, 64). Give concat, sum, average and max shapes.

Concat gives $(32, 10, 128)$. Sum, average and max each give $(32, 10, 64)$.

8. List the four Bi-RNN merge modes and state which one changes the size.

Concat, sum, average and max. Only concat changes size ($H$ to $2H$). The rest keep $H$.

9. Static bank vectors match in both sentences but contextual ones differ by 1.11 on the shown prefix. What does this show?

Static form stores one sense per word (gap 0). Contextual form stores sense plus context (gap 1.11). Hence river-bank and loan-bank split.

10. How do you handle OOV words, and what is wrong with a separate Word2Vec only for new words?

Map OOV words to an unknown vector or build them from sub-words. A separate Word2Vec can land in a new coordinate frame. Then its rows clash with the old space.