ANN blocks and activations
This lecture names the building blocks of a neural network (neuron, input layer, hidden layer, output layer). It shows how connections, learning rules and activation functions describe any model. After this page you can compute activation outputs and softmax probabilities by hand. You can also pair each task with the right output-layer activation.
Why networks need non-linearitySearch on YouTube
A single-layer network draws only straight boundaries. It cannot capture non-linear functions. Higher variants of the network add hidden layers and non-linear activation functions. Those variants learn curves and complex regions.
Components of an ANNSearch on YouTube
An ANN has four building blocks. They are the neuron, the input layer, the hidden layer and the output layer. Each block has one clear job.
NeuronSearch on YouTube
A neuron is a cell inside the ANN. It receives values and performs a computation. It holds the result , which shows the activation level of that neuron. The computation is a weighted sum of the inputs plus a bias value. The bias decides whether the neuron turns active.
is the weighted sum plus bias. are weights and are inputs. is the bias. is the activation function. is the activation level held by the neuron.
Worked example. One neuron has weights and bias . The input vector is .
Step 1. Weighted sum.
Step 2. Activation outputs. ReLU keeps positives, so . The sigmoid gives . The same gives different answers under different activation functions.
Input layerSearch on YouTube
The input layer is the first layer of the ANN. It receives the direct input, a set of values used for computation. An input with dimensions is an input vector with dimensions.
In theory each input neuron is a placeholder for one dimension. It performs no computation. So the neuron count equals the dimension count. For the input layer holds three neurons.
First dense layer in practiceSearch on YouTube
Libraries such as Keras design the first layer as a dense layer. Each neuron there receives a full copy of the whole input vector. So the neuron count need not equal the input dimension count. A 3-dimensional input can feed a first dense layer with 16 neurons.
Hidden layerSearch on YouTube
Hidden layers sit between the input layer and the output layer. Each hidden layer holds its own set of neurons, and counts vary across layers. Their job is to find patterns for each input and pass an output onward. Hidden layers give the network its non-linear power.
Output layerSearch on YouTube
The output layer is the last layer in the ANN. It reads the values from the previous layer. It returns one or more values as the answer.
Basic architecture of an ANNSearch on YouTube
The arrangement of neurons into layers is the network architecture. The connection pattern within and between layers is part of it too. Two networks with the same neurons but different wiring are different architectures.
Three aspects of an ANNSearch on YouTube
Models differ along three aspects:
- The synaptic interconnections of the model.
- The training or learning rules that update the connection weights.
- The activation functions of the neurons.
The next three sections follow this order.
InterconnectionsSearch on YouTube
Interconnections fall into two types: feedforward and feedback. Feedforward sends data forward only. Feedback sends outputs back as inputs.
Feedforward neural networksSearch on YouTube
Data flows only one way: from input to hidden layers to output. No cycles or loops exist. Information moves forward. The Sequential model with Dense layers is one example. It suits classification and regression with fixed-size inputs.
model = Sequential([Dense(16, activation="relu"), Dense(10, activation="softmax")])Single-layer perceptronSearch on YouTube
A single-layer perceptron has no hidden layers. One input layer feeds the output layer directly. The output layer can hold one or more perceptrons. Each perceptron produces its own output, which suits multi-class classification. Each output neuron acts like a linear perceptron.
Multilayer perceptronSearch on YouTube
A multilayer perceptron (MLP) stacks multiple neuron layers. It includes hidden layers. It handles linear and non-linear functions.
| Single-layer perceptron | Multilayer perceptron |
|---|---|
| No hidden layer | One or more hidden layers |
| Input layer feeds output directly | Signals pass through hidden layers |
| Only linear boundaries | Linear and non-linear functions |
| One output per perceptron | Each output neuron gives one value |
Feedback neural networksSearch on YouTube
A feedback network sends outputs back as inputs to the same layer or a previous layer. Feedback to the same layer is lateral feedback. Networks use it to model competition or cooperation between neurons. In competitive learning, strong neurons suppress weak ones through lateral inhibition.
Feedback networks are also called recurrent or dynamic networks. Two feedback types exist:
- Self-feedback: the output feeds back into the same neuron.
- Cross-neuron feedback: the output feeds other neurons in the same or previous layers. Lateral feedback is cross-neuron feedback restricted to the same layer.
Such networks suit systems that must stabilise or converge over time.
Do not confuse a feedback network with reinforcement learning. A feedback network is wiring: outputs loop back as inputs. Reinforcement learning is a learning model: rewards shape actions. The slides stress this difference.
Recurrent neural networksSearch on YouTube
A recurrent neural network (RNN) is a feedback network for sequences. Loops carry information from previous time steps into current processing. RNNs suit time series, language, speech and any sequential data. Variants include LSTM, GRU and bidirectional RNNs.
Single neuron with feedbackSearch on YouTube
This network looks like a perceptron. It adds one extra input: the feedback from its own output.
Single layer with feedbackSearch on YouTube
This network looks like a single-layer perceptron with no hidden layer. It takes feedback from the output as one of its inputs.
Multi-layer with feedbackSearch on YouTube
This network looks like an MLP. The output feeds back to itself or to another neuron in the same layer. It can also feed nodes in the preceding layer. That wiring forms a multilayer recurrent network.
Learning rulesSearch on YouTube
Learning in an ANN follows three aspects: parameter learning, structure learning and the learning model. Each aspect answers a different question: which values change, which wiring changes and which feedback signal guides change.
Parameter learningSearch on YouTube
Parameter learning tunes the weights and biases. These are the learnable parameters of the model. Backpropagation and optimizers perform the tuning. Optimizers include Adam, SGD and RMSprop. They minimise the loss function.
is the parameter vector (weights and biases). is the learning rate. The derivative sets the direction and size of each step.
Structure learningSearch on YouTube
Structure learning changes the network itself to find the best architecture. Neural Architecture Search (NAS) uses search algorithms such as RL, evolutionary algorithms and gradients. Pruning is another approach: it removes or zeroes out less useful neurons. Structure learning tunes the hyperparameters of the network. Tools include AutoML, Google NAS and ENAS.
Learning modelsSearch on YouTube
The learning model decides which machine-learning feedback the ANN uses. Three models exist: supervised, unsupervised and reinforcement learning.
Supervised learningSearch on YouTube
The input is a vector plus a target label. Together they form a training pair. The model trains on both inputs and labels. It can assess its own learning from this data. Parameters update toward optimal performance. Problems are variants of classification or regression.
Unsupervised learningSearch on YouTube
The input is a vector with no target label. Parameter updates use only the input vector. The model must find patterns and organise them as clusters. It updates through contrastive objectives, clustering objectives or reconstruction loss in autoencoders. Outputs stay internal and guide parameter updates. They are not shared externally as neuron inputs.
Reinforcement learningSearch on YouTube
This learning grows stronger through rewards and punishments for each action. The input vector defines the current state of the model. The output is the action taken. Feedback arrives as a response to the action. The response updates the current state of the model. The goal is an optimal strategy that maximises the cumulative reward.
| Supervised | Unsupervised | Reinforcement |
|---|---|---|
| Input vector plus label | Input vector only | State vector plus reward signal |
| Loss against labels | Clustering or reconstruction objectives | Cumulative reward objective |
| Classification or regression | Clusters and features | Optimal action strategy |
Activation functionsSearch on YouTube
Activation functions introduce non-linearity into the network. Without them, stacked layers would collapse into one linear map. Weights and biases are the learnable parameters. Activation functions shape what those parameters express.
Think of a dimmer switch with different brands. Each brand maps the same knob position to its own brightness. The knob is . The brand is the activation function.
What an activation function doesSearch on YouTube
An activation function sets the neuron activity level from . Each function defines its own outcome for the same . For example, equals 0 while stays above 0. Hidden layers and the output layer usually use different activation functions. The problem at hand decides the output-layer choice.
Types of activation functionsSearch on YouTube
The slides name seven functions: identity (linear), step, sign, sigmoid (logistic), tanh (hyperbolic), ReLU and softmax.
Identity returns unchanged. Step returns a binary 0 or 1. Sign returns , 0 or . Sigmoid maps into . Tanh maps into . ReLU zeroes negatives and keeps positives. SiLU multiplies by its own sigmoid.
Fix the step convention before the exam and state it. This page uses 0 for and 1 for . Some texts put 1 only for . Examiners accept either convention when stated.
Worked example. Compute all six single-input activation outputs for , and .
Step 1. Identity, step and sign.
Step 2. Sigmoid. Use .
Step 3. Tanh and ReLU.
Identity and ReLU match for and differ for . Sigmoid stays above 0 at while ReLU gives exactly 0.
SoftmaxSearch on YouTube
Softmax takes a vector and returns a vector of the same length. It converts arbitrary real values into a probability distribution. The probabilities sum to 1.
is the probability of class . The numerator exponentiates one logit. The denominator sums exponentials over all classes.
Worked example. Logits are . Turn them into probabilities.
Step 1. Exponentiate each logit.
Step 2. Sum and divide.
Step 3. Check. . Class 1 wins with probability 0.659.
Step 4. Stable repeat (extra check). Subtract the maximum 2.0 first: . Exponentials are 1, and . The sum is . Then . Shifting all logits leaves every probability unchanged.
Softmax is interdependentSearch on YouTube
ReLU and sigmoid map one input to one output. Their graphs fit a flat 2D plane of . Softmax is a multivariable function. Each class output depends on the exponential sum over all classes. Change one input number and every class probability shifts. The other inputs stay fixed, yet their shares still move.
Output-layer and hidden-layer rulesSearch on YouTube
The task decides the output activation. The network type suggests the hidden activation.
| Task | Output activation | Reason |
|---|---|---|
| Regression | Linear (identity) | Outputs need any real value |
| Binary classification | Sigmoid | One output in |
| Multiclass classification | Softmax | Probabilities over classes sum to 1 |
| Multi-label classification | Sigmoid per label | Each label gets its own probability; sums do NOT reach 1 |
| Network | Hidden activation | Reason |
|---|---|---|
| Convolutional network | ReLU | Sparse and cheap activations |
| Recurrent network | Tanh or sigmoid | Bounded values suit loops |
Worked example. Pair each task with its output activation.
House-price prediction. Prices are continuous values. Use a linear output. One neuron returns the price directly.
Spam detection. The mail is spam or not spam. Use one sigmoid neuron. Outputs near 1 mean spam.
Digit recognition (0 to 9). Each image holds exactly one digit. Use softmax over 10 neurons. The 10 probabilities sum to 1.
Photo tagging (cat, outdoor, night). One photo can carry several tags. Use one sigmoid neuron per tag. Each probability stands alone, so the three need not sum to 1.
I Step, I Sign, I Sigmoid, I Tanh, I Rectify, I Soften: I-S-S-S-T-R-S
Identity · Step · Sign · Sigmoid · Tanh · ReLU · Softmax. Seven functions in slide order.
Regression-Linear, Binary-Sigmoid, Multi-Softmax, Labels-Sigmoid: R-B-M-L
Regression uses Linear · Binary uses Sigmoid. Multiclass uses Softmax · multi-Label uses one Sigmoid per label. Hidden layers: CNN pairs with ReLU, RNN pairs with Tanh or sigmoid.
Differentiation of activation functionsSearch on YouTube
Backpropagation updates each weight and bias from the gradient of the loss. The chain rule passes through every activation function. So the differentiability of the activation function matters.
Three cases follow:
- Zero derivative: the parameter does not change. Training gets stuck.
- Very small derivative: the parameter changes too slowly. This is the vanishing gradient.
- Very large derivative: the parameter may jump too far. This is the exploding gradient.
Key derivatives (extra)Search on YouTube
These standard results show why the three cases above occur.
Sigmoid slopes peak at 0.25, so deep stacks shrink gradients fast. Tanh slopes peak at 1. ReLU passes full gradient for positives and zero for negatives.
Worked example. Compare gradient flow at three operating points.
Step 1. Sigmoid at . . Slope . This is the strongest sigmoid gradient.
Step 2. Sigmoid at . . Slope . The neuron is saturated, so the parameter barely moves.
Step 3. ReLU at and . Slope at . Slope at . The negative input blocks learning through that neuron.
Step 4. Tanh at . . Slope . Saturation strikes here too.
Step function vs sigmoidSearch on YouTube
The slides contrast the step curve with the sigmoid curve. Step jumps from 0 to 1 at the threshold. Its slope is zero almost everywhere, so no gradient flows. Sigmoid smooths the same jump into an S-curve. Its slope stays nonzero, so learning proceeds. Differentiability is the practical difference between them.
ReLUSearch on YouTube
ReLU means Rectified Linear Unit. It removes all negative outputs from the network. It joins two piece-wise linear parts, so the whole map stays non-linear. It allows sparse activations: many neurons output exact 0. Sparsity improves network efficiency. ReLU handles the vanishing gradient problem better than sigmoid and tanh. Positive inputs pass gradient 1, which never shrinks.
SiLUSearch on YouTube
SiLU means Sigmoid Linear Unit. It multiplies by . It is non-monotonic: its output does not always rise with . Small negatives dip below zero before the curve climbs. Image classification networks use it often.
GeLUSearch on YouTube
GeLU means Gaussian Error Linear Unit. It combines a linear unit with a non-linear tanh form. It maps negatives smoothly toward values near zero. It keeps strong positive outputs for positive inputs. NLP models use it widely, including BERT, transformers and GPT 2/3.
is the input. The tanh term bends the line smoothly. Large positives pass through almost unchanged. Large negatives fade toward zero.
Quick self-testSearch on YouTube
1. A neuron has weights (0.5, -0.4, 0.3), bias 0.1 and input (2.0, 1.0, -1.0). Find z, ReLU(z) and σ(z).
. . .
2. Compute ReLU(-1), σ(-1), tanh(-1), step(-1) and sign(-1).
. . . Step gives 0. Sign gives .
3. Logits are (2.0, 1.0, 0.1). Find the three softmax probabilities and name the winner.
Exponentials are 7.3891, 2.7183 and 1.1052. The sum is 11.2126. Probabilities are 0.659, 0.242 and 0.099. Class 1 wins.
4. Why does shifting all logits by the same constant leave softmax unchanged?
Each numerator and the denominator gain the same factor . The factor cancels in every ratio. The worked example shifts by and still gets 0.659.
5. Pair each task with its output activation: house prices, spam detection, digit 0-9, photo with several tags.
House prices use linear. Spam uses one sigmoid neuron. Digits use softmax over 10. Photo tags use one sigmoid neuron per tag, since sums need not reach 1.
6. Differentiate a single-layer perceptron from a multilayer perceptron.
A single-layer perceptron has no hidden layer and learns only linear boundaries. An MLP adds hidden layers and learns non-linear functions too.
7. Differentiate self-feedback from cross-neuron feedback. What is lateral feedback?
Self-feedback returns the output to the same neuron. Cross-neuron feedback sends it to other neurons in the same or previous layers. Lateral feedback is cross-neuron feedback within the same layer.
8. σ(5) ≈ 0.9933. Find the slope there and state the training effect.
Slope . The neuron is saturated. The parameter barely moves, so learning stalls (vanishing gradient).
9. Why does ReLU beat sigmoid on the vanishing gradient, and what is its own failure case?
Positive inputs pass slope 1, which never shrinks through depth. Its failure case is slope 0 for negatives: a stuck neuron stops learning.
- A single layer learns only linear boundaries. Hidden layers plus non-linear activation functions capture non-linear functions.
- A neuron forms and holds . gives ReLU and sigmoid .
- The input layer receives the input vector. Theory uses one placeholder neuron per dimension. Keras dense layers may use any neuron count.
- Hidden layers find patterns between input and output. The output layer returns the final values.
- Interconnections split into feedforward (one way, no cycles) and feedback (outputs return as inputs). RNNs are feedback networks for sequences.
- Learning splits into parameter learning (weights and biases via ), structure learning (NAS, pruning) and the learning model (supervised, unsupervised, reinforcement).
- Seven activation functions: identity, step, sign, sigmoid, tanh, ReLU, softmax. Memory hook: I-S-S-S-T-R-S.
- Softmax gives probabilities that sum to 1. Logits give .
- Output rules: regression uses linear, binary uses sigmoid, multiclass uses softmax, multi-label uses one sigmoid per label. Hidden rules: CNN uses ReLU, RNN uses tanh or sigmoid.
- Zero derivative stalls learning, tiny derivatives vanish and huge derivatives explode. Sigmoid peaks at slope . SiLU serves images, GeLU serves NLP models.
But what is a neural network?
ReLU In Action
ArgMax and SoftMax
Activation Functions Visually