What is Activation Function?
An activation function is a mathematical operation applied to the output of a neuron in a neural network that decides whether the neuron should 'fire' and pass on a signal.
It takes the weighted sum of inputs plus bias and transforms it into an output value, typically between a certain range like 0-1 or -1 to 1.
Without activation functions, a neural network would behave like a simple linear model and could not learn complex patterns in data.
Popular examples include ReLU, which outputs the input if positive or zero otherwise, and Sigmoid, which squashes values into a probability-like range.
Example
In a photo classifier, an activation function might turn a neuron's calculation about edge patterns into a 'yes' signal only if the pattern strongly matches an eye feature.
Why it matters
Activation functions enable deep networks to model non-linear relationships, which is essential for modern AI tasks like image recognition and language translation.
Frequently asked questions
Linear functions alone cannot capture complex data patterns, so non-linear activations allow networks to learn intricate relationships.
Related terms
Backpropagation is an algorithm for training neural networks by calculating how much each weight contributed to the prediction error and adjusting those weights accordingly. It uses the chain rule to efficiently compute gradients of the loss function.
An autoencoder is a neural network that learns to compress input data into a smaller representation and then reconstruct the original data from that compressed form.
A Convolutional Neural Network (CNN) is a specialized type of deep neural network designed to process grid-like data such as images by automatically learning spatial patterns and features.
In deep learning, a decoder is a neural network module that converts an encoded representation (like a context vector or latent features) into a final output such as text, images, or sequences.
Long Short-Term Memory (LSTM) is a type of recurrent neural network architecture designed to learn and retain information over long sequences of data.
A Recurrent Neural Network (RNN) is a type of neural network built to handle sequential data by passing information from one step to the next through a hidden state that acts like a memory.