What is Decoder?
In deep learning, a decoder is a neural network module that converts an encoded representation (like a context vector or latent features) into a final output such as text, images, or sequences.
Decoders are commonly paired with encoders in architectures like seq2seq models and transformers. They receive compressed information from the encoder and generate outputs step by step, often using mechanisms like attention to focus on relevant parts of the input.
In autoregressive decoders (e.g., in GPT-style models), each output token is produced conditioned on previously generated tokens, enabling tasks like text generation. They typically include layers for self-attention, cross-attention, and feed-forward processing.
Training often involves teacher forcing, where the model learns to predict the next element using ground-truth previous tokens, helping it capture sequential dependencies effectively.
Example
In machine translation, an encoder processes an English sentence into a context representation, while the decoder generates the corresponding French sentence word by word using that context.
Why it matters
Decoders power modern generative AI systems including large language models, enabling capabilities like chatbots, code generation, and image synthesis that drive today's AI applications.
Frequently asked questions
An encoder compresses input into a representation, while a decoder expands that representation into the desired output.
Related terms
An autoencoder is a neural network that learns to compress input data into a smaller representation and then reconstruct the original data from that compressed form.
A Recurrent Neural Network (RNN) is a type of neural network built to handle sequential data by passing information from one step to the next through a hidden state that acts like a memory.
An activation function is a mathematical operation applied to the output of a neuron in a neural network that decides whether the neuron should 'fire' and pass on a signal.
Backpropagation is an algorithm for training neural networks by calculating how much each weight contributed to the prediction error and adjusting those weights accordingly. It uses the chain rule to efficiently compute gradients of the loss function.
A Convolutional Neural Network (CNN) is a specialized type of deep neural network designed to process grid-like data such as images by automatically learning spatial patterns and features.
Long Short-Term Memory (LSTM) is a type of recurrent neural network architecture designed to learn and retain information over long sequences of data.