Activation Function Visualizer

An activation function bends a neuron's output so a network can learn curves. Compare 11 of them side by side, with their slopes, since the slope is what backpropagation multiplies through.

Each tab starts with a short guide for beginners. Use the tab All Functions Explained for a full explanation of every function with formulas, and the last tab, Paper, Python & R, for a practice problem, a pencil worksheet, and ready-to-run Python and R code. Nothing leaves your browser.

The problem: a neuron first computes a weighted sum z = w·x + b. That is a straight-line calculation. If every layer only did that, ten layers would still collapse into one straight line, and the network could never learn curves.

The fix: after the sum, pass z through a bent function f(z). That bend is the activation function. It is what lets a network draw curved boundaries.

  • Left graph f(x): what the neuron outputs for each input.
  • Right graph f′(x): the slope. During training, backpropagation multiplies gradients by this slope, so where the slope is near 0 the network stops learning.
  • Saturated = the curve is flat, slope near 0. Healthy = slope around 1.

Try this: press Vanishing gradient below. Sigmoid's slope never goes above 0.25. Then press Dying ReLU and compare the slope for negative inputs.

Pick functions — tap to show or hide

Quick experiments:

f(x): the output

-505-2024

f′(x): the slope backprop multiplies

-50500.51
Probe x: 1.5Drag to read exact values at that point. The dashed vertical line and dots move on both graphs.
Functionf(x)f′(x)Slope status
● Sigmoid0.81760.1491Healthy
● Tanh0.90510.1807Healthy
● ReLU1.51Healthy
0.8176
Sigmoid: output f(1.5)
0.1491
Slope f′(x) at this point
7.38e-5
Gradient left after 5 layers with this slope

Step-by-step working

Step 1: The formulas

σ(x) = 1 / (1 + e^(−x))

σ′(x) = σ(x) · (1 − σ(x))

Output range: (0, 1)

Step 2: Forward pass at x = 1.5

e^(−x) = 0.22313, so σ = 1 / (1 + 0.22313) = 0.81757

Result: 0.81757

Step 3: The slope at the same point

σ′ = 0.81757 × (1 − 0.81757) = 0.14915

Result: 0.14915

Step 4: What that slope means

Healthy: a useful amount of gradient passes through.

Step 5: Why it matters in training

Backpropagation multiplies the incoming gradient by this slope at every layer. Five layers with this same slope leave 7.38e-5 of the original gradient. Open the Vanishing Gradient Lab tab to test more layers.

Step 6: When to use it

Output layer for yes/no probability. Avoid in hidden layers: it flattens at both ends, and its slope never exceeds 0.25.

How this calculator works

The Explorer plots each function and its analytic derivative (checked against numerical differences), with exact GELU using an erf approximation accurate to about 1e-7. The lab multiplies the per-layer factor |w|·f′(z) once per layer to show how gradients vanish or explode on a log scale. The last tab generates worksheets and code from your own numbers.

Real networks have different z and w in every layer, so the lab is a simplified picture of the same effect. Learning tool only.

Common questions