Lesson 41: Activation Functions

Goal

Understand activation functions as geometric gates and gradient gates.

An affine layer computes:

z=Wx+b.

An activation then transforms each coordinate:

a=\phi(z).

The activation has two jobs:

It prevents a stack of layers from collapsing into one affine map.
Its local derivative controls how much sensitivity passes backward.

The forward value determines what information continues. The derivative determines what learning signal continues.

The Problem Pressure

Lesson 40 showed that two affine layers collapse:

W_2(W_1x+b_1)+b_2 = (W_2W_1)x+(W_2b_1+b_2).

Adding more affine layers changes the factorization, but not the family of functions. The final map still has one flat decision boundary per output.

Insert a nonlinear activation:

z_1=W_1x+b_1,

a_1=\phi(z_1),

z_2=W_2a_1+b_2.

Now the intermediate transformation cannot generally be absorbed into a single matrix and bias. Different inputs can activate different intermediate units, so the effective affine map can change from one region of input space to another.

Notation / Symbol Table

Symbol	Plain-English name	Meaning
$z\in\mathbb R^m$	pre-activation vector	Affine scores before the activation.
$\phi:\mathbb R\to\mathbb R$	scalar activation	One nonlinear rule applied to a scalar score.
$a=\phi(z)\in\mathbb R^m$	activation vector	Coordinatewise output with $a_i=\phi(z_i)$ .
$\phi'(z)$	local derivative vector	Coordinate $i$ is $\phi'(z_i)$ .
$\bar a=\nabla_aL$	incoming sensitivity	Gradient arriving from later computation.
$\bar z=\nabla_zL$	outgoing sensitivity	Gradient passed back through the activation.
$\odot$	Hadamard product	Elementwise multiplication of matching coordinates.
$\alpha$	negative slope	Small positive slope used by leaky ReLU for $z<0$ .

Elementwise Means Coordinate By Coordinate

For:

z= \begin{bmatrix} z_1\\z_2\\\vdots\\z_m \end{bmatrix},

the notation $a=\phi(z)$ means:

a= \begin{bmatrix} \phi(z_1)\\ \phi(z_2)\\ \vdots\\ \phi(z_m) \end{bmatrix}.

This is not matrix multiplication. Each output coordinate depends only on the matching input coordinate.

The Jacobian is therefore diagonal:

J_\phi(z) = \begin{bmatrix} \phi'(z_1)&0&\cdots&0\\ 0&\phi'(z_2)&\cdots&0\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\cdots&\phi'(z_m) \end{bmatrix}.

That diagonal structure is why the backward pass becomes an elementwise product.

The Activation Backward Pass

Let a scalar loss $L$ depend on:

a=\phi(z).

The chain rule gives:

\nabla_zL = J_\phi(z)^T\nabla_aL.

Because the Jacobian is diagonal, its transpose is unchanged. Coordinate by coordinate:

\frac{\partial L}{\partial z_i} = \frac{\partial L}{\partial a_i}\phi'(z_i).

In vector notation:

\boxed{ \nabla_zL = \nabla_aL\odot\phi'(z) }

The shapes remain:

\underbrace{\nabla_zL}_{m\times1} = \underbrace{\nabla_aL}_{m\times1} \odot \underbrace{\phi'(z)}_{m\times1}.

No transpose appears in the final elementwise form because the diagonal Jacobian does not mix coordinates.

Sigmoid

The sigmoid function is:

\sigma(z)=\frac{1}{1+e^{-z}}.

Its range is:

0<\sigma(z)<1.

It turns an unbounded score into a value that can be interpreted as a binary probability when paired with an appropriate probabilistic model and loss.

Derivative

Start from:

\sigma(z)=(1+e^{-z})^{-1}.

Differentiate:

\sigma'(z) = (1+e^{-z})^{-2}e^{-z}.

Now use:

\sigma(z)=\frac{1}{1+e^{-z}}

and:

1-\sigma(z)=\frac{e^{-z}}{1+e^{-z}}.

Therefore:

\boxed{ \sigma'(z)=\sigma(z)(1-\sigma(z)) }

The derivative can be computed from the stored forward output rather than recomputing the exponential.

Saturation

For large positive $z$ :

\sigma(z)\approx1 \qquad\Longrightarrow\qquad \sigma'(z)\approx0.

For large negative $z$ :

\sigma(z)\approx0 \qquad\Longrightarrow\qquad \sigma'(z)\approx0.

These flat outer regions are called saturation. The unit still has an output, but its local derivative passes almost no gradient backward.

The largest sigmoid derivative occurs at $z=0$ :

\sigma'(0)=\frac14.

Even in its most responsive region, sigmoid shrinks the incoming sensitivity by at least a factor of four.

Tanh

The hyperbolic tangent is:

\tanh(z) = \frac{e^z-e^{-z}}{e^z+e^{-z}}.

Its range is:

-1<\tanh(z)<1.

Unlike sigmoid, tanh is centered around zero:

\tanh(0)=0.

Its derivative is:

\boxed{ \frac{d}{dz}\tanh(z)=1-\tanh^2(z) }

The derivative reaches $1$ at the origin, but approaches zero for large positive or negative inputs. Tanh therefore also saturates.

ReLU

The rectified linear unit is:

\operatorname{ReLU}(z)=\max(0,z).

Equivalently:

\operatorname{ReLU}(z) = \begin{cases} z,&z>0,\\ 0,&z\le0. \end{cases}

Away from zero, its derivative is:

\operatorname{ReLU}'(z) = \begin{cases} 1,&z>0,\\ 0,&z<0. \end{cases}

At $z=0$ , the mathematical derivative is undefined. Implementations choose a convention, commonly zero.

ReLU does not saturate on its positive side. A positive pre-activation passes both its value and its incoming gradient unchanged.

Its negative side is a hard gate:

forward value:   blocked
backward signal: blocked

Dead ReLU Units

Suppose a ReLU unit has:

z=w^Tx+b<0

for every training example. Then:

\operatorname{ReLU}'(z)=0

for every example, so the gradient reaching $w$ and $b$ through that unit is zero.

The unit still receives inputs and still computes its pre-activation. It is called dead because it never activates and receives no parameter update that would move it back into the positive region.

Large learning rates or poorly chosen initial biases can push units into this state.

Leaky ReLU

Leaky ReLU keeps a small negative-side slope:

\operatorname{LeakyReLU}_\alpha(z) = \begin{cases} z,&z\ge0,\\ \alpha z,&z<0, \end{cases}

where $\alpha$ is a small positive constant.

Its derivative is:

\operatorname{LeakyReLU}_\alpha'(z) = \begin{cases} 1,&z>0,\\ \alpha,&z<0. \end{cases}

The negative region is suppressed rather than completely blocked. This reduces the chance that a unit becomes permanently unable to learn.

Worked Example: Forward And Backward

Let:

z= \begin{bmatrix} -2\\0.5\\3 \end{bmatrix}

and apply ReLU:

a= \operatorname{ReLU}(z) = \begin{bmatrix} 0\\0.5\\3 \end{bmatrix}.

Suppose the incoming sensitivity is:

\nabla_aL = \begin{bmatrix} 4\\-2\\0.3 \end{bmatrix}.

The local derivative vector is:

\operatorname{ReLU}'(z) = \begin{bmatrix} 0\\1\\1 \end{bmatrix}.

Therefore:

\nabla_zL = \nabla_aL\odot\operatorname{ReLU}'(z) = \begin{bmatrix} 4\\-2\\0.3 \end{bmatrix} \odot \begin{bmatrix} 0\\1\\1 \end{bmatrix} = \begin{bmatrix} 0\\-2\\0.3 \end{bmatrix}.

The first coordinate is blocked. The other two pass backward unchanged.

Why ReLU Networks Are Piecewise Affine

Consider one hidden ReLU layer:

a_1=\operatorname{ReLU}(W_1x+b_1).

For a fixed input region, every hidden pre-activation has a fixed sign. Represent the active coordinates with a diagonal matrix:

D= \operatorname{diag}(d_1,\ldots,d_m),

where:

d_i= \begin{cases} 1,&(W_1x+b_1)_i>0,\\ 0,&(W_1x+b_1)_i<0. \end{cases}

Inside that region:

a_1=D(W_1x+b_1).

A following affine layer becomes:

z_2 = W_2D(W_1x+b_1)+b_2

= (W_2DW_1)x+(W_2Db_1+b_2).

So the network is affine while the active-set matrix $D$ remains fixed. Crossing a ReLU boundary changes $D$ , selecting a different affine map.

This is the geometric source of ReLU expressiveness:

many affine regions
+ input-dependent gates
= one piecewise-affine function

Comparing Common Activations

The same pre-activation can produce very different forward values and backward signals. Move one shared input through all four functions before compressing the differences into the table.

Activation	Output range	Derivative behavior	Main tradeoff
Sigmoid	$(0,1)$	Small in both tails	Probability-shaped output, but saturates
Tanh	$(-1,1)$	Small in both tails	Zero-centered, but still saturates
ReLU	$[0,\infty)$	Zero for negative inputs, one for positive inputs	Efficient and nonsaturating when active, but can die
Leaky ReLU	$(-\infty,\infty)$	Small positive slope for negative inputs	Preserves some negative-side gradient

There is no universally best activation. The choice changes the function’s geometry and the gradient paths available during training.

Gradient Flow Through Depth

In a chain of scalar operations:

a_k=\phi_k(z_k),

the chain rule multiplies local derivatives:

\frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial a_k} \prod_{j=1}^{k} \phi_j'(z_j) \times \text{affine derivative factors}.

If many activation derivatives are much smaller than one, their product can become tiny. This is one source of the vanishing-gradient problem.

Sigmoid and saturated tanh units repeatedly multiply by small numbers. ReLU units avoid that shrinkage on their active side because their derivative is one, although inactive ReLUs block the path completely.

A Useful Programming View

An activation layer has no trainable parameters. It needs only the forward value required to reconstruct its local derivative:

type Vector = readonly number[];

type ActivationCache = Readonly<{
  input: Vector;
}>;

type ActivationResult = Readonly<{
  output: Vector;
  cache: ActivationCache;
}>;

type Activation = Readonly<{
  forward: (input: Vector) => ActivationResult;
  backward: (
    cache: ActivationCache,
    incomingGradient: Vector,
  ) => Vector;
}>;

For ReLU, the backward rule is coordinatewise:

const reluDerivative = (z: number) => (z > 0 ? 1 : 0);

const inputGradient = incomingGradient.map(
  (gradient, index) => gradient * reluDerivative(cache.input[index]),
);

For sigmoid, store either the input $z$ or the output $a$ . If the output is stored:

const sigmoidDerivativeFromOutput = (a: number) => a * (1 - a);

The implementation should reject an incoming gradient whose length differs from the cached activation length.

Numerical Stability For Sigmoid

The direct expression:

1 / (1 + Math.exp(-z))

can overflow internally when $z$ is a large negative number.

A stable branch is:

const sigmoid = (z: number) => {
  if (z >= 0) {
    return 1 / (1 + Math.exp(-z));
  }

  const expZ = Math.exp(z);
  return expZ / (1 + expZ);
};

Both branches compute the same mathematical function while avoiding an unnecessarily huge exponential.

Common Mistakes

Calling Elementwise Activation Matrix Multiplication

$a=\phi(z)$ applies one scalar function per coordinate. The diagonal Jacobian describes its derivative, but the forward pass does not construct or multiply by that matrix.

Confusing A Small Output With A Small Derivative

For ReLU, a small positive output still has derivative one. For sigmoid, an output near zero has a derivative near zero. Output magnitude and derivative magnitude are separate facts.

Saying A Dead ReLU Stops Receiving Inputs

The unit still receives $x$ and computes $z$ . Its local derivative is zero, so gradient-based training cannot change its incoming weights through that example.

Treating The Derivative At A Kink As A Training Mystery

ReLU is nondifferentiable at exactly zero. Software selects a conventional subgradient. The value at one point is usually less important than the behavior over whole positive and negative regions.

Assuming Nonlinearity Automatically Fixes Gradient Flow

Nonlinearity adds expressiveness, but its derivative can still shrink or block gradients.

Exercises

In plain language, distinguish a pre-activation $z$ , an activation $a$ , and an activation function $\phi$ . Which of these is a value, and which is a rule?
For each of sigmoid, tanh, ReLU, and leaky ReLU, state its output range and describe where its derivative becomes small or zero.
Let:
$z= \begin{bmatrix} -1\\0\\2 \end{bmatrix}, \qquad \nabla_aL= \begin{bmatrix} 3\\-4\\0.5 \end{bmatrix}.$
Using the convention $\operatorname{ReLU}'(0)=0$ , compute $a=\operatorname{ReLU}(z)$ and $\nabla_zL$ with shapes.
Derive $\sigma'(z)=\sigma(z)(1-\sigma(z))$ from the exponential definition. Then identify where the derivative is largest.
Derive the elementwise activation backward pass from the diagonal Jacobian and explain why it becomes a Hadamard product.
A chain contains six sigmoid activations, each evaluated where its derivative is $0.1$ . Ignoring the affine factors, by what factor is a sensitivity scaled across those activations? Explain the training implication.
Diagnose the claim: “A ReLU output of $0.001$ passes almost no gradient.” Separate output magnitude from local derivative.
Explain why leaky ReLU reduces dead-unit risk without making its negative and positive regions equally influential.
For a one-hidden-layer ReLU network, show that fixing the active-unit pattern reduces the network to one affine map. What event changes that affine map?
Implement sigmoid, tanh, ReLU, and leaky ReLU activation layers in TypeScript with:
- forward caches,
- dimension checks in the backward pass,
- stable sigmoid evaluation,
- and central finite-difference tests for every input coordinate.

Readiness Check

You are ready for the next lesson when you can:

explain why nonlinearities prevent affine-layer collapse,
compare activation functions using both forward geometry and derivative behavior,
identify saturation and dead-unit behavior,
derive $\nabla_zL=\nabla_aL\odot\phi'(z)$ from the Jacobian,
and explain why a fixed ReLU activation pattern defines one affine region.

Summary

Activations make stacked layers more expressive by changing the effective map according to the input.
Elementwise activations have diagonal Jacobians.
Their backward pass multiplies the incoming sensitivity by local derivatives coordinatewise.
Sigmoid and tanh saturate in their tails.
ReLU passes gradients on its positive side and blocks them on its negative side.
ReLU networks are piecewise affine because each activation pattern selects an affine map.

Next Lesson

Lesson 42 builds feedforward networks by composing linear and activation layers. It will track hidden representations through the forward pass and show how depth organizes many local affine regions into one model.

Goal

The Problem Pressure

Notation / Symbol Table

Elementwise Means Coordinate By Coordinate

The Activation Backward Pass

Sigmoid

Derivative

Saturation

Tanh

ReLU

Dead ReLU Units

Leaky ReLU

Worked Example: Forward And Backward

Why ReLU Networks Are Piecewise Affine

Comparing Common Activations

One Pre-activation, Four Different Gates

Sigmoid

Tanh

ReLU

Leaky ReLU

Gradient Flow Through Depth

A Useful Programming View

Numerical Stability For Sigmoid

Common Mistakes

Calling Elementwise Activation Matrix Multiplication

Confusing A Small Output With A Small Derivative

Saying A Dead ReLU Stops Receiving Inputs

Treating The Derivative At A Kink As A Training Mystery

Assuming Nonlinearity Automatically Fixes Gradient Flow

Exercises

Readiness Check

Summary

Next Lesson