Perceptrons and Linear Layers

Goal

Turn the affine maps you already know into trainable units.

A perceptron or linear layer does not introduce new linear algebra. It assigns machine-learning roles to familiar objects:

input vector
→ weighted sum
→ bias shift
→ optional activation
→ output

The core computation is an affine map:

A neural network is built by composing many such maps with nonlinear functions between them.

Why Start Here

Suppose an input has two features:

We want a model that produces a score saying which side of a boundary the point lies on. Choose weights:

and a bias $b$. The score is:

The model may classify using the sign:

The boundary itself is where the model is undecided:

That equation defines a line in two dimensions, a plane in three dimensions, and a hyperplane in higher dimensions.

Notation / Symbol Table

A Perceptron Is An Affine Map Plus A Decision Rule

The word linear is used loosely in neural-network programming. The map:

is technically affine because it includes the constant term $b$. It is linear only when $b=0$.

The distinction matters geometrically:

• $w^Tx=0$ must pass through the origin.
• $w^Tx+b=0$ may be shifted away from the origin.

The original perceptron used a hard threshold. Modern neural networks more often use differentiable or piecewise-differentiable activations, but the affine core is the same.

The Weight Vector Is Normal To The Boundary

Consider the decision boundary:

Let $x_1$ and $x_2$ be any two points on that boundary. Then:

and:

Subtract the equations:

So every displacement lying along the boundary is orthogonal to $w$. Therefore $w$ points straight across the boundary.

This gives the weights two simultaneous interpretations:

1. As coefficients in a weighted sum.
2. As the normal direction of the decision boundary.

The score changes most rapidly when moving parallel to $w$.

What The Bias Does

Holding $w$ fixed preserves the boundary’s orientation. Changing $b$ translates the boundary.

For a one-dimensional input:

has boundary location:

provided $w\ne0$.

In higher dimensions, the signed score scaled by the weight norm gives signed distance from the boundary:

The sign tells which side of the boundary contains $x$. The magnitude tells how far away it is in the normal direction.

Worked Example: A Two-Feature Perceptron

Let:

For input:

compute:

A hard threshold would classify this point as positive because $z>0$.

The boundary is:

or:

The vector $w=(2,-1)^T$ is perpendicular to this line.

From One Unit To A Linear Layer

A single unit produces one scalar. A layer with $m$ units produces $m$ scores.

Let each unit have its own weight vector $w_i\in\mathbb R^n$. Stack their transposes as rows:

Stack the biases into:

Then every unit can be evaluated at once:

The shape check is:

Each row of $W$ asks a different linear question about the same input.

Worked Example: A Three-Output Layer

Let:

Then:

Adding the bias gives:

The layer has transformed a two-dimensional input into a three-dimensional representation.

Why Activations Are Needed

Composing affine maps without nonlinear activations does not create a more expressive class of functions.

Suppose:

and:

Substitute the first into the second:

The composition is still one affine map.

A nonlinear activation prevents this collapse:

The nonlinearity is what lets depth create genuinely new functions.

Common Activation Examples

Hard Threshold

Useful for understanding the historical perceptron, but unsuitable for ordinary gradient-based training because its derivative is zero almost everywhere and undefined at zero.

Sigmoid

Maps scores into $(0,1)$. It is useful for probabilities and binary outputs, though deep networks often avoid it in hidden layers because large positive or negative inputs produce small gradients.

ReLU

Piecewise linear and inexpensive. Its derivative is:

At $z=0$, software chooses a convention, commonly zero.

The Local Derivatives Of A Single Unit

Let:

Treat $x$, $w$, and $b$ as separate inputs to this computation.

The derivative with respect to the input is:

The derivative with respect to the weights is:

The derivative with respect to the bias is:

These formulas are symmetric for a reason. The scalar score is the dot product pairing between $w$ and $x$. Holding either side fixed makes the other side’s gradient equal to the fixed vector.

For:

the chain rule gives:

and:

The forward pass stores $x$, $w$, $b$, and $z$. The backward pass multiplies the incoming sensitivity by these local derivatives.

Gradients For A Linear Layer

For:

suppose a scalar loss $L$ produces an incoming sensitivity:

Then:

and:

Check the shapes:

and:

The weight gradient is an outer product. Each output sensitivity scales the input vector to produce one row of the matrix gradient.

A Useful Programming View

A linear layer can keep forward data and backward behavior together:

type Vector = readonly number[];
type Matrix = readonly (readonly number[])[];

type LinearCache = Readonly<{
  input: Vector;
}>;

type LinearLayer = Readonly<{
  weights: Matrix;
  bias: Vector;
}>;

type LinearBackward = Readonly<{
  inputGradient: Vector;
  weightGradient: Matrix;
  biasGradient: Vector;
}>;

The layer’s forward pass computes $Wx+b$ and stores the input. Its backward pass receives $\bar z$, then computes the three gradients above.

This is the first reusable trainable operation in the neural-network implementation path.

Connection To Category Theory

The useful connection is composition, not terminology.

A linear layer is a morphism-like computational component with a typed input and output:

R^n → R^m

Layers compose only when adjacent shapes match. The forward maps compose in the ordinary direction. Reverse-mode sensitivities travel backward through the transposed local linear maps.

The nonlinear activation is crucial because a category containing only affine maps remains closed under composition: composing more arrows does not escape the same family of functions.

Why This Matters For Machine Learning

Linear layers appear throughout modern models:

• Feature classifiers use $w^Tx+b$.
• Multiclass classifiers use one row of $W$ per class.
• Hidden neural-network layers learn new feature coordinates.
• Convolution can be understood as a structured linear map with shared weights.
• Attention uses learned linear projections to create queries, keys, and values.
• Transformer feed-forward blocks are pairs of linear layers separated by a nonlinearity.

Understanding this layer precisely means much of neural-network architecture becomes repeated composition rather than new mathematics each time.

Common Mistakes

Calling The Bias Part Of The Weight Vector Without Saying So

It is possible to append a constant feature $1$ to $x$ and absorb $b$ into an enlarged weight vector. That representation is valid, but it hides the distinct role of the bias unless stated explicitly.

Stacking Weight Vectors In The Wrong Orientation

For $z=Wx+b$, each output unit’s weights are a row of $W$. Columns correspond to input features.

Treating $Wx+b$ As Nonlinear

The bias makes the map affine, not nonlinear. Nonlinearity comes from $\phi$.

Assuming Several Linear Layers Automatically Add Expressiveness

Without nonlinear activations between them, they collapse into one affine map.

Forgetting That Parameters Are Inputs To Differentiation

During inference, $W$ and $b$ are fixed. During training, they are variables whose gradients must be computed.

Using A Hard Threshold During Gradient-Based Training

The threshold’s derivative does not provide useful gradients. Smooth or piecewise-linear surrogate activations are used instead.

Exercises

1. In plain language, explain the separate roles of $w$, $x$, and $b$ in $z=w^Tx+b$. Which quantities are data, and which are trainable parameters?

2. Let $w=(3,-2)^T$ and $b=4$. Write the decision boundary in slope-intercept form. State a normal vector and give one point on each side of the boundary.

3. Show algebraically that changing $b$ while holding $w$ fixed translates a decision boundary without changing its orientation.

4. For:

   $$W= \begin{bmatrix} 1&-1&2\\ 0&3&1 \end{bmatrix}, \qquad b= \begin{bmatrix} 2\\-1 \end{bmatrix}, \qquad x= \begin{bmatrix} 2\\1\\-1 \end{bmatrix},$$

   compute $z=Wx+b$ and annotate every object’s shape.

5. Let $z=w^Tx+b$ and $a=\sigma(z)$. Derive $\nabla_w a$, $\nabla_x a$, and $\partial a/\partial b$ using the chain rule.

6. Prove that two affine layers without an intervening nonlinearity can be represented by one affine layer. Identify the combined weight matrix and bias vector.

7. For $z=Wx+b$, derive $\nabla_xL$, $\nabla_WL$, and $\nabla_bL$ from the differential:

   $$dz=dW\,x+W\,dx+db.$$

   Use the Frobenius inner product to identify the matrix gradient.

8. A ReLU unit receives $z=-0.4$. What output and local derivative does it produce? What practical training problem can occur if a unit remains in this region for every example?

9. Diagnose the claim: “The bias controls how important the input features are.” Explain what the bias actually controls and which object controls feature coefficients.

10. Implement a TypeScript linear layer with:

    • dimension checks,
    • a forward pass returning both output and cache,
    • a backward pass returning input, weight, and bias gradients,
    • tests using finite differences for every parameter and input coordinate.

Readiness Check

You are ready for the next lesson when you can:

• read $w^Tx+b$ both algebraically and geometrically,
• identify the decision boundary and its normal vector,
• generalize one unit to $z=Wx+b$,
• track every shape in the forward and backward passes,
• explain why nonlinear activations are required between layers,
• and derive the linear layer’s gradients rather than memorize them.

Summary

• A perceptron computes an affine score and applies a decision or activation rule.
• The weight vector is normal to the decision boundary; the bias translates that boundary.
• A linear layer stacks several units into $z=Wx+b$.
• Several affine layers without nonlinearities collapse into one affine map.
• For a scalar loss, the linear-layer gradients are $W^T\bar z$, $\bar z x^T$, and $\bar z$.
• Linear layers are the reusable trainable maps from which larger neural networks are assembled.

Next Lesson

Lesson 41 introduces activation functions as geometric gates. It will compare sigmoid, tanh, and ReLU, study their derivatives and saturation behavior, and show how nonlinearities change the functions a network can represent.