Backpropagation in a 3-Layer Neural Network

Absolutely. This is a great question. Just to make sure we're on the same page, by a 3-layer network, you mean an input layer, two hidden layers, and an output layer, right?

Okay, great. The best way to walk through this is to first set up the notation for the forward pass, and then we can use the chain rule to work backward from the loss function. I'll do my best to be clear with the notation.

• Let's call the input x.
• The first hidden layer will have pre-activation z⁽¹⁾ = W⁽¹⁾x + b⁽¹⁾ and activation a⁽¹⁾ = ReLU(z⁽¹⁾).
• The second hidden layer is similar: z⁽²⁾ = W⁽²⁾a⁽¹⁾ + b⁽²⁾ and activation a⁽²⁾ = ReLU(z⁽²⁾).
• And the output layer: z⁽³⁾ = W⁽³⁾a⁽²⁾ + b⁽³⁾ with the final predicted probabilities a⁽³⁾ = Softmax(z⁽³⁾).
• Our loss L will be the Cross-Entropy between the predictions a⁽³⁾ and the true one-hot labels y.

So, the goal is to find the partial derivatives of L with respect to all the W's and b's.

(Candidate pauses, takes a breath)

Okay, so for backpropagation, we start at the very end. The key for the output layer is the derivative of the Cross-Entropy loss with respect to the pre-activation z⁽³⁾. It simplifies really nicely to just a⁽³⁾ - y, which is the predicted probability minus the true label. It's a neat trick that saves a lot of messy calculus. Let's call this error term δ⁽³⁾.

So, δ⁽³⁾ = ∂L/∂z⁽³⁾ = a⁽³⁾ - y.

Now, to get the gradient for the weights W⁽³⁾, we apply the chain rule: it's that error δ⁽³⁾ times the derivative of z⁽³⁾ with respect to W⁽³⁾. Since z⁽³⁾ = W⁽³⁾a⁽²⁾ + b⁽³⁾, that derivative is just the activation from the previous layer, a⁽²⁾. So we get δ⁽³⁾ times (a⁽²⁾)ᵀ... the transpose is to get the dimensions right for the outer product.

Sure. Let's say we have 10 output classes and 128 neurons in the second hidden layer. δ⁽³⁾ would be a (10 x 1) vector. The activation a⁽²⁾ is a (128 x 1) vector. The weight matrix W⁽³⁾ needs to be (10 x 128). To get a (10 x 128) gradient, we need to perform an outer product of the (10 x 1) error vector with the transpose of the (128 x 1) activation vector, which is (1 x 128). That gives us the correct (10 x 128) shape for the gradient ∂L/∂W⁽³⁾. The gradient for the bias b⁽³⁾ is simpler, it's just δ⁽³⁾ itself.

Okay, so now we want δ⁽²⁾, which is ∂L/∂z⁽²⁾. This is where the 'backpropagation' really happens. We need to chain the error from the next layer back. The gradient will pass from z⁽³⁾ back to a⁽²⁾, and then from a⁽²⁾ back to z⁽²⁾.

Let me think about the chain... it's (∂L/∂z⁽³⁾) * (∂z⁽³⁾/∂a⁽²⁾) * (∂a⁽²⁾/∂z⁽²⁾).

• The first part, ∂L/∂z⁽³⁾, we already have—that's δ⁽³⁾.
• The second part, ∂z⁽³⁾/∂a⁽²⁾, is the derivative of W⁽³⁾a⁽²⁾ + b⁽³⁾ with respect to a⁽²⁾. That's just the weight matrix W⁽³⁾. So, this piece gives us (W⁽³⁾)ᵀδ⁽³⁾. Again, we use the transpose to propagate the error vector backward and match dimensions.
• The final link is ∂a⁽²⁾/∂z⁽²⁾, which is just the derivative of our activation function, in this case, ReLU'(z⁽²⁾).

So, putting it all together, δ⁽²⁾ is ((W⁽³⁾)ᵀ δ⁽³⁾) element-wise multiplied with ReLU'(z⁽²⁾). From there, getting the gradients for W⁽²⁾ and b⁽²⁾ follows the same pattern as before.

That's a great question because it points directly to the vanishing gradient problem. If you look at the formula for any δ⁽ˡ⁾, it's a product of the next layer's error, a weight matrix, and the derivative of the activation function.

If we go back many layers, say to δ⁽¹⁾, we're multiplying these terms over and over: (W⁽²⁾)ᵀ times ReLU' times (W⁽³⁾)ᵀ times ReLU'... and so on.

There are two big killers here:

1. The ReLU' term: If a neuron's input z is negative, its gradient is zero. The path is just dead. That's the 'dying ReLU' issue, which means parts of the network stop learning entirely.
2. The Weight Matrices: Even if the ReLUs are active (gradient is 1), if the norms of the weight matrices are consistently less than 1, you're repeatedly multiplying by numbers smaller than one. The gradient signal just shrinks exponentially until it's practically zero by the time it reaches the early layers. Those early layers, which should be learning fundamental features, get no meaningful updates.

Hmm, okay. ResNets are a really elegant solution. They change the function a block learns. Instead of learning a direct mapping H(x), the block learns a residual F(x), and the final output is H(x) = F(x) + x. That + x is the skip connection.

The magic happens during backpropagation. When we take the derivative of the loss L with respect to the input x of the block, the chain rule gives us ∂L/∂x = ∂L/∂H * ∂H/∂x.

And ∂H/∂x is ∂F/∂x + ∂x/∂x, which is ∂F/∂x + 1.

That +1 is the key. It creates an additive term, a sort of 'gradient superhighway' that bypasses the ∂F/∂x part. Even if the gradient through the weight layers F(x) completely vanishes—meaning ∂F/∂x goes to zero—the +1 ensures that the gradient from ∂L/∂H can flow directly back, unimpeded. It breaks the destructive chain of multiplications that causes the vanishing gradient problem in the first place.