Layer Normalisation

Layer norm is what keeps deep transformers trainable. Unlike batch norm it normalises across the features of each example independently, so it behaves identically at train and inference time and does not care about batch size.

mean = mean(x, over the last axis)
var  = mean((x - mean)^2, over the last axis)
out  = gamma * (x - mean) / sqrt(var + eps) + beta

x has shape (batch, d), and gamma and beta both have shape (d,) and broadcast across the batch.

Use the biased variance, dividing by d and not d - 1, which is what np.var gives you by default. The eps goes inside the square root.

Function signature

def layer_norm(x, gamma, beta, eps=1e-5):

Example

Input

x = np.array([[1.0, 2.0, 3.0]])
layer_norm(x, np.ones(3), np.zeros(3))

Output

array([[-1.22473569,  0.        ,  1.22473569]])

Reasoning

The row mean is 2 and the biased variance is 2/3. So the values become (x - 2) / sqrt(2/3 + eps), which is about -1.2247, 0 and 1.2247.

Constraints and notes

• Normalise over the last axis.
• Use the biased variance, and put eps inside the square root.