Gradient Clipping by Global Norm

Stop a single bad batch from destroying a training run.

Given a list of gradient arrays, compute the global L2 norm across all of them treated as one long vector:

total = sqrt(sum over all arrays of sum(g^2))

If total > max_norm, scale every array by max_norm / total. Otherwise return them unchanged.

Return a new list; do not modify the arrays you were given.

Scaling by one shared factor is the point. Clipping each array separately, or clipping elementwise, changes the direction of the update, so you are no longer descending the gradient you computed. A global rescale keeps the direction and only shortens the step.

This is what makes recurrent networks trainable. A single exploding step can throw the weights somewhere the model never recovers from, and clipping bounds the damage without touching the well behaved steps at all.

Guard total == 0, which happens on a converged or fully masked batch.

Function signature

def clip_by_global_norm(grads, max_norm):

Example

Input

clip_by_global_norm([np.array([3.0, 4.0])], max_norm=1.0)

Output

[array([0.6, 0.8])]

Reasoning

The global norm is sqrt(3^2 + 4^2) = 5, which is above 1.0. Every array is scaled by 1 / 5, so [3, 4] becomes [0.6, 0.8]. The direction does not change.

Constraints and notes

• Use one norm across all arrays together.
• Return a new list. Do not change the input arrays.
• Handle a total norm of 0.