Sinusoidal Positional Encoding

Build the sinusoidal position table from the original transformer paper.

Attention is permutation invariant: shuffle the tokens and the output shuffles with them, unchanged. Position has to be injected explicitly, and this is the encoding that does it without learning any parameters.

For position pos in [0, seq_len) and dimension index i in [0, d_model):

PE[pos, 2i]     = sin(pos / 10000^(2i / d_model))
PE[pos, 2i + 1] = cos(pos / 10000^(2i / d_model))

Even columns get sines, odd columns get cosines, and both share the same frequency, computed from the pair index i = column // 2.

Assume d_model is even. Return an array of shape (seq_len, d_model).

Each pair of columns is a rotation at its own frequency, running from one cycle per token at the low end to one cycle per ten thousand tokens at the high end. Because sin and cos of a shifted angle are linear combinations of the unshifted ones, a model can attend by relative offset, and it extrapolates to sequences longer than any it was trained on.

Function signature

def positional_encoding(seq_len, d_model):

Example

Input

positional_encoding(1, 4)

Output

array([[0., 1., 0., 1.]])

Reasoning

At position 0 every angle is 0. The even columns hold sin(0) = 0 and the odd columns hold cos(0) = 1.

Constraints and notes

• d_model is even.
• Columns 2i and 2i + 1 share the frequency of pair i.
• Return shape (seq_len, d_model).