Perplexity

Compute the perplexity of a language model on a piece of text.

A language model assigns a probability to each next token. You are given token_probs, the probability the model gave to the token that actually came next, at each position. Perplexity is the exponent of the average negative log probability:

perplexity = exp(-(1 / T) * sum_t log p_t)

It reads as an effective branching factor. A perplexity of 4 means the model was, on average, as unsure as if it were picking uniformly among 4 tokens. A perfect model has a perplexity of 1.

Function signature

def perplexity(token_probs):

Example

Input

perplexity(np.array([0.5, 0.25, 0.125]))

Output

4.0

Reasoning

In bits, the three tokens cost 1, 2 and 3 bits, so the mean cost is 2 bits. 2 ** 2 = 4. The base of the log does not matter as long as the exponent uses the same base.

Constraints and notes

• Every probability is in (0, 1].
• Average the log probabilities, not the probabilities. Taking the product of many small probabilities underflows to zero.
• Return a plain float.