KL Divergence

Compute the Kullback-Leibler divergence between two discrete distributions.

KL(p || q) = sum_i p_i * log(p_i / q_i)

It measures how many extra nats you pay for encoding samples from p using a code built for q. It is zero when the distributions match and always non-negative, but it is not symmetric, so KL(p || q) and KL(q || p) are different numbers. That asymmetry is exactly what separates the two directions of variational inference.

Two edge cases:

• Where p_i == 0 the term contributes 0, since 0 * log(0) is taken as 0 by convention.
• Where p_i > 0 but q_i == 0 the divergence is infinite. Clip q to a floor of 1e-12 so the result stays a large finite number rather than inf.

Return the result in nats, using the natural log.

Function signature

def kl_divergence(p, q):

Example

Input

p = np.array([0.5, 0.5])
q = np.array([0.9, 0.1])
kl_divergence(p, q)

Output

0.5108256237659907

Reasoning

0.5 * log(0.5 / 0.9) + 0.5 * log(0.5 / 0.1) = 0.5 * (-0.5878) + 0.5 * 1.6094, which is about 0.5108.

Constraints and notes

• Terms with p_i == 0 contribute 0.
• Clip q to at least 1e-12 so the result stays finite.
• Use the natural log.