Jensen-Shannon Divergence

Compute the Jensen-Shannon divergence between two discrete distributions.

KL divergence is not symmetric, and it is infinite when q puts zero mass where p does not. The Jensen-Shannon divergence fixes both problems. It compares each distribution to their average m:

m      = (p + q) / 2
JS(p, q) = 0.5 * KL(p || m) + 0.5 * KL(q || m)
KL(a || b) = sum_i a_i * log2(a_i / b_i)     (skip terms where a_i = 0)

With log base 2 the result is always between 0 and 1. The original GAN objective is, at its optimum, a Jensen-Shannon divergence.

Normalise p and q to sum to 1 before you start, so that raw counts work too.

Function signature

def js_divergence(p, q):

Example

Input

js_divergence(np.array([0.5, 0.5]), np.array([0.9, 0.1]))

Output

0.14679310

Reasoning

The average is m = [0.7, 0.3]. KL(p || m) = 0.1258 and KL(q || m) = 0.1678. Their mean is 0.1468. Swapping p and q gives the same value.

Constraints and notes

• p and q are non-negative arrays of the same shape with a positive sum.
• Distributions with disjoint support give exactly 1.0.