Shannon Entropy

Compute the Shannon entropy of a discrete distribution, in nats.

H(p) = -sum_i p_i * log(p_i)

Entropy measures how uncertain the distribution is. It is 0 when one outcome is certain and largest when every outcome is equally likely, where it equals log(n).

Outcomes with p_i == 0 contribute nothing. The convention 0 * log 0 = 0 comes from the limit, but computing it directly gives nan, so mask those entries out before taking any log.

Use the natural log so the units are nats, matching the KL divergence problem.

Function signature

def entropy(p):

Example

Input

entropy(np.array([0.5, 0.5]))
entropy(np.array([1.0, 0.0]))

Output

0.6931471805599453
0.0

Reasoning

Two equally likely outcomes give -2 * 0.5 * log 0.5 = log 2. A certain outcome gives 0, and the term with p = 0 is skipped, not computed as nan.

Constraints and notes

• Use the natural log, so the result is in nats.
• Leave out entries where p_i == 0 before taking the log.