Conditional Entropy

Compute the conditional entropy H(Y | X) from a joint probability table.

H(Y | X) is how many bits of uncertainty about Y remain on average once you know X. You are given joint, where joint[x, y] = P(X = x, Y = y). Then

H(Y | X) = -sum_{x, y} P(x, y) * log2(P(x, y) / P(x))

where P(x) = sum_y P(x, y) is the marginal of X. Terms with P(x, y) = 0 contribute nothing, by the convention 0 * log 0 = 0.

Function signature

def conditional_entropy(joint):

Example

Input

joint = np.array([[0.25, 0.25],
                  [0.50, 0.00]])
conditional_entropy(joint)

Output

0.5

Reasoning

P(X = 0) = 0.5, and given X = 0, Y is a fair coin, which is 1 bit. P(X = 1) = 0.5, and given X = 1, Y is always 0, which is 0 bits. The average is 0.5 * 1 + 0.5 * 0 = 0.5 bits.

Constraints and notes

• joint has shape (n_x, n_y), non-negative, and sums to 1.
• Use log base 2.
• If X and Y are independent, H(Y | X) = H(Y).