Gaussian Mixture E-Step

Compute the responsibilities in the E-step of the EM algorithm for a Gaussian mixture model.

A Gaussian mixture says each point comes from one of k Gaussians, chosen with probabilities weights. Component c has mean means[c] and covariance covs[c]. The responsibility r[i, c] is the posterior probability that point i came from component c:

r[i, c] = weights[c] * N(x_i | means[c], covs[c]) / sum_c' (same term for c')

where N is the multivariate normal density

N(x | mu, S) = exp(-0.5 * (x - mu)^T S^-1 (x - mu)) / sqrt((2 pi)^d * det(S))

Unlike k-means, which gives each point to one cluster, this gives each point a soft share of every cluster.

Function signature

def gmm_responsibilities(X, weights, means, covs):

Example

Input

X       = np.array([[0.0], [1.0], [4.0]])
weights = np.array([0.5, 0.5])
means   = np.array([[0.0], [4.0]])
covs    = np.array([[[1.0]], [[1.0]]])
gmm_responsibilities(X, weights, means, covs)

Output

array([[9.99664650e-01, 3.35350130e-04],
       [9.82013790e-01, 1.79862100e-02],
       [3.35350130e-04, 9.99664650e-01]])

Reasoning

For the point 1.0, the log densities differ by -0.5 * 1 + 0.5 * 9 = 4 in favour of the first component. The weights are equal, so the first responsibility is 1 / (1 + exp(-4)) = 0.982.

Constraints and notes

• X has shape (n, d), means has shape (k, d) and covs has shape (k, d, d).
• Work with log densities and subtract the row maximum before you take the exponent. Points far from every component must not give 0 / 0.
• Each row of the result sums to 1.