Population Stability Index

Measure how far a feature has drifted in production with the population stability index (PSI).

A model is trained on one distribution of data and then served on whatever arrives later. If the inputs drift, accuracy usually drops before anyone sees it in the labels. PSI is a common alarm for this. It bins a feature, then compares the share of rows in each bin between the reference data (expected) and the new data (actual):

1. Take bins + 1 quantiles of expected at np.linspace(0, 1, bins + 1) and keep the interior ones as the bin edges. So there are bins - 1 edges.
2. Put a value in bin k, where k is the number of edges that are less than or equal to the value. np.searchsorted(edges, values, side="right") does exactly this.
3. Let e_k and a_k be the fractions of expected and actual in bin k. Replace any fraction below 1e-6 with 1e-6, so empty bins do not give log(0).
4. PSI = sum_k (a_k - e_k) * ln(a_k / e_k)

A common rule of thumb: below 0.1 is stable, 0.1 to 0.25 is a moderate shift, and above 0.25 is a large shift.

Function signature

def population_stability_index(expected, actual, bins=10):

Example

Input

expected = np.arange(1.0, 9.0)        # 1, 2, ..., 8
actual   = np.arange(2.0, 10.0)       # 2, 3, ..., 9
population_stability_index(expected, actual, bins=4)

Output

0.13732654

Reasoning

The edges are the quartiles of expected: 2.75, 4.5 and 6.25. Each bin holds 2 of the 8 expected values, so e = [0.25, 0.25, 0.25, 0.25]. The actual values fall as [1, 2, 2, 3] per bin, so a = [0.125, 0.25, 0.25, 0.375]. Only the first and last bins add to the sum: (-0.125) * ln(0.5) + 0.125 * ln(1.5) = 0.0866 + 0.0507 = 0.137.

Constraints and notes

• expected and actual are 1D arrays and can have different lengths.
• PSI is 0 when both sets have the same share in every bin.