Loading
Information theory as machine learning uses it. The course starts with entropy as the cost of a code, then covers conditional entropy and information gain, mutual information, divergences between distributions, cross entropy as a training loss, and how to estimate information with a learnt critic.
01
Entropy is the average number of bits needed to encode samples from a distribution with the best possible code. Huffman coding builds a code that comes within one bit of it.
02
Conditional entropy is the uncertainty left in one variable once you know another. Information gain is how much a split reduces it, which is how decision trees choose their splits.
03
Mutual information is how much knowing one variable tells you about another. It is zero only for independent variables and sees any kind of dependence.
04
The KL divergence is the extra cost of coding one distribution with a code built for another, and it is not symmetric. The Jensen-Shannon divergence compares both to their average, so it is symmetric and bounded.
05
Cross entropy is the loss that trains classifiers and language models. Perplexity is its exponential, the number of equally likely choices the model is unsure between.
06
For continuous, high dimensional data, mutual information can be bounded by training a critic to tell real pairs from shuffled ones. The InfoNCE bound behind contrastive learning works this way.