The guide
Entropy, from the probability topic, measures the uncertainty of one variable. This topic is about the quantities built from it, and they turn up all over machine learning under other names.
Conditional entropy and mutual information measure how much one variable tells you about another, and decision trees choose their splits with exactly that. Perplexity is the exponent of cross entropy, and it is the number every language model reports. Jensen-Shannon divergence is what the original GAN objective minimises.
The Huffman problem shows the other side of entropy: it is a hard floor on compression. No code can use fewer bits per symbol on average, and a good code gets close.