Loading
How text becomes numbers that a model can use. The course covers tokenisers, word counts and TF-IDF, word embeddings, padding and masks, the position signal that attention needs, and the string and translation metrics that score text.
01
A tokeniser splits text into units and gives each unit an id. Byte-pair encoding learns its units by merging the most frequent pair again and again.
02
A bag of words counts each word and ignores order. TF-IDF then gives more weight to words that are frequent in one document but rare in the collection.
03
Word2vec learns a vector for each word from the words around it. Skip-gram with negative sampling trains it as binary classification of true and random pairs.
04
A batch of texts is padded to one length. A mask then keeps the padding out of every sum, mean and attention weight.
05
Attention treats its input as a set, so the model needs a separate signal for word order. Sinusoidal encodings give each position a unique pattern of waves.
06
Edit distance counts the fewest insertions, deletions and substitutions that turn one string into another. Dynamic programming fills a table of distances between prefixes.
Practise it
07
BLEU compares the n-grams of a candidate with those of reference texts. Clipped precision and a brevity penalty stop short or repetitive output from scoring well.
Practise it