Character Tokenizer

Build the smallest useful tokenizer: one token per character.

From a training corpus, collect every distinct character, sort them, and assign ids starting at 0. Reserve the ids before the characters for two special tokens:

0 -> "<bos>"   beginning of sequence
1 -> "<eos>"   end of sequence

So the first real character gets id 2.

Implement three functions:

• build_vocab(text) returns (stoi, itos), a dict from character to id and a list from id to character. itos[i] and stoi[c] must be inverses.
• encode(text, stoi) returns a list of ids wrapped in <bos> and <eos>.
• decode(ids, itos) returns the string, dropping any special tokens.

Sorting the characters is what makes the vocabulary reproducible. Iterating a set gives a different order per run, and a vocabulary that changes order silently invalidates every trained embedding table.

Function signature

def build_vocab(text):
def encode(text, stoi):
def decode(ids, itos):

Example

Input

stoi, itos = build_vocab("ab")
encode("ba", stoi)
decode([0, 3, 2, 1], itos)

Output

[0, 3, 2, 1]
"ba"

Reasoning

Ids 0 and 1 are reserved, so "a" gets 2 and "b" gets 3. encode wraps "ba" in <bos> and <eos>, and decode drops them again.

Constraints and notes

• Sort the characters so the vocabulary is the same on every run.
• stoi and itos must be inverses of each other.
• decode skips the special tokens.