summaryrefslogtreecommitdiff
path: root/tokenizer/bpe
Commit message (Expand)AuthorAgeFilesLines
* Handle unknown runesJonas Knobloch2026-03-313-1/+119
* Sort importsJonas Knobloch2026-03-251-0/+1
* Preserve per-byte error replacementsJonas Knobloch2026-03-244-0/+38
* Remove input bufferJonas Knobloch2026-03-242-36/+66
* Expose finite state automatonJonas Knobloch2026-03-244-1/+320
* Add tokenizer moduleJonas Knobloch2026-03-243-0/+113