x/tokenizer/bpe/utility.go, branch main
x
http://git.jknobloc.com/x/atom/tokenizer/bpe/utility.go?h=main
2026-04-24T09:50:02Z
Recover tokenization based on config
2026-04-24T09:50:02Z
Jonas Knobloch
jonas.knobloch@t-online.de
2026-04-24T09:49:26Z
urn:sha1:12f15fdd8c048e5958e294f12a9f163f6f950b3e
Add lesci module
2026-04-14T17:38:34Z
Jonas Knobloch
jonas.knobloch@t-online.de
2026-04-10T23:09:16Z
urn:sha1:0c97f22069d77cf3cac92d68173e38f83b258a9a
Expose finite state automaton
2026-03-24T14:42:10Z
Jonas Knobloch
jonas.knobloch@t-online.de
2026-03-24T14:37:50Z
urn:sha1:f796560cb4de0b6de118c00be54c58206bad5b60
Add tokenizer module
2026-03-24T14:40:19Z
Jonas Knobloch
jonas.knobloch@t-online.de
2026-03-24T12:02:49Z
urn:sha1:ac670c698540488c818a17c3749abf3ed4cb764a
* Wrap MBPE tokenizer