x/tokenizer/go.mod, branch wip-frequency
x
http://git.jknobloc.com/x/atom/tokenizer/go.mod?h=wip-frequency
2026-06-27T13:06:57Z
Bump github.com/jonasknobloch/mbpe to v0.1.2
2026-06-27T13:06:57Z
Jonas Knobloch
jonas.knobloch@t-online.de
2026-06-27T13:06:57Z
urn:sha1:b1ba8ff474afc003e956de7bf371887859fb1df1
Tidy dependencies
2026-03-25T13:04:35Z
Jonas Knobloch
jonas.knobloch@t-online.de
2026-03-25T13:04:35Z
urn:sha1:4127e0440dc6821b990064704314b0a9891caa72
Add tokenizer module
2026-03-24T14:40:19Z
Jonas Knobloch
jonas.knobloch@t-online.de
2026-03-24T12:02:49Z
urn:sha1:ac670c698540488c818a17c3749abf3ed4cb764a
* Wrap MBPE tokenizer