x/tokenizer, branch main x Bump github.com/jonasknobloch/mbpe to v0.1.2 2026-06-27T13:06:57+00:00 Jonas Knobloch jonas.knobloch@t-online.de 2026-06-27T13:06:57+00:00 b1ba8ff474afc003e956de7bf371887859fb1df1 






Refactor profiling helpers into module
2026-05-13T12:06:41+00:00

Jonas Knobloch
jonas.knobloch@t-online.de

2026-05-13T12:06:12+00:00

1a2ab17200ed5a529d16548377fcb0ba446dc28e












Recover tokenization based on config
2026-04-24T09:50:02+00:00

Jonas Knobloch
jonas.knobloch@t-online.de

2026-04-24T09:49:26+00:00

12f15fdd8c048e5958e294f12a9f163f6f950b3e












Access artifacts via shelf
2026-04-24T09:07:01+00:00

Jonas Knobloch
jonas.knobloch@t-online.de

2026-04-24T09:07:01+00:00

1e02a5ec7f7794bb542e99ceb804ed88048b6b54












Implement decoding
2026-04-22T10:54:45+00:00

Jonas Knobloch
jonas.knobloch@t-online.de

2026-04-21T17:49:54+00:00

d72bda93664657c341f605a9b238dcb88d63dcf8












Track state in local variable
2026-04-22T10:53:35+00:00

Jonas Knobloch
jonas.knobloch@t-online.de

2026-04-21T11:57:56+00:00

d2cf0b9bcf8ad3ab621b8ce7830fa40e203ec4d6












Fix FSA statics
2026-04-22T10:51:53+00:00

Jonas Knobloch
jonas.knobloch@t-online.de

2026-04-20T12:53:13+00:00

f28f740f3385c3bd9f1620776e7ca09d21ea03c4

* Add missing 've match





* Add missing 've match







Add sander module
2026-04-17T14:07:14+00:00

Jonas Knobloch
jonas.knobloch@t-online.de

2026-04-17T14:07:14+00:00

3fe37224ccc2500e7e767e5fc00163ef61182828












Add token mapping helpers
2026-04-17T13:48:10+00:00

Jonas Knobloch
jonas.knobloch@t-online.de

2026-04-17T13:48:10+00:00

4d8f647bfb54899e3052931790e288da7af97050












Fix reachable merges validator
2026-04-14T17:44:06+00:00

Jonas Knobloch
jonas.knobloch@t-online.de

2026-04-14T17:44:06+00:00

d9bed216c45aa1e0b817f4bccc7969bdac1adbff

* Previous implementation ignored byte replacements





* Previous implementation ignored byte replacements