x/tokenizer/bpe/cmd, branch wip-frequency x http://git.jknobloc.com/x/atom/tokenizer/bpe/cmd?h=wip-frequency 2026-05-13T12:06:41Z Refactor profiling helpers into module 2026-05-13T12:06:41Z Jonas Knobloch jonas.knobloch@t-online.de 2026-05-13T12:06:12Z urn:sha1:1a2ab17200ed5a529d16548377fcb0ba446dc28e Recover tokenization based on config 2026-04-24T09:50:02Z Jonas Knobloch jonas.knobloch@t-online.de 2026-04-24T09:49:26Z urn:sha1:12f15fdd8c048e5958e294f12a9f163f6f950b3e Access artifacts via shelf 2026-04-24T09:07:01Z Jonas Knobloch jonas.knobloch@t-online.de 2026-04-24T09:07:01Z urn:sha1:1e02a5ec7f7794bb542e99ceb804ed88048b6b54 Return document index in text iterator 2026-04-03T18:06:42Z Jonas Knobloch jonas.knobloch@t-online.de 2026-04-03T17:20:57Z urn:sha1:fda1267d7d6600c04a6cd7eb2a818ebefb8999e4 Scope text column to parquet reader 2026-04-03T16:45:57Z Jonas Knobloch jonas.knobloch@t-online.de 2026-04-03T14:02:05Z urn:sha1:c1644da32ecb9856f15e2c919a84d2666d5ef29c Handle unknown runes 2026-03-31T15:29:27Z Jonas Knobloch jonas.knobloch@t-online.de 2026-03-31T14:21:28Z urn:sha1:ce9c65cb1e13f7ae6eed4b419232a4da9dd63fdb Expose finite state automaton 2026-03-24T14:42:10Z Jonas Knobloch jonas.knobloch@t-online.de 2026-03-24T14:37:50Z urn:sha1:f796560cb4de0b6de118c00be54c58206bad5b60 Add tokenizer module 2026-03-24T14:40:19Z Jonas Knobloch jonas.knobloch@t-online.de 2026-03-24T12:02:49Z urn:sha1:ac670c698540488c818a17c3749abf3ed4cb764a * Wrap MBPE tokenizer