summaryrefslogtreecommitdiff
path: root/tokenizer/bpe/cmd/tokenize/main.go
Commit message (Collapse)AuthorAgeFilesLines
* Refactor profiling helpers into moduleJonas Knobloch2026-05-131-42/+4
|
* Recover tokenization based on configJonas Knobloch2026-04-241-1/+4
|
* Access artifacts via shelfJonas Knobloch2026-04-241-2/+3
|
* Return document index in text iteratorJonas Knobloch2026-04-031-5/+1
|
* Scope text column to parquet readerJonas Knobloch2026-04-031-1/+1
|
* Handle unknown runesJonas Knobloch2026-03-311-0/+2
|
* Expose finite state automatonJonas Knobloch2026-03-241-0/+45
|
* Add tokenizer moduleJonas Knobloch2026-03-241-0/+68
* Wrap MBPE tokenizer