summaryrefslogtreecommitdiff
path: root/tokenizer
Commit message (Collapse)AuthorAgeFilesLines
* Add lesci moduleJonas Knobloch2026-04-142-0/+34
|
* Return document index in text iteratorJonas Knobloch2026-04-031-5/+1
|
* Scope text column to parquet readerJonas Knobloch2026-04-031-1/+1
|
* Handle unknown runesJonas Knobloch2026-03-313-1/+119
|
* Sort importsJonas Knobloch2026-03-251-0/+1
|
* Tidy dependenciesJonas Knobloch2026-03-252-1/+198
|
* Preserve per-byte error replacementsJonas Knobloch2026-03-244-0/+38
|
* Remove input bufferJonas Knobloch2026-03-242-36/+66
|
* Expose finite state automatonJonas Knobloch2026-03-244-1/+320
|
* Add tokenizer moduleJonas Knobloch2026-03-244-0/+116
* Wrap MBPE tokenizer