summaryrefslogtreecommitdiff
path: root/tokenizer
Commit message (Expand)AuthorAgeFilesLines
* Refactor profiling helpers into moduleJonas Knobloch2026-05-131-42/+4
* Recover tokenization based on configJonas Knobloch2026-04-244-13/+31
* Access artifacts via shelfJonas Knobloch2026-04-241-2/+3
* Implement decodingJonas Knobloch2026-04-221-1/+9
* Track state in local variableJonas Knobloch2026-04-221-77/+37
* Fix FSA staticsJonas Knobloch2026-04-221-1/+1
* Add sander moduleJonas Knobloch2026-04-171-7/+36
* Add token mapping helpersJonas Knobloch2026-04-171-33/+27
* Fix reachable merges validatorJonas Knobloch2026-04-141-4/+29
* Add lesci moduleJonas Knobloch2026-04-142-0/+34
* Return document index in text iteratorJonas Knobloch2026-04-031-5/+1
* Scope text column to parquet readerJonas Knobloch2026-04-031-1/+1
* Handle unknown runesJonas Knobloch2026-03-313-1/+119
* Sort importsJonas Knobloch2026-03-251-0/+1
* Tidy dependenciesJonas Knobloch2026-03-252-1/+198
* Preserve per-byte error replacementsJonas Knobloch2026-03-244-0/+38
* Remove input bufferJonas Knobloch2026-03-242-36/+66
* Expose finite state automatonJonas Knobloch2026-03-244-1/+320
* Add tokenizer moduleJonas Knobloch2026-03-244-0/+116
generated by cgit v1.2.3 (git 2.25.1) at 2026-10-06 23:56:15 +0000