diff options
| author | Jonas Knobloch <jonas.knobloch@t-online.de> | 2026-03-24 13:02:49 +0100 |
|---|---|---|
| committer | Jonas Knobloch <jonas.knobloch@t-online.de> | 2026-03-24 15:40:19 +0100 |
| commit | ac670c698540488c818a17c3749abf3ed4cb764a (patch) | |
| tree | beaa4102bdbd24d2568431b0791dc6b780c1dd22 /tokenizer/go.mod | |
| parent | fc9be8f81f673c0b04a0381a544b78e5a2a0d0bb (diff) | |
Add tokenizer module
* Wrap MBPE tokenizer
Diffstat (limited to 'tokenizer/go.mod')
| -rw-r--r-- | tokenizer/go.mod | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/tokenizer/go.mod b/tokenizer/go.mod new file mode 100644 index 0000000..e66e1f8 --- /dev/null +++ b/tokenizer/go.mod @@ -0,0 +1,3 @@ +module go.jknobloc.com/x/tokenizer + +go 1.25 |