diff options
| author | Jonas Knobloch <jonas.knobloch@t-online.de> | 2026-03-24 13:02:49 +0100 |
|---|---|---|
| committer | Jonas Knobloch <jonas.knobloch@t-online.de> | 2026-03-24 15:40:19 +0100 |
| commit | ac670c698540488c818a17c3749abf3ed4cb764a (patch) | |
| tree | beaa4102bdbd24d2568431b0791dc6b780c1dd22 /go.work | |
| parent | fc9be8f81f673c0b04a0381a544b78e5a2a0d0bb (diff) | |
Add tokenizer module
* Wrap MBPE tokenizer
Diffstat (limited to 'go.work')
| -rw-r--r-- | go.work | 1 |
1 files changed, 1 insertions, 0 deletions
@@ -6,5 +6,6 @@ use ( ./llm ./mbpe ./tensor + ./tokenizer ./tui ) |