diff options
| author | Jonas Knobloch <jonas.knobloch@t-online.de> | 2025-11-13 23:04:57 +0100 |
|---|---|---|
| committer | Jonas Knobloch <jonas.knobloch@t-online.de> | 2025-11-17 22:41:09 +0100 |
| commit | 7073b124c5eb31169442ef18d1627b7ed7401280 (patch) | |
| tree | f472b8a937fb5e2e3f235723cd5bafe422b3b647 /llm/tokenizer.go | |
| parent | 019a25a6082a8cde372304e34dcd0ae3d5e875ed (diff) | |
Scaffold byte-pair correction
Diffstat (limited to 'llm/tokenizer.go')
| -rw-r--r-- | llm/tokenizer.go | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/llm/tokenizer.go b/llm/tokenizer.go new file mode 100644 index 0000000..c6ea3c1 --- /dev/null +++ b/llm/tokenizer.go @@ -0,0 +1,5 @@ +package llm + +type Tokenizer interface { + Tokenize(s string) []int +} |