summaryrefslogtreecommitdiff
path: root/llm
Commit message (Collapse)AuthorAgeFilesLines
* Return document index in text iteratorJonas Knobloch2026-04-032-12/+13
|
* Scope text column to parquet readerJonas Knobloch2026-04-032-2/+2
|
* Use token buffer to generate jobsJonas Knobloch2026-04-032-111/+36
|
* Return relative offset in push iteratorJonas Knobloch2026-04-032-28/+32
|
* Refactor tail cleanupJonas Knobloch2026-04-031-13/+13
|
* Track already seen tokensJonas Knobloch2026-04-022-9/+75
|
* Add token bufferJonas Knobloch2026-04-022-0/+250
|
* Count skipped textsJonas Knobloch2026-03-311-2/+14
|
* Refactor llm moduleJonas Knobloch2026-03-252-0/+0
|
* Expose progress bar titleJonas Knobloch2026-03-254-6/+6
|
* Add helper to collect resultsJonas Knobloch2026-03-253-44/+39
|
* Refactor evaluator initializationJonas Knobloch2026-03-254-27/+9
|
* Sort importsJonas Knobloch2026-03-251-2/+1
|
* Tidy dependenciesJonas Knobloch2026-03-252-1/+17
|
* Refactor tokenizer usageJonas Knobloch2026-03-241-12/+7
|
* Refactor progress bar life cycleJonas Knobloch2026-03-241-6/+3
|
* Sync dependenciesJonas Knobloch2026-03-232-44/+26
|
* Add log probs extractorJonas Knobloch2026-03-174-35/+355
|
* Update perplexity evaluatorJonas Knobloch2026-03-179-208/+304
| | | | | | * Introduce results channel * Introduce jobs channel * Generalize evaluation via callback
* Rename dataset readerJonas Knobloch2026-03-162-4/+4
|
* Use vanity import pathJonas Knobloch2026-03-163-5/+7
|
* Add ppl Python scriptJonas Knobloch2026-03-091-0/+112
|
* Fix perplexity calculationJonas Knobloch2026-03-094-12/+31
| | | | | * Track total evaluated tokens * Use first token as context
* Expose window and strideJonas Knobloch2026-03-092-7/+7
|
* Add ppl commandJonas Knobloch2026-03-091-0/+69
|
* Use dataset packageJonas Knobloch2026-03-091-16/+13
|
* Update module pathsJonas Knobloch2026-02-281-1/+1
|
* Bump Go to 1.25.0Jonas Knobloch2026-02-101-1/+1
|
* Bump github.com/jonasknobloch/mbpe to v0.1.1Jonas Knobloch2026-02-103-1/+89
|
* Wait for progress barJonas Knobloch2025-11-261-0/+2
|
* Use seen tokens to warm up cacheJonas Knobloch2025-11-261-2/+2
|
* Add perplexity evaluationJonas Knobloch2025-11-185-0/+340
|
* Scaffold byte-pair correctionJonas Knobloch2025-11-173-0/+13
generated by cgit v1.3.1 (git 2.54.0) at 2026-10-07 04:03:20 +0000