tqcq/llama.cpp
0
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-06-28 12:25:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
dd0dc366dab10e8df28d3924e7f313b5c695e908
llama.cpp/examples/perplexity
History
Kawrakow 463173a6c0 llama : speedup tokenization (#2831) 
* Speedup tokenization

On current master it takes ~3.2 seconds to tokenize
Wikitext. With this change it becomes ~525 ms.

* Fixit: it was missing the piece after the last found occurence

---------

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
2023-08-27 16:50:33 +03:00
..
CMakeLists.txt
cmake : install targets (#2256)
2023-07-19 10:01:11 +03:00
perplexity.cpp
llama : speedup tokenization (#2831)
2023-08-27 16:50:33 +03:00
README.md
Fix whitespace, add .editorconfig, add GitHub workflow (#883)
2023-04-11 19:45:44 +00:00

README.md

perplexity

TODO