tqcq/llama.cpp
0
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-07-27 19:53:42 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
04bab6b7da0ed2b0a57174a57784d602aabb6c10
llama.cpp/include
History
Michael Coppola 940362224d llama : add support for Tekken pre-tokenizer (#8579) 
* llama : Added support for Tekken pre-tokenizer (#8577)

Removed uneeded `vocab.tokenizer_clean_spaces` assignment

* llama : fix order of pre-tokenizers

* * Tekken pre-tokenizer no longer uses clean_up_tokenization_spaces
* Updated chkhsh for Tekken tokenizer

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2024-07-20 16:43:51 +03:00
..
llama.h
llama : add support for Tekken pre-tokenizer (#8579)
2024-07-20 16:43:51 +03:00