Galunid daab3d7f45 Add more tokenizer tests (#3742) ...

* Add more tokenizer tests

* Add starcoder

* Update test vocab files

* Restrict bpe tokenizer tests to unicode planes

* Update comment

* Comment cosmetics

* Remove bloom vocab/test

2023-10-24 09:17:17 +02:00

..

CMakeLists.txt

Add more tokenizer tests (#3742)

2023-10-24 09:17:17 +02:00

test-c.c

…

test-double-float.cpp

gguf : support big endian platform (#3552)

2023-10-20 14:19:40 +03:00

test-grad0.cpp

sync : ggml (conv 1d + 2d updates, UB fixes) (#3468)

2023-10-04 15:29:58 +03:00

test-grammar-parser.cpp

…

test-llama-grammar.cpp

…

test-opt.cpp

sync : ggml (conv 1d + 2d updates, UB fixes) (#3468)

2023-10-04 15:29:58 +03:00

test-quantize-fns.cpp

…

test-quantize-perf.cpp

sync : ggml (conv 1d + 2d updates, UB fixes) (#3468)

2023-10-04 15:29:58 +03:00

test-rope.cpp

…

test-sampling.cpp

sampling : refactor init to use llama_sampling_params (#3696)

2023-10-20 21:07:23 +03:00

test-tokenizer-0-falcon.cpp

Minor improvements in GPT2 tokenizer (#3567)

2023-10-10 18:59:52 +02:00

test-tokenizer-0-falcon.py

Minor improvements in GPT2 tokenizer (#3567)

2023-10-10 18:59:52 +02:00

test-tokenizer-0-llama.cpp

Minor improvements in GPT2 tokenizer (#3567)

2023-10-10 18:59:52 +02:00

test-tokenizer-0-llama.py

Minor improvements in GPT2 tokenizer (#3567)

2023-10-10 18:59:52 +02:00

test-tokenizer-1-bpe.cpp

Add more tokenizer tests (#3742)

2023-10-24 09:17:17 +02:00

test-tokenizer-1-llama.cpp

…