jaime-m-p b43272afa2 Unicode codepoint flags for custom regexs (#7245) ...

* Replace CODEPOINT_TYPE_* with codepoint_flags
* Update and bugfix brute force random test
* Deterministic brute force random test
* Unicode normalization NFD
* Get rid of BOM

2024-05-18 01:09:13 +02:00

..

.gitignore

…

CMakeLists.txt

llama : lookup word in vocab before doing BPE merges (#7193)

2024-05-11 11:12:06 +03:00

get-model.cpp

…

get-model.h

…

run-json-schema-to-grammar.mjs

…

test-autorelease.cpp

…

test-backend-ops.cpp

ggml : add ggml_upscale_ext (ggml/814)

2024-05-15 13:23:33 +03:00

test-c.c

…

test-chat-template.cpp

…

test-double-float.cpp

…

test-grad0.cpp

…

test-grammar-integration.cpp

Add left recursion check: quit early instead of going into an infinite loop (#7083)

2024-05-14 15:25:56 +10:00

test-grammar-parser.cpp

…

test-json-schema-to-grammar.cpp

JSON: [key] -> .at(key), assert() -> GGML_ASSERT (#7143)

2024-05-08 21:53:08 +02:00

test-llama-grammar.cpp

…

test-model-load-cancel.cpp

…

test-opt.cpp

…

test-quantize-fns.cpp

…

test-quantize-perf.cpp

…

test-rope.cpp

…

test-sampling.cpp

…

test-tokenizer-0.cpp

…

test-tokenizer-0.py

py : logging and flake8 suppression refactoring (#7081)

2024-05-05 08:07:48 +03:00

test-tokenizer-0.sh

…

test-tokenizer-1-bpe.cpp

llama : lookup word in vocab before doing BPE merges (#7193)

2024-05-11 11:12:06 +03:00

test-tokenizer-1-spm.cpp

…

test-tokenizer-random.py

Unicode codepoint flags for custom regexs (#7245)

2024-05-18 01:09:13 +02:00