llama : refactor unicode stuff (#5992)

* llama : refactor unicode stuff ggml-ci * unicode : names * make : fix c++ compiler * unicode : names * unicode : straighten tables * zig : fix build * unicode : put nfd normalization behind API ggml-ci * swift : fix build * unicode : add BOM * unicode : add <cstdint> ggml-ci * unicode : pass as cpts as const ref
2025-08-30 11:59:59 -04:00 · 2024-03-11 17:47:47 +02:00
parent 828defefb6
commit 83796e62bc
9 changed files with 1744 additions and 836 deletions
--- a/Package.swift
+++ b/Package.swift
@@ -31,6 +31,7 @@ let package = Package(
            sources: [
                "ggml.c",
                "llama.cpp",
+                "unicode.cpp",
                "ggml-alloc.c",
                "ggml-backend.c",
                "ggml-quants.c",