tqcq/llama.cpp
0
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-07-26 19:23:37 -04:00
Code Issues Packages Projects Releases Wiki Activity
3,168 Commits 465 Branches 4,030 Tags
c637fcd34d135a9ff4f97d3a53ad03a910a4a31f
Commit Graph

3 Commits

Author SHA1 Message Date
Johannes Gäßler
1f0dabda8d CUDA: use tensor cores for MMQ (#7676) 
* CUDA: int8 tensor cores for MMQ (legacy quants)

* fix out-of-bounds writes

* __builtin_assume -> GGML_CUDA_ASSUME

* fix writeback returning too early
 2024-06-10 11:45:13 +02:00
Johannes Gäßler
750f60c03e CUDA: fix Pascal FA, deq. KV to FP16 for batch > 8 (#7681) 2024-06-01 15:47:04 +02:00
Johannes Gäßler
9b596417af CUDA: quantized KV support for FA vec (#7527) 
* CUDA: quantized KV support for FA vec

* try CI fix

* fix commented-out kernel variants

* add q8_0 q4_0 tests

* fix nwarps > batch size

* split fattn compile via extern templates

* fix flake8

* fix metal tests

* fix cmake

* make generate_cu_files.py executable

* add autogenerated .cu files

* fix AMD

* error if type_v != FP16 and not flash_attn

* remove obsolete code
 2024-06-01 08:44:14 +02:00