tqcq/llama.cpp
0
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-07-16 07:38:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

CUDA: quantized KV support for FA vec (#7527)

Browse Source 
* CUDA: quantized KV support for FA vec

* try CI fix

* fix commented-out kernel variants

* add q8_0 q4_0 tests

* fix nwarps > batch size

* split fattn compile via extern templates

* fix flake8

* fix metal tests

* fix cmake

* make generate_cu_files.py executable

* add autogenerated .cu files

* fix AMD

* error if type_v != FP16 and not flash_attn

* remove obsolete code
This commit is contained in:
Johannes Gäßler
2024-06-01 08:44:14 +02:00
committed by GitHub
parent a323ec60af
commit 9b596417af
110 changed files with 2697 additions and 1200 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
ggml-cuda/template-instances/fattn-vec-f16-instance-hs64-f16-f16.cu Normal file
View File

@ -0,0 +1,5 @@
// This file has been autogenerated by generate-variants.py, do not edit manually.
#include "../fattn-vec-f16.cuh"
DECL_FATTN_VEC_F16_CASE(64, GGML_TYPE_F16, GGML_TYPE_F16);