tqcq/llama.cpp
0
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-06-29 12:35:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

metal : optimize multi-sequence FA vec kernel (#13493)

Browse Source 
* batched-bench : fix pp batch contents

* metal : optimize multi-sequence FA vec kernel

ggml-ci
This commit is contained in:
Georgi Gerganov
2025-05-13 18:04:00 +03:00
committed by GitHub
parent 4f711afed5
commit c252e0c409
1 changed files with 5 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
ggml/src/ggml-metal/ggml-metal.metal
View File

@ -3887,6 +3887,11 @@ kernel void kernel_flash_attn_ext_vec(
sm[tiisg] = pm[ic + tiisg]; sm[tiisg] = pm[ic + tiisg];
} }
// skip -INF blocks
if (simd_max(sm[tiisg]) == -INFINITY) {
continue;
}
// Q*K^T // Q*K^T
{ {
// each simdgroup processes 1 query and NE (NW/NL) head elements // each simdgroup processes 1 query and NE (NW/NL) head elements