tqcq/llama.cpp
0
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-09-03 05:39:25 -04:00
Code Issues Packages Projects Releases Wiki Activity

quantize : add '--keep-split' to quantize model into shards (#6688)

Browse Source 
* Implement '--keep-split' to quantize model into several shards

* Add test script

* Update examples/quantize/quantize.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Split model correctly even if tensor id is out-of-order

* Update llama_model_quantize_params

* Fix preci failures

---------

Co-authored-by: z5269887 <z5269887@unsw.edu.au>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit is contained in:
jiez
2024-04-25 18:29:35 +08:00
committed by GitHub
parent 784e11dea1
commit 1966eb2615
4 changed files with 148 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
llama.h
View File

@@ -288,6 +288,7 @@ extern "C" {
bool quantize_output_tensor; // quantize output.weight
bool only_copy; // only copy tensors - ftype, allow_requantize and quantize_output_tensor are ignored
bool pure; // quantize all tensors to the default type
bool keep_split; // quantize to the same number of shards
void * imatrix; // pointer to importance matrix data
void * kv_overrides; // pointer to vector containing overrides
} llama_model_quantize_params;