tqcq/llama.cpp
0
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-06-29 20:45:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

llama : accept a list of devices to use to offload a model (#10497)

Browse Source 
* llama : accept a list of devices to use to offload a model

* accept `--dev none` to completely disable offloading

* fix dev list with dl backends

* rename env parameter to LLAMA_ARG_DEVICE for consistency
This commit is contained in:
Diego Devesa
2024-11-25 19:30:06 +01:00
committed by GitHub
parent 1f922254f0
commit 10bce0450f
9 changed files with 104 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
include/llama.h
View File

@ -272,6 +272,9 @@ extern "C" {
};
struct llama_model_params {
// NULL-terminated list of devices to use for offloading (if NULL, all available devices are used)
ggml_backend_dev_t * devices;
int32_t n_gpu_layers; // number of layers to store in VRAM
enum llama_split_mode split_mode; // how to split the model across multiple GPUs