mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2025-06-27 03:55:20 +00:00
33eff40240
* server : (experimental) vision support via libmtmd * mtmd : add more api around mtmd_image_tokens * mtmd : add more api around mtmd_image_tokens * mtmd : ability to calc image hash * shared_ptr for mtmd_image_tokens * move hash to user-define ID (fixed) * abstract out the batch management * small fix * refactor logic adding tokens to batch * implement hashing image * use FNV hash, now hash bitmap instead of file data * allow decoding image embedding to be split into batches * rm whitespace * disable some features when mtmd is on * fix --no-mmproj-offload * mtmd_context_params no timings * refactor server_inp to server_tokens * fix the failing test case * init * wip * working version * add mtmd::bitmaps * add test target * rm redundant define * test: mtmd_input_chunks_free * rm outdated comment * fix merging issue * explicitly create mtmd::input_chunks * mtmd_input_chunk_copy * add clone() * improve server_input struct * clip : fix confused naming ffn_up and ffn_down * rm ffn_i/o/g naming * rename n_embd, n_ff * small fix * no check n_ff * fix detokenize * add const to various places * add warning about breaking changes * add c api * helper: use mtmd_image_tokens_get_n_pos * fix ctx_shift * fix name shadowing * more strict condition * support remote image_url * remote image_url log * add CI test * do not log base64 * add "has_multimodal" to /props * remove dangling image * speculative: use slot.cache_tokens.insert * Apply suggestions from code review Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * rm can_be_detokenized * on prmpt processing done, assert cache_tokens.size * handle_completions_impl returns void * adapt the new web ui * update docs and hot topics * rm assert * small fix (2) --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
70 lines
2.4 KiB
Markdown
70 lines
2.4 KiB
Markdown
# Multimodal
|
|
|
|
llama.cpp supports multimodal input via `libmtmd`. Currently, there are 2 tools support this feature:
|
|
- [llama-mtmd-cli](../tools/mtmd/README.md)
|
|
- [llama-server](../tools/server/README.md) via OpenAI-compatible `/chat/completions` API
|
|
|
|
To enable it, can use use one of the 2 methods below:
|
|
|
|
- Use `-hf` option with a [supported model](../../docs/multimodal.md)
|
|
- To load a model using `-hf` while disabling multimodal, use `--no-mmproj`
|
|
- To load a model using `-hf` while using a custom mmproj file, use `--mmproj local_file.gguf`
|
|
- Use `-m model.gguf` option with `--mmproj file.gguf` to specify text and multimodal projector respectively
|
|
|
|
By default, multimodal projector will be offloaded to GPU. To disable this, add `--no-mmproj-offload`
|
|
|
|
For example:
|
|
|
|
```sh
|
|
# simple usage with CLI
|
|
llama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF
|
|
|
|
# simple usage with server
|
|
llama-server -hf ggml-org/gemma-3-4b-it-GGUF
|
|
|
|
# using local file
|
|
llama-server -m gemma-3-4b-it-Q4_K_M.gguf --mmproj mmproj-gemma-3-4b-it-Q4_K_M.gguf
|
|
|
|
# no GPU offload
|
|
llama-server -hf ggml-org/gemma-3-4b-it-GGUF --no-mmproj-offload
|
|
```
|
|
|
|
## Pre-quantized models
|
|
|
|
These are ready-to-use models, most of them come with `Q4_K_M` quantization by default.
|
|
|
|
Replaces the `(tool_name)` with the name of binary you want to use. For example, `llama-mtmd-cli` or `llama-server`
|
|
|
|
NOTE: some models may require large context window, for example: `-c 8192`
|
|
|
|
```sh
|
|
# Gemma 3
|
|
(tool_name) -hf ggml-org/gemma-3-4b-it-GGUF
|
|
(tool_name) -hf ggml-org/gemma-3-12b-it-GGUF
|
|
(tool_name) -hf ggml-org/gemma-3-27b-it-GGUF
|
|
|
|
# SmolVLM
|
|
(tool_name) -hf ggml-org/SmolVLM-Instruct-GGUF
|
|
(tool_name) -hf ggml-org/SmolVLM-256M-Instruct-GGUF
|
|
(tool_name) -hf ggml-org/SmolVLM-500M-Instruct-GGUF
|
|
(tool_name) -hf ggml-org/SmolVLM2-2.2B-Instruct-GGUF
|
|
(tool_name) -hf ggml-org/SmolVLM2-256M-Video-Instruct-GGUF
|
|
(tool_name) -hf ggml-org/SmolVLM2-500M-Video-Instruct-GGUF
|
|
|
|
# Pixtral 12B
|
|
(tool_name) -hf ggml-org/pixtral-12b-GGUF
|
|
|
|
# Qwen 2 VL
|
|
(tool_name) -hf ggml-org/Qwen2-VL-2B-Instruct-GGUF
|
|
(tool_name) -hf ggml-org/Qwen2-VL-7B-Instruct-GGUF
|
|
|
|
# Qwen 2.5 VL
|
|
(tool_name) -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF
|
|
(tool_name) -hf ggml-org/Qwen2.5-VL-7B-Instruct-GGUF
|
|
(tool_name) -hf ggml-org/Qwen2.5-VL-32B-Instruct-GGUF
|
|
(tool_name) -hf ggml-org/Qwen2.5-VL-72B-Instruct-GGUF
|
|
|
|
# Mistral Small 3.1 24B (IQ2_M quantization)
|
|
(tool_name) -hf ggml-org/Mistral-Small-3.1-24B-Instruct-2503-GGUF
|
|
```
|