Daniel Bevenius 4fbd8098e6 gguf : add special tokens metadata for FIM/Infill (#6689) ...

This commit adds special token metadata for Fill-In-the-Middle
(FIM)/Infill to the GGUF model.

The motivation for this is that currently there is support for CodeLlama
but other models exist now like CodeGemma, but the different models use
different token ids for the special tokens and this commit allows for
supporting multiple models.

Signed-off-by: Daniel Bevenius <daniel.bevenius@gmail.com>

2024-04-16 09:13:13 +03:00

..

__init__.py

gguf-py: Refactor and allow reading/modifying existing GGUF files (#3981)

2023-11-11 08:04:50 +03:00

constants.py

gguf : add special tokens metadata for FIM/Infill (#6689)

2024-04-16 09:13:13 +03:00

gguf_reader.py

gguf : add support for I64 and F64 arrays (#6062)

2024-03-15 10:46:51 +02:00

gguf_writer.py

gguf : add special tokens metadata for FIM/Infill (#6689)

2024-04-16 09:13:13 +03:00

gguf.py

gguf-py: Refactor and allow reading/modifying existing GGUF files (#3981)

2023-11-11 08:04:50 +03:00

py.typed

convert : various script cleanups/fixes + merges and special token handling (#2842)

2023-08-30 11:25:50 +03:00

tensor_mapping.py

model: support arch DbrxForCausalLM (#6515)

2024-04-13 11:33:52 +02:00

vocab.py

fix(gguf-py): special tokens are no longer skipped when add_<token>_token is set to false (#5487)

2024-02-15 14:14:37 +01:00