Brian f7cab35ef9 gguf-hash: model wide and per tensor hashing using xxhash and sha1 (#8048) ...

CLI to hash GGUF files to detect difference on a per model and per tensor level

The hash type we support is:

- `--xxh64`: use xhash 64bit hash mode (default)
- `--sha1`: use sha1
- `--uuid`: use uuid
- `--sha256`: use sha256

While most POSIX systems already have hash checking programs like sha256sum, it
is designed to check entire files. This is not ideal for our purpose if we want
to check for consistency of the tensor data even if the metadata content of the
gguf KV store has been updated.

This program is designed to hash a gguf tensor payload on a 'per tensor layer'
in addition to a 'entire tensor model' hash. The intent is that the entire
tensor layer can be checked first but if there is any detected inconsistencies,
then the per tensor hash can be used to narrow down the specific tensor layer
that has inconsistencies.

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

2024-07-07 22:58:43 +10:00

..

package.json

gguf-hash: model wide and per tensor hashing using xxhash and sha1 (#8048)

2024-07-07 22:58:43 +10:00

sha1.c

gguf-hash: model wide and per tensor hashing using xxhash and sha1 (#8048)

2024-07-07 22:58:43 +10:00

sha1.h

gguf-hash: model wide and per tensor hashing using xxhash and sha1 (#8048)

2024-07-07 22:58:43 +10:00