python : add check-requirements.sh and GitHub workflow (#4585)

* python: add check-requirements.sh and GitHub workflow

This script and workflow forces package versions to remain compatible
across all convert*.py scripts, while allowing secondary convert scripts
to import dependencies not wanted in convert.py.

* Move requirements into ./requirements

* Fail on "==" being used for package requirements (but can be suppressed)

* Enforce "compatible release" syntax instead of ==

* Update workflow

* Add upper version bound for transformers and protobuf

* improve check-requirements.sh

* small syntax change

* don't remove venvs if nocleanup is passed

* See if this fixes docker workflow

* Move check-requirements.sh into ./scripts/

---------

Co-authored-by: Jared Van Bortel <jared@nomic.ai>
This commit is contained in:
crasm
2023-12-29 09:50:29 -05:00
committed by GitHub
parent 68eccbdc5b
commit 04ac0607e9
16 changed files with 360 additions and 130 deletions

View File

@ -0,0 +1,2 @@
-r ./requirements-convert.txt
torch~=2.1.1

View File

@ -0,0 +1 @@
-r ./requirements-convert.txt

View File

@ -0,0 +1,2 @@
-r ./requirements-convert.txt
torch~=2.1.1

View File

@ -0,0 +1,2 @@
-r ./requirements-convert.txt
torch~=2.1.1

View File

@ -0,0 +1,5 @@
numpy~=1.24.4
sentencepiece~=0.1.98
transformers>=4.35.2,<5.0.0
gguf>=0.1.0
protobuf>=4.21.0,<5.0.0