Skip to content

Update llama2.c converter to read vocab and write models in GGUF format #2751

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Aug 27, 2023
Merged

Update llama2.c converter to read vocab and write models in GGUF format #2751

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5132130
llama2.c: direct gguf output (WIP)
Aug 23, 2023
463e117
Simplify vector building logic
Aug 23, 2023
63174b8
llama2.c gguf conversion: fix token types in converter
Aug 26, 2023
c88362d
llama2.c: support copying vocab from a llama gguf model file
Aug 26, 2023
df3b81a
llama2.c: update default path for vocab model + readme
Aug 26, 2023
20c4471
llama2.c: use defines for gguf keys
Aug 26, 2023
0722e58
llama2.c: escape whitespaces w/ U+2581 in vocab converter the llama.c…
Aug 26, 2023
c58792c
llama2.c converter: cleanups + take n_ff from config
Aug 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions examples/convert-llama2c-to-ggml/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,14 @@ usage: ./convert-llama2c-to-ggml [options]

options:
-h, --help show this help message and exit
--copy-vocab-from-model FNAME model path from which to copy vocab (default 'tokenizer.bin')
--copy-vocab-from-model FNAME path of gguf llama model or llama2.c vocabulary from which to copy vocab (default 'models/7B/ggml-model-f16.gguf')
--llama2c-model FNAME [REQUIRED] model path from which to load Karpathy's llama2.c model
--llama2c-output-model FNAME model path to save the converted llama2.c model (default ak_llama_model.bin')
```

An example command using a model from [karpathy/tinyllamas](https://huggingface.co/karpathy/tinyllamas) is as follows:

`$ ./convert-llama2c-to-ggml --copy-vocab-from-model ../llama2.c/tokenizer.bin --llama2c-model stories42M.bin --llama2c-output-model stories42M.ggmlv3.bin`

For now the generated model is in the legacy GGJTv3 format, so you need to convert it to gguf manually:

`$ python ./convert-llama-ggmlv3-to-gguf.py --eps 1e-5 --input stories42M.ggmlv3.bin --output stories42M.gguf.bin`
`$ ./convert-llama2c-to-ggml --copy-vocab-from-model llama-2-7b-chat.gguf.q2_K.bin --llama2c-model stories42M.bin --llama2c-output-model stories42M.gguf.bin`

Now you can use the model with a command like:

Expand Down
Loading