I can't figure out how to make it 4 bit

[Xanmal877]

I am trying to follow the guide and I can usually get all the way to

quantize the model to 4-bits (using q4_0 method)

./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin q4_0

without error. but there is no ./quantize! no visual folder for me and no file found for the code!

Steps.txt

log.txt

[mqy]

but there is no ./quantize

This is a bit fantastic! Did you run make?

[Xanmal877]

I apologize, I don't understand your first sentence.

I am on windows so I was using cmake and use I created the build folder using cmake.

[mqy]

Try this:

cd build
cmake ..
cmake --build . --config Release

Then have look at README.md, which contains several examples about how to build with cmake for difference purposes.

[jparmstr]

You say there is no error, but your log file says:

(base) PS C:\Users\kmmf8\Downloads\llama.cpp> ./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin q4_0 ./quantize : The term './quantize' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

Looking at the steps you're following, it looks like you built from source. In this case the executables should be in the build directory. Based on your log I can tell that you're in the directory "llama.cpp" not "llama.cpp\build\bin\Release". If you run the quantize command from the folder containing quantize.exe it should work.

[jparmstr]

Also, I'm not sure which model you're trying to quantize, but there are tons of quantized GGML format models available here:
https://huggingface.co/TheBloke

[Xanmal877]

I will try this when I get home. Thank you for giving good advice. I'll set the source folder to build and try it from there.