Skip to content

scripts: fix compare-llama-bench commit hash logic #11891

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

JohannesGaessler
Copy link
Collaborator

Currently compare-llama-bench.py assumes that all rows have the same commit hash length and simply retrieves this length from the first row. This change seems to have been made in #10714 without further documentation as to why, I assume it's because the llama.cpp commit hash has a variable length. By default it's 8 hexadecimal characters but if there is a hash collision the length of git rev-parse --short is increased to yield a unique prefix. On the master branch this does not seem to be necessary but on a local repository the hash can be longer. compare-llama-bench.py thus may receive hashes that are longer than 8 characters so the trimming to 8 characters that I originally implemented can fail. However, for my workflows this fix does not work. I have a desktop where I do most of my development and on this machine 9 characters are needed for unique hashes. However, I have other machines that I primarily use for testing/batch jobs where the hash is only 8 characters. So sqlite files that contain data from multiple machines (or from the same machine when the hash length changes) may not work correctly. This PR generalizes the logic to cover this use case by shortening the prefixes and making them comparable. It is in principle possible that this will cause incorrect results due to hash collisions but the probability is extremely low. Still, if this is necessary the script prints a warning, recommending that old commits be purged.

@github-actions github-actions bot added script Script related python python script changes labels Feb 15, 2025
@JohannesGaessler JohannesGaessler merged commit 6dde178 into ggml-org:master Feb 15, 2025
4 checks passed
orca-zhang pushed a commit to orca-zhang/llama.cpp that referenced this pull request Feb 26, 2025
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Feb 26, 2025
mglambda pushed a commit to mglambda/llama.cpp that referenced this pull request Mar 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python python script changes script Script related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants