Skip to content

Update readme examples to use newer Qwen2 model #1544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 21, 2024
Merged

Update readme examples to use newer Qwen2 model #1544

merged 1 commit into from
Jun 21, 2024

Conversation

jncraton
Copy link
Contributor

Qwen2 generally outperforms Qwen1.5 according to released benchmarks:

Datasets Qwen1.5-0.5B-Chat Qwen2-0.5B-Instruct
MMLU 35.0 37.9
HumanEval 9.1 17.1
GSM8K 11.3 40.1
C-Eval 37.2 45.2
IFEval (Prompt Strict-Acc.) 14.6 20.0

This model is working fine for me, and I would expect it to generally provide a better experience for users exploring this package. I would expect speed to be nearly identical given the similar sizes of these two models.

@abetlen abetlen merged commit 27d5358 into abetlen:main Jun 21, 2024
@abetlen
Copy link
Owner

abetlen commented Jun 21, 2024

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants