Qualcomm AI Engine Direct - Update the evaluator API call for Llama

[winskuo-quic]

Summary

Due to the changes introduced in add option to run mmlu with 5 shots by helunwencser · Pull Request #6146 · pytorch/executorch (github.com), the evaluate_model function has been removed. Please use simple_evaluate for evaluation purposes.

PR Owner: YuYang Zhuang

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6386

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures

As of commit 7f26d31 with merge base ad0e5e8 ():

NEW FAILURES - The following jobs have failed:

• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 376e9d8d4dcf7107f78eda2904aeacab8333f868991f477f2d7278cc71470076 /exec failed with exit code 127
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t b3b2d865997bead138ed73af805230123fc4e3336939bc0c64f8cc810dfe3d81 /exec failed with exit code 127
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t c5d3bf7a84b861773bb34ca07a4e5b0d2a9226f9b389537c9980eb71c6bba74e /exec failed with exit code 127

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[winskuo-quic]

Hi @cccclai,
This PR is to use simple_evaluate for evaluation purposes since evaluate_model function has been removed.
Please have a look.
Thanks

[cccclai]

Thanks - we should set up some simple CI for quantized model, maybe at least 16a16w, so it can be caught early

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.