add proper calibration to pt2e flow

[cccclai]

Stack from ghstack (oldest at bottom):

• -> add proper calibration to pt2e flow #4452
• move get tokenizer to export_llama_lib #4451

Differential Revision: D60419364

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4452

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures

As of commit 290ca5d with merge base a743a3b ():

NEW FAILURES - The following jobs have failed:

• pull / test-export-llava-linux / linux-job (gh)
  RuntimeError: Command docker exec -t ea83bd7c92622e0bbab46a51f59ad7547b7a10faba824abe3d13296aecbfd37a /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, buck2, portable) / linux-job (gh)
  RuntimeError: Command docker exec -t 2f2326281ca7d4b0660b9f7414c3b5fea918e275e722861497bdd738f5fa8a0d /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, buck2, xnnpack+custom) / linux-job (gh)
  RuntimeError: Command docker exec -t 7afaf45d19eec471e0d4f350e1ffc64e7b3ae562b5cce5b2043173c1e156d4bc /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, buck2, xnnpack+custom+qe) / linux-job (gh)
  RuntimeError: Command docker exec -t a85f715c81b66c9a59edb2be53c5a0a8ec297dec1bfb68daba68a0f523794c61 /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, cmake, portable) / linux-job (gh)
  RuntimeError: Command docker exec -t 446e700165b43d9133e40130a0d3f4a5de07e0026e2e49a813ce5c3a02d44f22 /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, cmake, xnnpack+custom) / linux-job (gh)
  RuntimeError: Command docker exec -t 1c2aa6ef70387e016859837af8b769684393504c8357509648b05451bbb15750 /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, cmake, xnnpack+custom+qe) / linux-job (gh)
  RuntimeError: Command docker exec -t a27fa14c7e2e0bfcff9b5990f13314ce0bb34fc6372916deebce65edab0d0e7f /exec failed with exit code 1
• pull / unittest / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 2

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D60419364

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D60419364

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D60419364

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D60419364

[shewu-quic]

Hi @cccclai,
Thanks for this change.
It seems to have a problem when I capture seq_len = 1 in kv_cache mode but calibrate with whole sequence in batch process.
From my understanding, after we capture, some variable will be fixed such as "batch, seqlen, _ = x.shape".
If I am mistaken, please correct me.

[cccclai]

Hi thank you for checking out the pr! I pulled some changes from another local patch and haven't tested this pr properly. What you mentioned is true...I vaguelly remember I modified the code there to use kv cache instead, however it was too slow to calibrate (as you observed too)

[cccclai]

Create a new pr #5095 instead as this pr wasn't tested properly