Update quantize.py to use torchao Quantizers

[larryliu0820]

Summary:

Remove duplicate code for Int8DynActInt4WeightQuantizer and use torchao API.

Test Plan:

python torchchat.py generate llama2 --quantize '{"linear:a8w4dq": {"groupsize": 256}, "precision": {"dtype":"float16"}, "executor":{"accelerator":"cpu"}}' --prompt "Once upon a time," --max-new-tokens 256

Reviewers:

Subscribers:

Tasks:

Tags:

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/882

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job

As of commit f85339a with merge base ee681bf ():

CANCELLED JOB - The following job was cancelled. Please retry:

• Run the README instructions - with stories - on MPS/MacOS / test-readme-mps-macos / macos-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Jack-Khuu]

Looking good contextually, we'll want a stamp from the AO side though

[byjlw]

Can you update the readme and the --help to explain the --quantize flag?
Also when you pass --quantize during generate what exactly happens if the model on disk is fp16 and you want 8bit? Does it quantize then and there? Does it get saved for future use? We need enough information in the help and readme for users to know what to expect and how to do what they want.

[larryliu0820]

Can you update the readme and the --help to explain the --quantize flag? Also when you pass --quantize during generate what exactly happens if the model on disk is fp16 and you want 8bit? Does it quantize then and there? Does it get saved for future use? We need enough information in the help and readme for users to know what to expect and how to do what they want.

Good question we will make sure that is added. Basically I believe --quantize only has effect on the model in memory but not the one on disk, unless we explicitly ask generate.py to save it.

[jackzhxng]

I just pushed out a fix for the failing tests, all compile tests now pass for x86. We should be able to get this across the line now. Approved pending re-review from @Jack-Khuu and @byjlw

Early approval, I thought only the test that I fixed was failing. Needs more work to pass tests.

[Jack-Khuu]

Looks legit, I'll can check on the perf when it land

[larryliu0820]

@byjlw We will add documentation in next PR. Any other concerns?

Merge this PR asap