Update quantization.md (#659)

Update support tables and explanatory text.

Author: mikekgfb

docs/quantization.md

@@ -10,7 +10,7 @@ While quantization can potentially degrade the model's performance, the methods
 ### Weight Quantization
 | compression | FP Precision | bitwidth| group size | dynamic activation quantization | Eager | AOTI | ExecuTorch |
 |--|--|--|--|--|--|--|--|
-| linear (asymmetric) | fp32, fp16, bf16 | [8, 4]* | [32, 64, 128, 256]** | | ✅ | ✅ | ✅ |
+| linear (asymmetric) | fp32, fp16, bf16 | [8, 4]* | [32, 64, 128, 256]** | | ✅ | ✅ | 🚧 |
 | linear with dynamic activations (symmetric) | fp32^ | | [32, 64, 128, 256]** | a8w4dq | 🚧 |🚧 | ✅ |
 | linear with GPTQ*** (asymmetric) | | |[32, 64, 128, 256]**  | | ✅ | ✅ | ❌ |
 | linear with HQQ*** (asymmetric) | | |[32, 64, 128, 256]**  | | ✅ | ✅ | ❌ |
@@ -22,14 +22,23 @@ Due to the larger vocabulary size of llama3, we also recommend quantizing the em
 |--|--|--|--|--|--|--|--|
 | embedding (symmetric) | fp32, fp16, bf16 | [8, 4]* | [32, 64, 128, 256]** | | ✅ | ✅ | ✅ |
 
-^a8w4dq quantization scheme requires model to be converted to fp32, due to lack of support for fp16 and bf16.
+^ The a8w4dq quantization scheme requires inouts to be converted to fp32, due to lack of support for fp16 and bf16.
 
-*These are the only valid bitwidth options.
+* These are the only valid bitwidth options.
 
-**There are many valid group size options, including 512, 1024, etc. Note that smaller groupsize tends to be better for preserving model quality and accuracy, and larger groupsize for further improving performance. Set 0 for channelwise quantization.
+** There are many valid group size options, including 512, 1024, etc. Note that smaller groupsize tends to be better for preserving model quality and accuracy, and larger groupsize for further improving performance. Set 0 for channelwise quantization.
 
 *** [GPTQ](https://arxiv.org/abs/2210.17323) and [HQQ](https://mobiusml.github.io/hqq_blog/) are two different algorithms to address accuracy loss when using lower bit quantization. Due to HQQ relying on data/calibration free quantization, it tends to take less time to quantize model.
 
+## Quantization Profiles
+Torchchat quantization supports profiles with multiple settings such as accelerator, dtype, and quantization specified in a JSON file.  Four sample profiles are included wwith the torchchat distributin in config/data: `cuda.json`, `desktop.json`, `mobile.json`, `pi5.json` with profiles optimizing for execution on cuda, desktop, mobile and raspberry Pi devices.
+
+In addition to quantization recipes described below, the profiles also enable developers to specify the accelerator and dtype to be used.  
+
+At present torchchat supports the fast, cuda, mps, and cpu devices.  The default device in torchchat is "fast". The "fast" device is a virtual device that defaults to the fastest executor available in the system, selecting cuda, mps, and cpu in this order.
+
+At present torchchat supports the fast16, fast, bf16, fp16 and fp32 data types. The default data type for models is "fast16".  The "fast16" data type is a virtual data type that defaults to the best 16-bit floating point data type available on the selected device. The "fast" data type is a virtual data type that defaults to the best floating point data type available on the selected device.   ("Best" tangibly representing a combination of speed and accuracy.) 
+
 ## Quantization API
 Quantization options are passed in json format either as a config file (see [cuda.json](../config/data/cuda.json) and [mobile.json](../config/data/mobile.json)) or a JSON string.
 
@@ -73,7 +82,7 @@ python3 generate.py llama3 --dso-path llama3.dso  --prompt "Hello my name is"
 ```
 ### ExecuTorch
 ```
-python3 torchchat.py export llama3 --dtype fp32 --quantize '{"embedding": {"bitwidth": 4, "groupsize":32}, "linear:a8w4dq": {"groupsize" : 256}}' --output-pte-path llama3.pte
+python3 torchchat.py export llama3 --quantize '{"embedding": {"bitwidth": 4, "groupsize":32}, "linear:a8w4dq": {"groupsize" : 256}}' --output-pte-path llama3.pte
 
 python3 generate.py llama3 --pte-path llama3.pte  --prompt "Hello my name is"
 ```
@@ -82,12 +91,13 @@ python3 generate.py llama3 --pte-path llama3.pte  --prompt "Hello my name is"
 On top of quantizing models with integer quantization schemes mentioned above, models can be converted to lower bit floating point precision to reduce the memory bandwidth requirement and take advantage of higher density compute available. For example, many GPUs and some of the CPUs have good support for BFloat16 and Float16. This can be taken advantage of via `--dtype` arg as shown below.
 
 ```
-python3 generate.py --dtype [bf16 | fp16 | fp32] ...
-python3 export.py --dtype [bf16 | fp16 | fp32] ...
+python3 generate.py --dtype [ fast16 | fast | bf16 | fp16 | fp32] ...
+python3 export.py --dtype [ fast16 | fast | bf16 | fp16 | fp32] ...
 ```
 
-Unlike gpt-fast which uses bfloat16 as default, torchchat uses float32 as the default. As a consequence you will have to set to --dtype bf16 or --dtype fp16 on server / desktop for best performance.
-Support for FP16 and BF16 is limited in many embedded processors. Additional ExecuTorch support for 16-bit floating point types may be added in the future based on hardware support.
+Unlike gpt-fast which uses bfloat16 as default, torchchat uses the dtype "fast16" as the default. Torchchat will pick the appropriate 16-bit floating point type available and offering the best performance (for execution with Executorch, macOS/ARM and Linux/x86 platforms).  For macOS, support depends on the OS version, with versions starting with 14.0 supporting bfloat16 as support, and float16 for earlier OS version based on system support for these data types.  
+
+Support for FP16 and BF16 is limited in many embedded processors and -dtype fp32 may be required in some environments. Additional ExecuTorch support for 16-bit floating point types may be added in the future based on hardware support.
 
 ## Adding additional quantization schemes
 We invite contributors to submit established quantization schemes, with accuracy and performance results demonstrating soundness.