Creating an initial Quantization Directory (#863)

* Initial Creation of a quantization directory

* Moving qops

* updating import

* Updating lm_eval version (#865)

Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets

* Pinning numpy to under 2.0 (#867)

* Update Quant call using llama.cpp (#868)

llama.cpp did a BC breaking refactor: ggml-org/llama.cpp@1c641e6
resulting in some of our CI breaking

This updates our CI to match llama.cpp's schema

* Updating torch nightly to pick up aoti improvements in 128339 (#862)

* Updating torch nightly to pick up aoti improvements in 128339

* Update the torch version to 2.5

* Updating lm_eval version (#865)

Fixing CI related to EleutherAI/wikitext_document_level change requirements from using HF Datasets

* Pinning numpy to under 2.0 (#867)

Author: Jack-Khuu

build/builder.py

@@ -17,7 +17,7 @@
 import torch._inductor.config
 
 from config.model_config import resolve_model_config
-from quantize import quantize_model
+from quantization.quantize import quantize_model
 
 from build.model import Transformer
 from build.utils import device_sync, is_cpu_device, is_cuda_or_cpu_device, name_to_dtype

build/gguf_loader.py

@@ -14,7 +14,7 @@
 import torch
 
 from gguf import GGUFValueType
-from quantize import pack_scales_and_zeros, WeightOnlyInt4Linear
+from quantization.quantize import pack_scales_and_zeros, WeightOnlyInt4Linear
 
 from build.gguf_util import Q4_0, to_float
 from build.model import ModelArgs, Transformer

build/gguf_util.py

@@ -6,7 +6,7 @@
 
 import gguf
 import torch
-from quantize import group_dequantize_tensor_from_qparams
+from quantization.quantize import group_dequantize_tensor_from_qparams
 
 
 def to_float(t: gguf.gguf_reader.ReaderTensor):

docs/quantization.md

@@ -96,7 +96,7 @@ for valid `bitwidth` and `groupsize` values.
 | linear with HQQ (asymmetric) |`'{"linear:hqq" : {"groupsize" : <groupsize>}}'`|
 | embedding | `'{"embedding": {"bitwidth": <bitwidth>, "groupsize":<groupsize>}}'` |
 
-See the available quantization schemes [here](https://github.com/pytorch/torchchat/blob/main/quantize.py#L1260-L1266).
+See the available quantization schemes [here](https://github.com/pytorch/torchchat/blob/main/quantization/quantize.py#L1260-L1266).
 
 ## Examples
 We can mix and match weight quantization with embedding quantization.

qops.py renamed to quantization/qops.py

@@ -390,7 +390,7 @@ def _check_k(cls, *, k, groupsize=1, inner_k_tiles=1):
     def _prepare_weight_and_scales_and_zeros(
         cls, weight_bf16, groupsize, inner_k_tiles
     ):
-        from quantize import group_quantize_tensor
+        from quantization.quantize import group_quantize_tensor
 
         weight_int32, scales_and_zeros = group_quantize_tensor(
             weight_bf16, n_bit=4, groupsize=groupsize

quantize.py renamed to quantization/quantize.py

@@ -23,7 +23,7 @@
     state_dict_device,
 )
 
-from qops import (
+from quantization.qops import (
     LinearAct8Int4DQ,
     LinearInt4 as WeightOnlyInt4Linear,
     LinearInt8 as WeightOnlyInt8Linear,