do weight transform on cpu #508
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
mikekgfb
force-pushed
the
quantize_in_cpu
branch
from
8e60a72 to
59e6275
Compare
mikekgfb
force-pushed
the
quantize_in_cpu
branch
from
59e6275 to
fe3e608
Compare
jerryzh168
reviewed
Apr 27, 2024
| @@ -856,8 +870,8 @@ def create_quantized_state_dict(self): | |||
| weight.to(torch.float), self.groupsize, self.inner_k_tiles | |||
| ) | |||
| ) | |||
| weight_int4pack = weight_int4pack.to(device=self.device) | |||
| scales_and_zeros = scales_and_zeros.to(device=self.device) | |||
| weight_int4pack = weight_int4pack.to(device=dict_device) | |||
There was a problem hiding this comment.
so cpu packed weight and cuda packed weight are calling different int4mm kernels, and the weights prepared in one device may not be compatible with another (gives wrong results) as recently discovered by @HDCharles, and it's a silent error right now. have we done any evaluation on accuracy for this change (on cuda)?
There was a problem hiding this comment.
We have not done this, but @malfet and I have discussed this, and Intel had previously promised us an unpack routine - so we would be bale to unpack() the different formats. BTW, we need an unpack for the GPU packing format as well
malfet
pushed a commit
that referenced
this pull request
Jul 17, 2024
malfet
pushed a commit
that referenced
this pull request
Jul 17, 2024
malfet
pushed a commit
that referenced
this pull request
Jul 17, 2024
malfet
pushed a commit
that referenced
this pull request
Jul 17, 2024
malfet
pushed a commit
that referenced
this pull request
Jul 17, 2024
malfet
pushed a commit
that referenced
this pull request
Jul 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
to avoid OOM situations, do weight transform on cpu
load_state_dict does not have a map_location argument lich torch.load().
However, a quick check for mps suggested that when the state dict is loaded back into a model, it's placed on the device of the pre-existing device. If this does not work as expected in scenarios, it should be quick to write a new version of load_state_dict that takes a map_location or amend the APUI for the exixsting function....
We might also avoid instantiating the state dict altogether - this is inherited from gpt fast, to enable sharing of quantization algos between torchchat and gpt-fast.