Fixing exl2 and other quanize tests again. #2419
Conversation
| // #[deprecated( | ||
| // since = "1.1.0", | ||
| // note = "Use `eetq` instead, which provides better latencies overall and is drop-in in most cases" | ||
| // )] |
There was a problem hiding this comment.
We cna't keep that easily because Deserialize macro will use that field, and there's no way for us to insert #[allow(deprecated)] in the generated code.
There was a problem hiding this comment.
Would it be possible to add the deprecation in the Display impl instead? I think only Clap uses Display.
| Bitsandbytes, | ||
| /// Bitsandbytes 4bit. Can be applied on any model, will cut the memory requirement by 4x, | ||
| /// but it is known that the model will be much slower to run than the native f16. | ||
| BitsandbytesNF4, | ||
| BitsandbytesNf4, |
There was a problem hiding this comment.
This is the same for ValueEnum and make serde values transparent with kebab-case.
launcher/src/main.rs
Outdated
| ), | ||
| ) => { | ||
| tracing::info!("Bitsandbytes doesn't work with cuda graphs, deactivating them"); | ||
| vec![] | ||
| } | ||
| (None, Some(Quantization::Exl2)) => { | ||
| tracing::info!("Exl2 doesn't work with cuda graphs, deactivating them"); |
There was a problem hiding this comment.
New inserted logic, we remove cuda graphs when the model is exl2.
| @@ -46,6 +46,7 @@ marlin-kernels = [ | |||
| { url = "https://github.com/danieldk/marlin-kernels/releases/download/v0.2.0/marlin_kernels-0.2.0+cu123torch2.4-cp311-cp311-linux_x86_64.whl", python = "~3.11", optional = true }, | |||
| { url = "https://github.com/danieldk/marlin-kernels/releases/download/v0.2.0/marlin_kernels-0.2.0+cu123torch2.4-cp312-cp312-linux_x86_64.whl", python = "~3.12", optional = true }, | |||
| ] | |||
| rich = "^13.7.1" | |||
There was a problem hiding this comment.
This is a new dep of exl2 (which we don't care that much about, but makes evolving versions easier since we don't have to maintain a fork).
| // #[deprecated( | ||
| // since = "1.1.0", | ||
| // note = "Use `eetq` instead, which provides better latencies overall and is drop-in in most cases" | ||
| // )] |
There was a problem hiding this comment.
Would it be possible to add the deprecation in the Display impl instead? I think only Clap uses Display.
What does this PR do?
Fixes # (issue)
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.