Skip to content

Update docs for max-autotune usage #1405

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 9, 2024
+5 −0
Merged

Update docs for max-autotune usage #1405

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b964320
Update docs for max-autotune usage
yanbing-j Dec 9, 2024
a5cd2ba
Merge branch 'main' into yanbing/maxautotune_doc
Jack-Khuu Dec 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/ADVANCED-USERS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,8 @@ To improve performance, you can compile the model with `--compile`
trading off the time to first token processed with time per token. To
improve performance further, you may also compile the prefill with
`--compile-prefill`. This will increase further compilation times though.
For CPU, you can use `--max-autotune` to further improve the performance
with `--compile` and `compile-prefill`. See [`max-autotune on CPU tutorial`](https://pytorch.org/tutorials/prototype/max_autotune_on_CPU_tutorial.html).

Parallel prefill is not yet supported by exported models, and may be
supported in a future release.
Expand Down
3 changes: 3 additions & 0 deletions docs/model_customization.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ prefill with `--compile_prefill`.

To learn more about compilation, check out: https://pytorch.org/get-started/pytorch-2.0/

For CPU, you can use `--max-autotune` to further improve the performance with `--compile` and `compile-prefill`.

See [`max-autotune on CPU tutorial`](https://pytorch.org/tutorials/prototype/max_autotune_on_CPU_tutorial.html).

## Model Precision

Expand Down
Loading