[MPS] Fix variable number of inputs in mps_example.py #918
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for resplendent-gnome-14e531 canceled.
|
facebook-github-bot
added
the
CLA Signed
shoumikhin
approved these changes
Oct 14, 2023
kimishpatel
approved these changes
Oct 14, 2023
|
@kimishpatel has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@kimishpatel merged this pull request in bfb0fe0. |
This was referenced Oct 15, 2023
dbort
pushed a commit
that referenced
this pull request
Oct 15, 2023
Summary: - fix models with variable number of inputs (>1). Needed for the models `add`, `linear`, `add_mul` where the inputs array is usually made out of 2/3 tensors. - llama2 is also working correctly through MPS delegate. Set the `atol` to 1e-4 when doing the bundled program comparison check with the eager output from CPU. Pull Request resolved: #918 Reviewed By: kirklandsign Differential Revision: D50301445 Pulled By: kimishpatel fbshipit-source-id: 38576363de60fc45dabc92833723cf8f293cc7c5
Gasoonjia
pushed a commit
that referenced
this pull request
Jul 30, 2024
This PR refactors the chat function in generate.py by creating a `Generator` class, removing unnecessary global variables and simplifying the code structure. Tokens and metrics are now yielded by the Generator rather than being printed directly to stdout, making it easier to re-use this code for non-CLI tools. **Tests:** Generate ``` python3 torchchat.py generate stories15M --prompt "Once upon a time," Using device=mps Loading model... Time to load model: 0.46 seconds ----------------------------------------------------------- Once upon a time, there was a little girl named Lily. One day, she went to the park with her mom. They saw a big tree with lots of pears on it. Lily wanted to eat a pear, but they were too high up. She tried to jump, but she couldn't reach them. Then, a boy came and took a pear from the tree. Lily was surprised! She thought the boy would be mean, but he was harmless and didn't mean to hurt her. She asked him if she could have a pear too. The boy said yes and gave her a pear. Lily was very happy and thanked the boy for sharing. They sat under the tree and ate the pear together. It was a good day at the park. Once upon a time, there was a little girl named Lily. She loved to play in the garden with her mommy. One day, they were planting some Time for inference 1: 3.55 sec total, time to first token 0.00 sec with parallel prefill, 199 tokens, 56.01 tokens/sec, 17.85 ms/token Bandwidth achieved: 2.73 GB/s *** This first iteration will include cold start effects for dynamic import, hardware caches. *** ======================================== Average tokens/sec: 56.01 ======================================== Average tokens/sec: 119.35 Memory used: 0.00 GB ``` Eval ``` python3 torchchat.py eval stories15M --tasks wikitext --limit 10 NumExpr defaulting to 10 threads. PyTorch version 2.5.0.dev20240629 available. Using device=mps Loading model... Time to load model: 0.38 seconds ----------------------------------------------------------- Using device 'mps' [Task: wikitext] metric word_perplexity is defined, but aggregation is not. using default aggregation=weighted_perplexity [Task: wikitext] metric word_perplexity is defined, but higher_is_better is not. using default higher_is_better=False [Task: wikitext] metric byte_perplexity is defined, but aggregation is not. using default aggregation=weighted_perplexity [Task: wikitext] metric byte_perplexity is defined, but higher_is_better is not. using default higher_is_better=False [Task: wikitext] metric bits_per_byte is defined, but aggregation is not. using default aggregation=bits_per_byte [Task: wikitext] metric bits_per_byte is defined, but higher_is_better is not. using default higher_is_better=False Repo card metadata block was not found. Setting CardData to empty. Repo card metadata block was not found. Setting CardData to empty. Building contexts for wikitext on rank 0... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 541.59it/s] Running loglikelihood_rolling requests 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00, 2.25it/s] Time to run eval: 22.57s. Time in model.forward: 1.01s, over 33 model evaluations forward run time stats - Median: 0.02s Min: 0.01s Max: 0.27s For model /Users/puri/.torchchat/model-cache/stories15M/stories15M.pt wikitext: word_perplexity,none: 47350.8811 byte_perplexity,none: 7.7811 bits_per_byte,none: 2.9600 alias: wikitext ``` Co-authored-by: vmpuri <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
add,linear,add_mulwhere the inputs array is usually made out of 2/3 tensors.atolto 1e-4 when doing the bundled program comparison check with the eager output from CPU.