loose sdk example verification requirement #906
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for resplendent-gnome-14e531 ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D50280300 |
|
This pull request was exported from Phabricator. Differential Revision: D50280300 |
Gasoonjia
force-pushed
the
export-D50280300
branch
from
e228e68 to
17d32b7
Compare
|
This pull request was exported from Phabricator. Differential Revision: D50280300 |
Gasoonjia
force-pushed
the
export-D50280300
branch
from
17d32b7 to
5b68669
Compare
Summary: Pull Request resolved: pytorch/executorch#880 Update the source of flatten_tree functino from executorch.extension to torch core to support OrderedDict type. Reviewed By: tarun292 Differential Revision: D50245226 fbshipit-source-id: f85b1d1a8711f593155799d0652f9e48b8bbc5c3
Summary: Pull Request resolved: pytorch/executorch#879 1. Remove deprecated profiler stuffs. 2. Add print_output flag for sdk_example_runners. Printing output each scalar per line is sometimes pretty verbose, especially some outputs have thousands or even millions of scalars. Make the print as optional in this diff. Reviewed By: tarun292 Differential Revision: D50246087 fbshipit-source-id: 49c76f0ebcf9ec75ef070e66156b28f70b19dcd9
Summary: Pull Request resolved: pytorch/executorch#906 previous requirement is too tight for every model. Reviewed By: tarun292 Differential Revision: D50280300 fbshipit-source-id: 634ade693ea0e893c26ce705c868748eb4c73ec9
|
This pull request was exported from Phabricator. Differential Revision: D50280300 |
Gasoonjia
force-pushed
the
export-D50280300
branch
from
5b68669 to
66788e9
Compare
|
This pull request has been merged in 5c00a83. |
Gasoonjia
pushed a commit
that referenced
this pull request
Jul 30, 2024
This PR refactors the chat function in generate.py by creating a `Generator` class, removing unnecessary global variables and simplifying the code structure. Tokens and metrics are now yielded by the Generator rather than being printed directly to stdout, making it easier to re-use this code for non-CLI tools. **Tests:** Generate ``` python3 torchchat.py generate stories15M --prompt "Once upon a time," Using device=mps Loading model... Time to load model: 0.46 seconds ----------------------------------------------------------- Once upon a time, there was a little girl named Lily. One day, she went to the park with her mom. They saw a big tree with lots of pears on it. Lily wanted to eat a pear, but they were too high up. She tried to jump, but she couldn't reach them. Then, a boy came and took a pear from the tree. Lily was surprised! She thought the boy would be mean, but he was harmless and didn't mean to hurt her. She asked him if she could have a pear too. The boy said yes and gave her a pear. Lily was very happy and thanked the boy for sharing. They sat under the tree and ate the pear together. It was a good day at the park. Once upon a time, there was a little girl named Lily. She loved to play in the garden with her mommy. One day, they were planting some Time for inference 1: 3.55 sec total, time to first token 0.00 sec with parallel prefill, 199 tokens, 56.01 tokens/sec, 17.85 ms/token Bandwidth achieved: 2.73 GB/s *** This first iteration will include cold start effects for dynamic import, hardware caches. *** ======================================== Average tokens/sec: 56.01 ======================================== Average tokens/sec: 119.35 Memory used: 0.00 GB ``` Eval ``` python3 torchchat.py eval stories15M --tasks wikitext --limit 10 NumExpr defaulting to 10 threads. PyTorch version 2.5.0.dev20240629 available. Using device=mps Loading model... Time to load model: 0.38 seconds ----------------------------------------------------------- Using device 'mps' [Task: wikitext] metric word_perplexity is defined, but aggregation is not. using default aggregation=weighted_perplexity [Task: wikitext] metric word_perplexity is defined, but higher_is_better is not. using default higher_is_better=False [Task: wikitext] metric byte_perplexity is defined, but aggregation is not. using default aggregation=weighted_perplexity [Task: wikitext] metric byte_perplexity is defined, but higher_is_better is not. using default higher_is_better=False [Task: wikitext] metric bits_per_byte is defined, but aggregation is not. using default aggregation=bits_per_byte [Task: wikitext] metric bits_per_byte is defined, but higher_is_better is not. using default higher_is_better=False Repo card metadata block was not found. Setting CardData to empty. Repo card metadata block was not found. Setting CardData to empty. Building contexts for wikitext on rank 0... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 541.59it/s] Running loglikelihood_rolling requests 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00, 2.25it/s] Time to run eval: 22.57s. Time in model.forward: 1.01s, over 33 model evaluations forward run time stats - Median: 0.02s Min: 0.01s Max: 0.27s For model /Users/puri/.torchchat/model-cache/stories15M/stories15M.pt wikitext: word_perplexity,none: 47350.8811 byte_perplexity,none: 7.7811 bits_per_byte,none: 2.9600 alias: wikitext ``` Co-authored-by: vmpuri <[email protected]>
Gasoonjia
pushed a commit
that referenced
this pull request
Jul 30, 2024
Remove the existing browser UI and replace it with a UI built with Streamlit. This reduces complexity & leverages the functionality introduced in PR #906 to display chunked responses. **Testing** ``` streamlit run torchchat.py -- browser stories110M --compile --max-new-tokens 256 You can now view your Streamlit app in your browser. Local URL: http://localhost:8501 Network URL: http://192.0.0.2:8501 ``` <img width="1002" alt="image" src="https://github.com/user-attachments/assets/df305943-2326-4d01-a48b-61dd2006fa28">
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: previous requirement is too tight for every model.
Differential Revision: D50280300