Skip to content

stream results in generate.py #571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 30, 2024
Merged

stream results in generate.py #571

merged 1 commit into from
Apr 30, 2024

Conversation

metascroy
Copy link
Contributor

Stream output in generate.py

python torchchat.py generate stories15M --prompt "Once upon a time"
Using device=cpu Apple M1 Pro
Loading model...
Time to load model: 0.02 seconds
Once upon a time, there was a little girl named Lily. She loved to play with her toys and spend time with her mom. One day, Lily's mom said to her, "Lily, let's go for a walk and enjoy the beautiful view outside."
Lily was so excited and followed her mom outside. They walked and walked until they found a big park with lots of trees and flowers. Lily was very happy and said, "Mommy, this is so much fun!"
As they were walking, Lily saw some other kids playing a game. He asked her mom if she wanted to play too, and she said yes. They played for a while and had so much fun. When it was time to go home, Lily said, "Mommy, can we come back to this beautiful park and play again?"
Her mom smiled and said, "Of course we can, Lily. We can come back and play anyOnce upon a time, there was a little girl named Lily. She loved to play with her toys and spend time with her mom. One day, Lily's mom said to her, "Lily, let's go for a walk and enjoy the beautiful view outside."
Lily was so excited and followed her mom outside. They walked and walked until they found a big park with lots of trees and flowers. Lily was very happy and said, "Mommy, this is so much fun!"
As they were walking, Lily saw some other kids playing a game. He asked her mom if she wanted to play too, and she said yes. They played for a while and had so much fun. When it was time to go home, Lily said, "Mommy, can we come back to this beautiful park and play again?"
Her mom smiled and said, "Of course we can, Lily. We can come back and play any
Max Sequence Length Reached. Ending Conversation.
==========
Average tokens/sec: 210.69
Memory used: 0.00 GB

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Apr 30, 2024

def callback(x):
return x
def callback(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we have to copy-paste the callback above instead of sharing it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid point.

Copy link
Contributor

@mikekgfb mikekgfb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the other Scott’s comment.

OK to do as a seperate PR, but please create an issue to remember, if you go this route

@mikekgfb mikekgfb merged commit 2e3b8d7 into main Apr 30, 2024
mikekgfb added a commit that referenced this pull request Apr 30, 2024
* code beautification

* code beautification, move functions together

* make --device fast the default (#515)

* make --device fast the default

* Update iOS.md (#517)

* Update iOS.md

* Update iOS.md

* Pip to pip3 (#504)

* remove macos-12 test

* pip to pip3

* break aoti CI jobs separately (#500)

* init

* fixes

* more fixes

* fixes

* fix

* fix

* bug fix

* add objcopy update

* suppress int8

* undefined variable

---------

Co-authored-by: Michael Gschwind <[email protected]>

* Support llama3 in chat in run.cpp  (#486)

* refactor chat runner in preparation for llama3

* add sketch for llama3 prompt template and move to returning tokens

* fix tiktoken

* fixes to chat

* add default llama_ver

* Add tests for quantize json, add cuda device specification and precision to cuda.json (#519)

* remove code for no KV Cache path (#527)

* Update ADVANCED-USERS.md (#529)

Update Advanced Users description to reflect changes in the repo since the description was initially created.

* runner-aoti on cuda (#531)

* runner-aoti on cuda

* transfer results back to CPU

* transfer results back to CPU

* runner-aoti on cuda

* Update runner_build.md (#530)

Update description of runner and build process in runner_build.md

* clean up runner code a little (#532)

* clean up runner code a little

* update

* update

* pull out generate loop in chat

* updates

* edit docs

* typo

* move int8 linear class and function into qops.py (#534)

* add dtype tests for runner-aoti + runner-et (#539)

* add dtype tests for runner-aoti + runner-et

* typo

* Quantized embedding (#536)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* Move Linear int4 to qops (#537)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* move int4 linear to qops

* Revert "add dtype tests for runner-aoti + runner-et (#539)" (#548)

This reverts commit a7a2457.

* fix generate for llama3 (#538)

* fix generate for llama3

* switch more things to C

* remove C++ header

* add delegation visualization instructions (#551)

* Add dtype runner aoti (#552)

* add dtype tests for runner-aoti + runner-et

* typo

* add dtype test runner-aoti

* test sdpa with fp16 (#553)

* test sdpa with fp16

* kv cache fp32

* typo

* update (#560)

* Only support newest versions of lm-eval (#556)

Summary:
remove support for lm-eval 0.3 to reduce the options we have

Test Plan:
CI

Reviewers:

Subscribers:

Tasks:

Tags:

* split cpu eval CI by dtype (#554)

* split cpu eval CI by dtype

* fix

* differentiate names with checks

* keep one name the same as old

* fix

* Removing duplicate HF issue message from README (#559)

Co-authored-by: Michael Gschwind <[email protected]>

* doc updates (#567)

* Add VM-safe MPS check

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>

* add unpacking support (#525)

* add unpacking support

* fix typos and linter

* perform parallel prefill when possible (#568)

* perform parallel prefill when possible

* typo

* disable hack

* remove print

* remove debug messages which prevent export

* fixes

* stream results in generate.py (#571)

* remove logging interfering with export

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>
@malfet malfet deleted the stream-in-generate branch April 30, 2024 16:49
malfet pushed a commit that referenced this pull request Jul 17, 2024
malfet added a commit that referenced this pull request Jul 17, 2024
* code beautification

* code beautification, move functions together

* make --device fast the default (#515)

* make --device fast the default

* Update iOS.md (#517)

* Update iOS.md

* Update iOS.md

* Pip to pip3 (#504)

* remove macos-12 test

* pip to pip3

* break aoti CI jobs separately (#500)

* init

* fixes

* more fixes

* fixes

* fix

* fix

* bug fix

* add objcopy update

* suppress int8

* undefined variable

---------

Co-authored-by: Michael Gschwind <[email protected]>

* Support llama3 in chat in run.cpp  (#486)

* refactor chat runner in preparation for llama3

* add sketch for llama3 prompt template and move to returning tokens

* fix tiktoken

* fixes to chat

* add default llama_ver

* Add tests for quantize json, add cuda device specification and precision to cuda.json (#519)

* remove code for no KV Cache path (#527)

* Update ADVANCED-USERS.md (#529)

Update Advanced Users description to reflect changes in the repo since the description was initially created.

* runner-aoti on cuda (#531)

* runner-aoti on cuda

* transfer results back to CPU

* transfer results back to CPU

* runner-aoti on cuda

* Update runner_build.md (#530)

Update description of runner and build process in runner_build.md

* clean up runner code a little (#532)

* clean up runner code a little

* update

* update

* pull out generate loop in chat

* updates

* edit docs

* typo

* move int8 linear class and function into qops.py (#534)

* add dtype tests for runner-aoti + runner-et (#539)

* add dtype tests for runner-aoti + runner-et

* typo

* Quantized embedding (#536)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* Move Linear int4 to qops (#537)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* move int4 linear to qops

* Revert "add dtype tests for runner-aoti + runner-et (#539)" (#548)

This reverts commit a7a2457.

* fix generate for llama3 (#538)

* fix generate for llama3

* switch more things to C

* remove C++ header

* add delegation visualization instructions (#551)

* Add dtype runner aoti (#552)

* add dtype tests for runner-aoti + runner-et

* typo

* add dtype test runner-aoti

* test sdpa with fp16 (#553)

* test sdpa with fp16

* kv cache fp32

* typo

* update (#560)

* Only support newest versions of lm-eval (#556)

Summary:
remove support for lm-eval 0.3 to reduce the options we have

Test Plan:
CI

Reviewers:

Subscribers:

Tasks:

Tags:

* split cpu eval CI by dtype (#554)

* split cpu eval CI by dtype

* fix

* differentiate names with checks

* keep one name the same as old

* fix

* Removing duplicate HF issue message from README (#559)

Co-authored-by: Michael Gschwind <[email protected]>

* doc updates (#567)

* Add VM-safe MPS check

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>

* add unpacking support (#525)

* add unpacking support

* fix typos and linter

* perform parallel prefill when possible (#568)

* perform parallel prefill when possible

* typo

* disable hack

* remove print

* remove debug messages which prevent export

* fixes

* stream results in generate.py (#571)

* remove logging interfering with export

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>
malfet pushed a commit that referenced this pull request Jul 17, 2024
malfet added a commit that referenced this pull request Jul 17, 2024
* code beautification

* code beautification, move functions together

* make --device fast the default (#515)

* make --device fast the default

* Update iOS.md (#517)

* Update iOS.md

* Update iOS.md

* Pip to pip3 (#504)

* remove macos-12 test

* pip to pip3

* break aoti CI jobs separately (#500)

* init

* fixes

* more fixes

* fixes

* fix

* fix

* bug fix

* add objcopy update

* suppress int8

* undefined variable

---------

Co-authored-by: Michael Gschwind <[email protected]>

* Support llama3 in chat in run.cpp  (#486)

* refactor chat runner in preparation for llama3

* add sketch for llama3 prompt template and move to returning tokens

* fix tiktoken

* fixes to chat

* add default llama_ver

* Add tests for quantize json, add cuda device specification and precision to cuda.json (#519)

* remove code for no KV Cache path (#527)

* Update ADVANCED-USERS.md (#529)

Update Advanced Users description to reflect changes in the repo since the description was initially created.

* runner-aoti on cuda (#531)

* runner-aoti on cuda

* transfer results back to CPU

* transfer results back to CPU

* runner-aoti on cuda

* Update runner_build.md (#530)

Update description of runner and build process in runner_build.md

* clean up runner code a little (#532)

* clean up runner code a little

* update

* update

* pull out generate loop in chat

* updates

* edit docs

* typo

* move int8 linear class and function into qops.py (#534)

* add dtype tests for runner-aoti + runner-et (#539)

* add dtype tests for runner-aoti + runner-et

* typo

* Quantized embedding (#536)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* Move Linear int4 to qops (#537)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* move int4 linear to qops

* Revert "add dtype tests for runner-aoti + runner-et (#539)" (#548)

This reverts commit a7a2457.

* fix generate for llama3 (#538)

* fix generate for llama3

* switch more things to C

* remove C++ header

* add delegation visualization instructions (#551)

* Add dtype runner aoti (#552)

* add dtype tests for runner-aoti + runner-et

* typo

* add dtype test runner-aoti

* test sdpa with fp16 (#553)

* test sdpa with fp16

* kv cache fp32

* typo

* update (#560)

* Only support newest versions of lm-eval (#556)

Summary:
remove support for lm-eval 0.3 to reduce the options we have

Test Plan:
CI

Reviewers:

Subscribers:

Tasks:

Tags:

* split cpu eval CI by dtype (#554)

* split cpu eval CI by dtype

* fix

* differentiate names with checks

* keep one name the same as old

* fix

* Removing duplicate HF issue message from README (#559)

Co-authored-by: Michael Gschwind <[email protected]>

* doc updates (#567)

* Add VM-safe MPS check

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>

* add unpacking support (#525)

* add unpacking support

* fix typos and linter

* perform parallel prefill when possible (#568)

* perform parallel prefill when possible

* typo

* disable hack

* remove print

* remove debug messages which prevent export

* fixes

* stream results in generate.py (#571)

* remove logging interfering with export

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>
malfet pushed a commit that referenced this pull request Jul 17, 2024
malfet added a commit that referenced this pull request Jul 17, 2024
* code beautification

* code beautification, move functions together

* make --device fast the default (#515)

* make --device fast the default

* Update iOS.md (#517)

* Update iOS.md

* Update iOS.md

* Pip to pip3 (#504)

* remove macos-12 test

* pip to pip3

* break aoti CI jobs separately (#500)

* init

* fixes

* more fixes

* fixes

* fix

* fix

* bug fix

* add objcopy update

* suppress int8

* undefined variable

---------

Co-authored-by: Michael Gschwind <[email protected]>

* Support llama3 in chat in run.cpp  (#486)

* refactor chat runner in preparation for llama3

* add sketch for llama3 prompt template and move to returning tokens

* fix tiktoken

* fixes to chat

* add default llama_ver

* Add tests for quantize json, add cuda device specification and precision to cuda.json (#519)

* remove code for no KV Cache path (#527)

* Update ADVANCED-USERS.md (#529)

Update Advanced Users description to reflect changes in the repo since the description was initially created.

* runner-aoti on cuda (#531)

* runner-aoti on cuda

* transfer results back to CPU

* transfer results back to CPU

* runner-aoti on cuda

* Update runner_build.md (#530)

Update description of runner and build process in runner_build.md

* clean up runner code a little (#532)

* clean up runner code a little

* update

* update

* pull out generate loop in chat

* updates

* edit docs

* typo

* move int8 linear class and function into qops.py (#534)

* add dtype tests for runner-aoti + runner-et (#539)

* add dtype tests for runner-aoti + runner-et

* typo

* Quantized embedding (#536)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* Move Linear int4 to qops (#537)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* move int4 linear to qops

* Revert "add dtype tests for runner-aoti + runner-et (#539)" (#548)

This reverts commit a7a2457.

* fix generate for llama3 (#538)

* fix generate for llama3

* switch more things to C

* remove C++ header

* add delegation visualization instructions (#551)

* Add dtype runner aoti (#552)

* add dtype tests for runner-aoti + runner-et

* typo

* add dtype test runner-aoti

* test sdpa with fp16 (#553)

* test sdpa with fp16

* kv cache fp32

* typo

* update (#560)

* Only support newest versions of lm-eval (#556)

Summary:
remove support for lm-eval 0.3 to reduce the options we have

Test Plan:
CI

Reviewers:

Subscribers:

Tasks:

Tags:

* split cpu eval CI by dtype (#554)

* split cpu eval CI by dtype

* fix

* differentiate names with checks

* keep one name the same as old

* fix

* Removing duplicate HF issue message from README (#559)

Co-authored-by: Michael Gschwind <[email protected]>

* doc updates (#567)

* Add VM-safe MPS check

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>

* add unpacking support (#525)

* add unpacking support

* fix typos and linter

* perform parallel prefill when possible (#568)

* perform parallel prefill when possible

* typo

* disable hack

* remove print

* remove debug messages which prevent export

* fixes

* stream results in generate.py (#571)

* remove logging interfering with export

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>
malfet pushed a commit that referenced this pull request Jul 17, 2024
malfet added a commit that referenced this pull request Jul 17, 2024
* code beautification

* code beautification, move functions together

* make --device fast the default (#515)

* make --device fast the default

* Update iOS.md (#517)

* Update iOS.md

* Update iOS.md

* Pip to pip3 (#504)

* remove macos-12 test

* pip to pip3

* break aoti CI jobs separately (#500)

* init

* fixes

* more fixes

* fixes

* fix

* fix

* bug fix

* add objcopy update

* suppress int8

* undefined variable

---------

Co-authored-by: Michael Gschwind <[email protected]>

* Support llama3 in chat in run.cpp  (#486)

* refactor chat runner in preparation for llama3

* add sketch for llama3 prompt template and move to returning tokens

* fix tiktoken

* fixes to chat

* add default llama_ver

* Add tests for quantize json, add cuda device specification and precision to cuda.json (#519)

* remove code for no KV Cache path (#527)

* Update ADVANCED-USERS.md (#529)

Update Advanced Users description to reflect changes in the repo since the description was initially created.

* runner-aoti on cuda (#531)

* runner-aoti on cuda

* transfer results back to CPU

* transfer results back to CPU

* runner-aoti on cuda

* Update runner_build.md (#530)

Update description of runner and build process in runner_build.md

* clean up runner code a little (#532)

* clean up runner code a little

* update

* update

* pull out generate loop in chat

* updates

* edit docs

* typo

* move int8 linear class and function into qops.py (#534)

* add dtype tests for runner-aoti + runner-et (#539)

* add dtype tests for runner-aoti + runner-et

* typo

* Quantized embedding (#536)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* Move Linear int4 to qops (#537)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* move int4 linear to qops

* Revert "add dtype tests for runner-aoti + runner-et (#539)" (#548)

This reverts commit a7a2457.

* fix generate for llama3 (#538)

* fix generate for llama3

* switch more things to C

* remove C++ header

* add delegation visualization instructions (#551)

* Add dtype runner aoti (#552)

* add dtype tests for runner-aoti + runner-et

* typo

* add dtype test runner-aoti

* test sdpa with fp16 (#553)

* test sdpa with fp16

* kv cache fp32

* typo

* update (#560)

* Only support newest versions of lm-eval (#556)

Summary:
remove support for lm-eval 0.3 to reduce the options we have

Test Plan:
CI

Reviewers:

Subscribers:

Tasks:

Tags:

* split cpu eval CI by dtype (#554)

* split cpu eval CI by dtype

* fix

* differentiate names with checks

* keep one name the same as old

* fix

* Removing duplicate HF issue message from README (#559)

Co-authored-by: Michael Gschwind <[email protected]>

* doc updates (#567)

* Add VM-safe MPS check

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>

* add unpacking support (#525)

* add unpacking support

* fix typos and linter

* perform parallel prefill when possible (#568)

* perform parallel prefill when possible

* typo

* disable hack

* remove print

* remove debug messages which prevent export

* fixes

* stream results in generate.py (#571)

* remove logging interfering with export

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>
malfet pushed a commit that referenced this pull request Jul 17, 2024
malfet added a commit that referenced this pull request Jul 17, 2024
* code beautification

* code beautification, move functions together

* make --device fast the default (#515)

* make --device fast the default

* Update iOS.md (#517)

* Update iOS.md

* Update iOS.md

* Pip to pip3 (#504)

* remove macos-12 test

* pip to pip3

* break aoti CI jobs separately (#500)

* init

* fixes

* more fixes

* fixes

* fix

* fix

* bug fix

* add objcopy update

* suppress int8

* undefined variable

---------

Co-authored-by: Michael Gschwind <[email protected]>

* Support llama3 in chat in run.cpp  (#486)

* refactor chat runner in preparation for llama3

* add sketch for llama3 prompt template and move to returning tokens

* fix tiktoken

* fixes to chat

* add default llama_ver

* Add tests for quantize json, add cuda device specification and precision to cuda.json (#519)

* remove code for no KV Cache path (#527)

* Update ADVANCED-USERS.md (#529)

Update Advanced Users description to reflect changes in the repo since the description was initially created.

* runner-aoti on cuda (#531)

* runner-aoti on cuda

* transfer results back to CPU

* transfer results back to CPU

* runner-aoti on cuda

* Update runner_build.md (#530)

Update description of runner and build process in runner_build.md

* clean up runner code a little (#532)

* clean up runner code a little

* update

* update

* pull out generate loop in chat

* updates

* edit docs

* typo

* move int8 linear class and function into qops.py (#534)

* add dtype tests for runner-aoti + runner-et (#539)

* add dtype tests for runner-aoti + runner-et

* typo

* Quantized embedding (#536)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* Move Linear int4 to qops (#537)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* move int4 linear to qops

* Revert "add dtype tests for runner-aoti + runner-et (#539)" (#548)

This reverts commit a7a2457.

* fix generate for llama3 (#538)

* fix generate for llama3

* switch more things to C

* remove C++ header

* add delegation visualization instructions (#551)

* Add dtype runner aoti (#552)

* add dtype tests for runner-aoti + runner-et

* typo

* add dtype test runner-aoti

* test sdpa with fp16 (#553)

* test sdpa with fp16

* kv cache fp32

* typo

* update (#560)

* Only support newest versions of lm-eval (#556)

Summary:
remove support for lm-eval 0.3 to reduce the options we have

Test Plan:
CI

Reviewers:

Subscribers:

Tasks:

Tags:

* split cpu eval CI by dtype (#554)

* split cpu eval CI by dtype

* fix

* differentiate names with checks

* keep one name the same as old

* fix

* Removing duplicate HF issue message from README (#559)

Co-authored-by: Michael Gschwind <[email protected]>

* doc updates (#567)

* Add VM-safe MPS check

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>

* add unpacking support (#525)

* add unpacking support

* fix typos and linter

* perform parallel prefill when possible (#568)

* perform parallel prefill when possible

* typo

* disable hack

* remove print

* remove debug messages which prevent export

* fixes

* stream results in generate.py (#571)

* remove logging interfering with export

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>
malfet pushed a commit that referenced this pull request Jul 17, 2024
malfet added a commit that referenced this pull request Jul 17, 2024
* code beautification

* code beautification, move functions together

* make --device fast the default (#515)

* make --device fast the default

* Update iOS.md (#517)

* Update iOS.md

* Update iOS.md

* Pip to pip3 (#504)

* remove macos-12 test

* pip to pip3

* break aoti CI jobs separately (#500)

* init

* fixes

* more fixes

* fixes

* fix

* fix

* bug fix

* add objcopy update

* suppress int8

* undefined variable

---------

Co-authored-by: Michael Gschwind <[email protected]>

* Support llama3 in chat in run.cpp  (#486)

* refactor chat runner in preparation for llama3

* add sketch for llama3 prompt template and move to returning tokens

* fix tiktoken

* fixes to chat

* add default llama_ver

* Add tests for quantize json, add cuda device specification and precision to cuda.json (#519)

* remove code for no KV Cache path (#527)

* Update ADVANCED-USERS.md (#529)

Update Advanced Users description to reflect changes in the repo since the description was initially created.

* runner-aoti on cuda (#531)

* runner-aoti on cuda

* transfer results back to CPU

* transfer results back to CPU

* runner-aoti on cuda

* Update runner_build.md (#530)

Update description of runner and build process in runner_build.md

* clean up runner code a little (#532)

* clean up runner code a little

* update

* update

* pull out generate loop in chat

* updates

* edit docs

* typo

* move int8 linear class and function into qops.py (#534)

* add dtype tests for runner-aoti + runner-et (#539)

* add dtype tests for runner-aoti + runner-et

* typo

* Quantized embedding (#536)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* Move Linear int4 to qops (#537)

* move int8 linear class and function into qops.py

* move Quantized Embedding to qops.py

* move int4 linear to qops

* Revert "add dtype tests for runner-aoti + runner-et (#539)" (#548)

This reverts commit a7a2457.

* fix generate for llama3 (#538)

* fix generate for llama3

* switch more things to C

* remove C++ header

* add delegation visualization instructions (#551)

* Add dtype runner aoti (#552)

* add dtype tests for runner-aoti + runner-et

* typo

* add dtype test runner-aoti

* test sdpa with fp16 (#553)

* test sdpa with fp16

* kv cache fp32

* typo

* update (#560)

* Only support newest versions of lm-eval (#556)

Summary:
remove support for lm-eval 0.3 to reduce the options we have

Test Plan:
CI

Reviewers:

Subscribers:

Tasks:

Tags:

* split cpu eval CI by dtype (#554)

* split cpu eval CI by dtype

* fix

* differentiate names with checks

* keep one name the same as old

* fix

* Removing duplicate HF issue message from README (#559)

Co-authored-by: Michael Gschwind <[email protected]>

* doc updates (#567)

* Add VM-safe MPS check

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>

* add unpacking support (#525)

* add unpacking support

* fix typos and linter

* perform parallel prefill when possible (#568)

* perform parallel prefill when possible

* typo

* disable hack

* remove print

* remove debug messages which prevent export

* fixes

* stream results in generate.py (#571)

* remove logging interfering with export

---------

Co-authored-by: Anthony Shoumikhin <[email protected]>
Co-authored-by: metascroy <[email protected]>
Co-authored-by: Nikita Shulga <[email protected]>
Co-authored-by: lucylq <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Jack-Khuu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants