Add time to first token for llama runner #2141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

vmpuri wants to merge 1 commit into pytorch:main from vmpuri:export-D54223564

vmpuri commented Feb 26, 2024

Summary: Add time to first generated

Differential Revision: D54223564

pytorch-bot bot commented Feb 26, 2024 •

edited

Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/2141

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 6c28e7f with merge base 3e414fb ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot added the CLA Signed label

Contributor

facebook-github-bot commented Feb 26, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

facebook-github-bot added the fb-exported label

vmpuri force-pushed the export-D54223564 branch from 77984d6 to c7fa90f Compare

March 6, 2024 23:44

Contributor

facebook-github-bot commented Mar 6, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

095524e

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.

To implement:
* Sample time - amount of time spent sampling per token (present in llama.cpp)

Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from c7fa90f to 095524e Compare

March 6, 2024 23:47

Contributor

facebook-github-bot commented Mar 6, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from 095524e to 3d7d04e Compare

March 7, 2024 22:29

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

3d7d04e

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.

To implement:
* Sample time - amount of time spent sampling per token (present in llama.cpp)

Differential Revision: D54223564

Contributor

facebook-github-bot commented Mar 7, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

06083b3

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.

To implement:
* Sample time - amount of time spent sampling per token (present in llama.cpp)

Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from 3d7d04e to 06083b3 Compare

March 7, 2024 23:21

Contributor

facebook-github-bot commented Mar 7, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from 06083b3 to e478f8b Compare

March 11, 2024 23:44

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

e478f8b

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.

To implement:
* Sample time - amount of time spent sampling per token (present in llama.cpp)

Reviewed By: digantdesai

Differential Revision: D54223564

Contributor

facebook-github-bot commented Mar 11, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

abd98c1

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.

To implement:
* Sample time - amount of time spent sampling per token (present in llama.cpp)

Reviewed By: digantdesai

Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from e478f8b to abd98c1 Compare

March 11, 2024 23:53

Contributor

facebook-github-bot commented Mar 11, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from abd98c1 to 52545fe Compare

March 12, 2024 00:53

Contributor

facebook-github-bot commented Mar 12, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from 52545fe to 1936568 Compare

March 12, 2024 21:55

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

Reviewed By: digantdesai

Differential Revision: D54223564

Contributor

facebook-github-bot commented Mar 12, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

2c92b19

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks

Reviewed By: digantdesai

Differential Revision: D54223564

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

c0526bb

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks

Reviewed By: digantdesai

Differential Revision: D54223564

Jack-Khuu approved these changes

View reviewed changes

Contributor

facebook-github-bot commented Mar 13, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

Contributor

facebook-github-bot commented Mar 13, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

5f0067c

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks

Reviewed By: digantdesai, Jack-Khuu

Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch 2 times, most recently from 5f0067c to d0e6269 Compare

March 13, 2024 17:15

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

d0e6269

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks

Reviewed By: digantdesai, Jack-Khuu

Differential Revision: D54223564

Contributor

facebook-github-bot commented Mar 13, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks

Reviewed By: digantdesai, Jack-Khuu

Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from d0e6269 to 1734687 Compare

March 13, 2024 21:00

Contributor

facebook-github-bot commented Mar 13, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

8c6cb45

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks
bypass-github-pytorch-ci-checks

Reviewed By: digantdesai, Jack-Khuu

Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from 1734687 to 8c6cb45 Compare

March 13, 2024 22:41

Contributor

facebook-github-bot commented Mar 13, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

ff13f01

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks
bypass-github-pytorch-ci-checks

Reviewed By: digantdesai, Jack-Khuu

Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from 8c6cb45 to ff13f01 Compare

March 13, 2024 22:42

Contributor

facebook-github-bot commented Mar 13, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

1 similar comment

Contributor

facebook-github-bot commented Mar 14, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from ff13f01 to 0aa3659 Compare

March 14, 2024 18:05

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

0aa3659

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks
bypass-github-pytorch-ci-checks

Reviewed By: digantdesai, Jack-Khuu

Differential Revision: D54223564

Contributor

facebook-github-bot commented Mar 14, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564


          Add time to first token for llama runner (pytorch#2141)

6c28e7f

Summary:

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks
bypass-github-pytorch-ci-checks

Reviewed By: digantdesai, Jack-Khuu

Differential Revision: D54223564

vmpuri force-pushed the export-D54223564 branch from 0aa3659 to 6c28e7f Compare

March 14, 2024 18:20

Contributor

facebook-github-bot commented Mar 14, 2024

This pull request was exported from Phabricator. Differential Revision: D54223564

vmpuri pushed a commit to vmpuri/executorch-1 that referenced this pull request


          Add time to first token for llama runner (pytorch#2141)

Summary:
bypass-github-pytorch-ci-checks

Add time to first generated token & other features



- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks
bypass-github-pytorch-ci-checks

Reviewed By: digantdesai, Jack-Khuu

Differential Revision: D54223564

facebook-github-bot closed this in

caee336

facebook-github-bot added the Merged label

Contributor

facebook-github-bot commented Mar 14, 2024

This pull request has been merged in caee336.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed fb-exported Merged