Skip to content

Add tiktoken OFF/ON options for android prebuilt aar #4203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 8 commits into from
Closed

Add tiktoken OFF/ON options for android prebuilt aar #4203

wants to merge 8 commits into from

Conversation

kirklandsign
Copy link
Contributor

@kirklandsign kirklandsign commented Jul 10, 2024 •
edited
Loading

BPE (tiktoken OFF) is for llama2, and tiktoken ON is for llama3.

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jul 10, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4203

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 1bbb2aa with merge base f9efb05 (image):

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

  • Android / test-llama-app / mobile-job (android) (gh) (#3344)
    Could not assume role with user credentials: User: arn:aws:sts::308535385114:assumed-role/gh-ci-github-action-runners-runner-role/i-06354c773870ca00d is not authorized to perform: sts:TagSession on resource: arn:aws:iam::308535385114:role/gha_workflow_mobile_job

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 10, 2024
@kirklandsign kirklandsign changed the title add tiktoken build for android Add tiktoken OFF/ON options for android prebuilt aar Jul 10, 2024
@facebook-github-bot
Copy link
Contributor

@kirklandsign has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@kirklandsign has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@kirklandsign has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@kirklandsign has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@kirklandsign has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@kirklandsign merged this pull request in e9aa542.

@huydhn huydhn mentioned this pull request Jul 17, 2024
Closed
@kirklandsign kirklandsign deleted the tiktoken branch September 5, 2024 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cccclai cccclai cccclai approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kirklandsign @facebook-github-bot @cccclai