Skip to content

Bug fix in bpe tokenizer #4149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Bug fix in bpe tokenizer #4149

wants to merge 1 commit into from

Conversation

guangy10
Copy link
Contributor

@guangy10 guangy10 commented Jul 3, 2024

  • Record bos/eos in the binary format
  • Updated tests

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jul 3, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4149

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 1022e69 with merge base e4eeadc (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 3, 2024
@guangy10 guangy10 mentioned this pull request Jul 3, 2024
Closed
@guangy10 guangy10 force-pushed the fix_bpe_tokenizer branch 2 times, most recently from 7d4a02c to 05b3788 Compare July 3, 2024 21:19
@facebook-github-bot
Copy link
Contributor

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@guangy10 guangy10 force-pushed the fix_bpe_tokenizer branch from 05b3788 to 7260263 Compare July 3, 2024 22:13
@guangy10
Copy link
Contributor Author

guangy10 commented Jul 3, 2024

It looks like .gitignore prevents checking in the new tokenizer.bin. Fixed it now.

@facebook-github-bot
Copy link
Contributor

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@guangy10 guangy10 force-pushed the fix_bpe_tokenizer branch from 7260263 to 42775b5 Compare July 3, 2024 22:39
@facebook-github-bot
Copy link
Contributor

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@guangy10 guangy10 force-pushed the fix_bpe_tokenizer branch from 42775b5 to f6c9350 Compare July 4, 2024 00:18
@facebook-github-bot
Copy link
Contributor

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@guangy10 guangy10 force-pushed the fix_bpe_tokenizer branch from f6c9350 to 1022e69 Compare July 8, 2024 18:36
@facebook-github-bot
Copy link
Contributor

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@guangy10 merged this pull request in d948c1a.

@guangy10 guangy10 deleted the fix_bpe_tokenizer branch August 21, 2024 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@larryliu0820 larryliu0820 larryliu0820 approved these changes

@shoumikhin shoumikhin Awaiting requested review from shoumikhin

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@guangy10 @facebook-github-bot @larryliu0820