Let models provider their own specific special tokens #4227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

helunwencser wants to merge 6 commits into gh/helunwencser/8/base from gh/helunwencser/8/head

Contributor

helunwencser commented Jul 11, 2024 •

edited

Loading

Stack from ghstack (oldest at bottom):

Differential Revision: D59651199


          Let models provider their own specific special tokens

cf39cde

Differential Revision: [D59651199](https://our.internmc.facebook.com/intern/diff/D59651199/)

[ghstack-poisoned]

pytorch-bot bot commented Jul 11, 2024 •

edited

Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4227

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit d21c440 with merge base f9efb05 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest / linux / linux-job (gh) (trunk failure)
curl: (22) The requested URL returned error:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot added the CLA Signed label

Contributor

facebook-github-bot commented Jul 11, 2024

This pull request was exported from Phabricator. Differential Revision: D59651199

helunwencser added a commit that referenced this pull request


          Let models provider their own specific special tokens

225ebe8

Differential Revision: [D59651199](https://our.internmc.facebook.com/intern/diff/D59651199/)

ghstack-source-id: 233429303
Pull Request resolved: #4227

facebook-github-bot added the fb-exported label


          Update on "Let models provider their own specific special tokens"

724a909

Differential Revision: [D59651199](https://our.internmc.facebook.com/intern/diff/D59651199/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Jul 12, 2024

This pull request was exported from Phabricator. Differential Revision: D59651199

helunwencser added a commit that referenced this pull request


          Let models provider their own specific special tokens

67afae7

Pull Request resolved: #4227

Differential Revision: [D59651199](https://our.internmc.facebook.com/intern/diff/D59651199/)
ghstack-source-id: 233472217


          Update on "Let models provider their own specific special tokens"

df1153b

Differential Revision: [D59651199](https://our.internmc.facebook.com/intern/diff/D59651199/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Jul 12, 2024

This pull request was exported from Phabricator. Differential Revision: D59651199

helunwencser added a commit that referenced this pull request


          Let models provider their own specific special tokens

b78d8b2

Pull Request resolved: #4227


ghstack-source-id: 233473801

Differential Revision: [D59651199](https://our.internmc.facebook.com/intern/diff/D59651199/)

This was referenced Jul 12, 2024

Only include base64.h in tiktoken.cpp #4234

Closed

Remove llama related stuff out of bpe_tokenizer #4235

Closed

larryliu0820 reviewed

View reviewed changes

Contributor

larryliu0820 left a comment

I like composition more than inheritance. Do you think it's better to keep the Tiktoken class and pass the special tokens through constructor? This way we don't have to create one class per model architecture.


          Update on "Let models provider their own specific special tokens"

1c8184c

Differential Revision: [D59651199](https://our.internmc.facebook.com/intern/diff/D59651199/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Jul 12, 2024

This pull request was exported from Phabricator. Differential Revision: D59651199


          Update on "Let models provider their own specific special tokens"

3ee6876

Differential Revision: [D59651199](https://our.internmc.facebook.com/intern/diff/D59651199/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Jul 12, 2024

This pull request was exported from Phabricator. Differential Revision: D59651199


          Update on "Let models provider their own specific special tokens"

d21c440

Differential Revision: [D59651199](https://our.internmc.facebook.com/intern/diff/D59651199/)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Jul 12, 2024

This pull request was exported from Phabricator. Differential Revision: D59651199

helunwencser mentioned this pull request

Move tokenizer.py into extension/llm/tokenizer #4255

Closed

larryliu0820 approved these changes

View reviewed changes

facebook-github-bot closed this in

d159de2

Contributor

facebook-github-bot commented Jul 12, 2024

This pull request has been merged in d159de2.

facebook-github-bot added the Merged label

helunwencser added a commit that referenced this pull request


          Fix build failure instroduced by #4227

62ce99a

Differential Revision: [D59765823](https://our.internmc.facebook.com/intern/diff/D59765823/)

[ghstack-poisoned]

helunwencser added a commit that referenced this pull request


          Fix build failure instroduced by #4227

eb888e4

Differential Revision: [D59765823](https://our.internmc.facebook.com/intern/diff/D59765823/)

ghstack-source-id: 233783029
Pull Request resolved: #4267

helunwencser added a commit that referenced this pull request


          Update on "Fix build failure instroduced by #4227"

2cf6aae

Differential Revision: [D59765823](https://our.internmc.facebook.com/intern/diff/D59765823/)

[ghstack-poisoned]

helunwencser added a commit that referenced this pull request


          Fix build failure instroduced by #4227

3680ef0

Pull Request resolved: #4267

Differential Revision: [D59765823](https://our.internmc.facebook.com/intern/diff/D59765823/)
ghstack-source-id: 233787226

helunwencser added a commit that referenced this pull request


          Update on "Fix build failure instroduced by #4227"

7c9c0d4

Differential Revision: [D59765823](https://our.internmc.facebook.com/intern/diff/D59765823/)

[ghstack-poisoned]

helunwencser added a commit that referenced this pull request


          Fix build failure instroduced by #4227

9ef685a

Pull Request resolved: #4267


ghstack-source-id: 233789064

Differential Revision: [D59765823](https://our.internmc.facebook.com/intern/diff/D59765823/)

helunwencser added a commit that referenced this pull request


          Update on "Fix build failure instroduced by #4227"

9ead3f8

Differential Revision: [D59765823](https://our.internmc.facebook.com/intern/diff/D59765823/)

[ghstack-poisoned]

helunwencser added a commit that referenced this pull request


          Fix build failure instroduced by #4227

6cab3e8

Pull Request resolved: #4267


ghstack-source-id: 233791856

Differential Revision: [D59765823](https://our.internmc.facebook.com/intern/diff/D59765823/)

facebook-github-bot pushed a commit that referenced this pull request


          Fix build failure instroduced by #4227 (#4267)

fc20d21

Summary:
Pull Request resolved: #4267

ghstack-source-id: 233791856

Reviewed By: kirklandsign

Differential Revision: D59765823

fbshipit-source-id: 80d10fb5a87f164f9c107487247b6f8022293d33

kedarnath03 pushed a commit to kedarnath03/executorch that referenced this pull request


          Let models provider their own specific special tokens

f487112

Pull Request resolved: pytorch/executorch#4227


ghstack-source-id: 233588006

Differential Revision: [D59651199](https://our.internmc.facebook.com/intern/diff/D59651199/)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed fb-exported Merged