Skip to content

Add support for tiktoken and refactored runner structure #435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Apr 24, 2024
Merged

Add support for tiktoken and refactored runner structure #435

merged 8 commits into from
Apr 24, 2024

Conversation

larryliu0820
Copy link
Contributor

Summary:

Unified runner and move runner-et/CMakeLists.txt to runner/et.cmake and runner-aoti/CMakeLists.txt to runner/aoti.cmake.

Added a root level CMakeLists.txt to build a tokenizer library and link to both targets separately.

In CLI we need to specify the target to run:

cmake --build ./cmake-out --target et_run/aoti_run

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Apr 23, 2024
mikekgfb
mikekgfb approved these changes Apr 24, 2024
View reviewed changes
Copy link
Contributor

@mikekgfb mikekgfb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@larryliu0820
Add support for tiktoken and refactored runner structure
ab90ffd 
Summary:

Unified runner and move runner-et/CMakeLists.txt to runner/et.cmake and
runner-aoti/CMakeLists.txt to runner/aoti.cmake.

Added a root level CMakeLists.txt to build a tokenizer library and link
to both targets separately.

In CLI we need to specify the target to run:

```
cmake --build ./cmake-out --target et_run/aoti_run
```

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@larryliu0820
Fix CI
2f3ed1a 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
mikekgfb
mikekgfb approved these changes Apr 24, 2024
View reviewed changes
Copy link
Contributor

@mikekgfb mikekgfb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

.gitmodules
@@ -0,0 +1,6 @@
[submodule "tokenizer/third-party/abseil-cpp"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to update instructions to do git clone recursive, git submodule updates?

This is what PyTorch tells users:

git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
# if you are updating an existing checkout
git submodule sync
git submodule update --init --recursive

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah where should I update the doc?

@larryliu0820
Fix more CI
123a13d 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@larryliu0820
Further fix CI
aa2e182 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@larryliu0820
Lint`
9428d15 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@larryliu0820
Update build_android.sh
16dc690 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@larryliu0820
Rebase
b475e6d 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@larryliu0820
Fix cmake commands in CI job
a975f2b 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@larryliu0820 larryliu0820 merged commit b54be7b into main Apr 24, 2024
malfet pushed a commit that referenced this pull request Jul 17, 2024
@larryliu0820 @malfet
Add support for tiktoken and refactored runner structure (#435)
895ce3e 
* Add support for tiktoken and refactored runner structure

Summary:

Unified runner and move runner-et/CMakeLists.txt to runner/et.cmake and
runner-aoti/CMakeLists.txt to runner/aoti.cmake.

Added a root level CMakeLists.txt to build a tokenizer library and link
to both targets separately.

In CLI we need to specify the target to run:

```
cmake --build ./cmake-out --target et_run/aoti_run
```

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix more CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Further fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Lint`

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Update build_android.sh

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Rebase

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix cmake commands in CI job

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
malfet pushed a commit that referenced this pull request Jul 17, 2024
@larryliu0820 @malfet
Add support for tiktoken and refactored runner structure (#435)
99145e7 
* Add support for tiktoken and refactored runner structure

Summary:

Unified runner and move runner-et/CMakeLists.txt to runner/et.cmake and
runner-aoti/CMakeLists.txt to runner/aoti.cmake.

Added a root level CMakeLists.txt to build a tokenizer library and link
to both targets separately.

In CLI we need to specify the target to run:

```
cmake --build ./cmake-out --target et_run/aoti_run
```

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix more CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Further fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Lint`

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Update build_android.sh

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Rebase

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix cmake commands in CI job

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
malfet pushed a commit that referenced this pull request Jul 17, 2024
@larryliu0820 @malfet
Add support for tiktoken and refactored runner structure (#435)
344fd4f 
* Add support for tiktoken and refactored runner structure

Summary:

Unified runner and move runner-et/CMakeLists.txt to runner/et.cmake and
runner-aoti/CMakeLists.txt to runner/aoti.cmake.

Added a root level CMakeLists.txt to build a tokenizer library and link
to both targets separately.

In CLI we need to specify the target to run:

```
cmake --build ./cmake-out --target et_run/aoti_run
```

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix more CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Further fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Lint`

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Update build_android.sh

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Rebase

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix cmake commands in CI job

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
malfet pushed a commit that referenced this pull request Jul 17, 2024
@larryliu0820 @malfet
Add support for tiktoken and refactored runner structure (#435)
2f3d837 
* Add support for tiktoken and refactored runner structure

Summary:

Unified runner and move runner-et/CMakeLists.txt to runner/et.cmake and
runner-aoti/CMakeLists.txt to runner/aoti.cmake.

Added a root level CMakeLists.txt to build a tokenizer library and link
to both targets separately.

In CLI we need to specify the target to run:

```
cmake --build ./cmake-out --target et_run/aoti_run
```

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix more CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Further fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Lint`

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Update build_android.sh

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Rebase

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix cmake commands in CI job

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
malfet pushed a commit that referenced this pull request Jul 17, 2024
@larryliu0820 @malfet
Add support for tiktoken and refactored runner structure (#435)
5727984 
* Add support for tiktoken and refactored runner structure

Summary:

Unified runner and move runner-et/CMakeLists.txt to runner/et.cmake and
runner-aoti/CMakeLists.txt to runner/aoti.cmake.

Added a root level CMakeLists.txt to build a tokenizer library and link
to both targets separately.

In CLI we need to specify the target to run:

```
cmake --build ./cmake-out --target et_run/aoti_run
```

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix more CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Further fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Lint`

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Update build_android.sh

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Rebase

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix cmake commands in CI job

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
malfet pushed a commit that referenced this pull request Jul 17, 2024
@larryliu0820 @malfet
Add support for tiktoken and refactored runner structure (#435)
de9c414 
* Add support for tiktoken and refactored runner structure

Summary:

Unified runner and move runner-et/CMakeLists.txt to runner/et.cmake and
runner-aoti/CMakeLists.txt to runner/aoti.cmake.

Added a root level CMakeLists.txt to build a tokenizer library and link
to both targets separately.

In CLI we need to specify the target to run:

```
cmake --build ./cmake-out --target et_run/aoti_run
```

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix more CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Further fix CI

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Lint`

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Update build_android.sh

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Rebase

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

* Fix cmake commands in CI job

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mikekgfb mikekgfb mikekgfb approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@larryliu0820 @mikekgfb @facebook-github-bot