MTK Android Llama Runner #6208

cmodi-meta · 2024-10-15T00:30:23Z

Adding the new MediaTek Runner that will work with the Android Demo app.

Run script to generate aar like below:

export NEURON_BUFFER_ALLOCATOR_LIB=path_to_buffer_allocator/libneuron_buffer_allocator.so 
export NEURON_USDK_ADAPTER_LIB=path_to_usdk_adapter/libneuronusdk_adapter.mtk.so  
export ANDROID_ABIS=arm64-v8a
sh build/build_android_llm_demo.sh

.aar file will live in examples/demo-apps/android/Llamademo/app/libs as executorch-llama.aar

Note: The new runner (mtk_llama_runner.cpp) is a fork of the existing mtk_llama_executor_runner.cpp. If mtk_llama_executor_runner.cpp is modified then mtk_llama_runner.cpp will need to as well. Another alternative is to adopt the mtk_llama_runner.cpp and it's flow as primary.

pytorch-bot · 2024-10-15T00:30:25Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6208

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 6db8726 with merge base 2c32bf3 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

cmodi-meta · 2024-10-15T00:34:58Z

TODO: Name space changes (reflect latest on #5478)

cmodi-meta · 2024-10-15T00:37:27Z

Also pointing to @kirklandsign on how he wants to handle the extension/android/CMakeLists.txt changes.

ref: https://github.com/pytorch/executorch/pull/5301/files#diff-6cfdc4894f08602902337ade7d271ab292754ef664107aa216ac6f30350c48d5

examples/mediatek/executor_runner/llama_runner/llm_helper/include/llama_runner_values.h

kirklandsign · 2024-10-15T17:11:28Z

extension/android/CMakeLists.txt

@@ -155,6 +155,26 @@ if(EXECUTORCH_BUILD_LLAMA_JNI)
    ${EXECUTORCH_ROOT}/examples/models/llama2/runner
    ${CMAKE_CURRENT_BINARY_DIR}/../../examples/models/llama2/runner
  )
+
+    target_sources(


We should protect this under a flag

aligned with @kirklandsign that he will help make change to this as it is CMake

kirklandsign · 2024-10-15T17:12:05Z

extension/android/jni/jni_layer_llama.cpp

@@ -17,6 +17,7 @@

 #include <executorch/examples/models/llama2/runner/runner.h>
 #include <executorch/examples/models/llava/runner/llava_runner.h>
+#include <executorch/examples/mediatek/executor_runner/mtk_llama_runner.h>


Probably need to put this under a preprocessor macro. We don't want to add MTK header library to our build always

aligned with @kirklandsign that he will help make change to this as he would like to refactor jni a bit.

cmodi-meta · 2024-10-16T18:31:10Z

TODO: Name space changes (reflect latest on #5478)

completed in commit 4e310cb in PR stack

cmodi-meta · 2024-10-16T18:41:23Z

@kirklandsign as part of the build scripts and jni changes you'll make. Just an fyi that there is an lintrunner error on the mixed upper and lower case:

    >>> 175  |  ADD_LIBRARY(libneuron_buffer_allocator SHARED IMPORTED)
    >>> 176  |  SET_PROPERTY(TARGET libneuron_buffer_allocator PROPERTY IMPORTED_LOCATION ${NEURON_BUFFER_ALLOCATOR_LIB})

extension/android/CMakeLists.txt

kirklandsign · 2024-10-18T23:30:19Z

examples/mediatek/executor_runner/llama_runner/llm_helper/include/llama_runner_values.h

+const std::string TOKENIZER_PATH =
+    "/data/local/tmp/et-mtk/llama3/tokenizer.model";
+const std::string TOKEN_EMBEDDING_PATH =
+    "/data/local/tmp/et-mtk/llama3/embedding_llama3-8B-instruct_fp32.bin";


Need to fix those

for right now, tokenizer path, token embedding path and model paths will be hardcoded in aar. We will then make changes to see if we want to have a different flow.

cccclai · 2024-10-21T23:21:08Z

@neuropilot-captain please take a look at the media llama runner change

facebook-github-bot · 2024-10-22T18:26:58Z

@cmodi-meta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

cccclai

Accept the same reason as #6304, @neuropilot-captain please let us know if you have any concern

cccclai · 2024-10-22T19:14:35Z

build/build_android_llm_demo.sh

@@ -37,6 +37,7 @@ build_android_native_library() {
  cmake . -DCMAKE_INSTALL_PREFIX="${CMAKE_OUT}" \
    -DCMAKE_TOOLCHAIN_FILE="${ANDROID_NDK}/build/cmake/android.toolchain.cmake" \
    -DANDROID_ABI="${ANDROID_ABI}" \
+    -DANDROID_PLATFORM=android-26 \


I thought we bump to more recent version?

Seems that here 26 is needed? Thought 30 is stricter.

26 is needed here since otherwise I get errors in the build like attached

cmodi-meta · 2024-10-29T18:37:30Z

rebase

facebook-github-bot · 2024-10-29T18:37:52Z

@cmodi-meta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-10-29T18:46:11Z

@cmodi-meta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 15, 2024

kirklandsign reviewed Oct 15, 2024

View reviewed changes

examples/mediatek/executor_runner/llama_runner/llm_helper/include/llama_runner_values.h Show resolved Hide resolved

kirklandsign reviewed Oct 15, 2024

View reviewed changes

cmodi-meta force-pushed the mtk-runner-landing branch from c097f9f to 4e310cb Compare October 16, 2024 18:29

cmodi-meta mentioned this pull request Oct 16, 2024

Android App with MediaTek Mode #6304

Merged

cmodi-meta commented Oct 18, 2024

View reviewed changes

extension/android/CMakeLists.txt Show resolved Hide resolved

cmodi-meta force-pushed the mtk-runner-landing branch from 7020c1a to a167d66 Compare October 18, 2024 00:13

kirklandsign force-pushed the mtk-runner-landing branch 4 times, most recently from 82d4ff2 to 03c12c8 Compare October 18, 2024 23:29

kirklandsign reviewed Oct 18, 2024

View reviewed changes

kirklandsign approved these changes Oct 21, 2024

View reviewed changes

cmodi-meta force-pushed the mtk-runner-landing branch from f56b488 to b2bca6e Compare October 21, 2024 23:00

cccclai approved these changes Oct 22, 2024

View reviewed changes

cmodi-meta mentioned this pull request Oct 29, 2024

Add MediaTek Llama Runner in Android App Readme #6548

Merged

cmodi-meta added 6 commits October 29, 2024 11:36

MTK Android Llama Runner

22a1264

Enable JNI with MTK Llama Runner core functions

5aa82ad

Cmake to include mtk target source

00946af

namespace changes to runner and jni layer

54123a4

lintrunner formatting

55eba6f

protect cmakelist for extension under NEURON_BUFFER_ALLOCATOR_LIB flag

5ef4ed2

kirklandsign and others added 5 commits October 29, 2024 11:36

llama2 -> llama

314c8dd

Use common LLM interface

cdbcab2

Add android-26 and rename runner_inferface to irunner

56a83dd

lint fix

12993c8

linter

6db8726

cmodi-meta force-pushed the mtk-runner-landing branch from a4ff680 to 6db8726 Compare October 29, 2024 18:37

facebook-github-bot merged commit 47bca20 into main Oct 29, 2024
40 checks passed

facebook-github-bot deleted the mtk-runner-landing branch October 29, 2024 20:20

MTK Android Llama Runner #6208

MTK Android Llama Runner #6208

Uh oh!

Conversation

cmodi-meta commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6208

✅ No Failures

Uh oh!

cmodi-meta commented Oct 15, 2024

Uh oh!

cmodi-meta commented Oct 15, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmodi-meta commented Oct 16, 2024

Uh oh!

cmodi-meta commented Oct 16, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cccclai commented Oct 21, 2024

Uh oh!

facebook-github-bot commented Oct 22, 2024

Uh oh!

cccclai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmodi-meta commented Oct 29, 2024

Uh oh!

facebook-github-bot commented Oct 29, 2024

Uh oh!

facebook-github-bot commented Oct 29, 2024

Uh oh!

Uh oh!

Uh oh!

cmodi-meta commented Oct 15, 2024 •

edited

Loading

pytorch-bot bot commented Oct 15, 2024 •

edited

Loading