Skip to content

[Android] Use prebuilts; add instructions for using them #469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 25, 2024
Merged

[Android] Use prebuilts; add instructions for using them #469

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
26dfb11
[Android] Use prebuilts; add instructions for using them
kirklandsign Apr 24, 2024
c733c76
Use persistent S3
kirklandsign Apr 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion docs/Android.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Executing LLM models on Android

## Option 1: Use ExecuTorch LLAMA Demo App

Check out the [tutorial on how to build an Android app running your
PyTorch models with
ExecuTorch](https://pytorch.org/executorch/main/llm/llama-demo-android.html),
Expand All @@ -9,4 +11,43 @@ and give your torchchat models a spin.

Detailed step by step in conjunction with ET Android build, to run on
simulator for Android. `scripts/android_example.sh` for running a
model on an Android simulator (on Mac)
model on an Android simulator (on Mac)

## Option 2: Integrate the Java API with your own app

We provide a Java library for you to integrate LLM runner to your own app.
See [this file](https://github.com/pytorch/executorch/blob/main/extension/android/src/main/java/org/pytorch/executorch/LlamaModule.java)
for Java APIs.

To add the Java library to your app, use helper functions `download_jar_library`
and `download_jni_library` from `scripts/android_example.sh` to download the
prebuilt libraries.

```bash
# my_build_script.sh
source scripts/android_example.sh
download_jar_library
download_jni_library
```

In your app working directory (for example executorch/examples/demo-apps/android/LlamaDemo),
copy the jar to your app libs:
```bash
mkdir -p app/libs
cp ${TORCHCHAT_ROOT}/build/android/executorch.jar app/libs/executorch.jar
```

In your Java app, add the jar file path to your gradle build rule.
```
# app/build.grardle.kts
dependencies {
implementation(files("libs/executorch.jar"))
}
```

Then copy the corresponding JNI library to your app:

```bash
mkdir -p app/src/main/jniLibs/arm64-v8a
cp ${TORCHCHAT_ROOT}/build/android/arm64-v8a/libexecutorch_llama_jni.so app/src/main/jniLibs/arm64-v8a
```
37 changes: 31 additions & 6 deletions scripts/android_example.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.

set -eu
set -eux

cd ${TORCHCHAT_ROOT}
echo "Inside: $TORCHCHAT_ROOT"
Expand All @@ -25,6 +25,10 @@ else
exit -1
fi

LLAMA_JNI_ARM64_URL="https://ossci-android.s3.us-west-1.amazonaws.com/executorch/release/0.2/arm64-v8a/libexecutorch_llama_jni.so"
Copy link
Contributor

@malfet malfet Apr 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, do you know what are the restrictions for uploading/modifying files in that bucket? I'm not concerned if those binaries got compromised for CI system, but I would never install anything from that bucket on user system without hardcoding a checksum on the receiving side

See how it's done here for example (not advising a separate file for this one, just general workflow
https://github.com/pytorch/pytorch/blob/751d9a319d5e703c2b726d39a5711c0112ff5648/tools/linter/adapters/s3_init.py#L120
and https://github.com/pytorch/pytorch/blob/751d9a319d5e703c2b726d39a5711c0112ff5648/tools/linter/adapters/s3_init_config.json#L11

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Huy: dev infra admins has permission to write to that bucket

LLAMA_JNI_X86_64_URL="https://ossci-android.s3.us-west-1.amazonaws.com/executorch/release/0.2/x86_64/libexecutorch_llama_jni.so"
LLAMA_JAR_URL="https://ossci-android.s3.us-west-1.amazonaws.com/executorch/release/0.2/executorch.jar"

mkdir -p ${TORCHCHAT_ROOT}/build/android

setup_java() {
Expand Down Expand Up @@ -64,30 +68,50 @@ setup_android_sdk_manager() {
setup_android_sdk() {
sdkmanager "platforms;android-34"
sdkmanager "platform-tools"
sdkmanager "emulator"
sdkmanager "system-images;android-34;google_apis;${ANDROID_ABI}"
}

setup_android_ndk() {
sdkmanager "ndk;25.0.8775105"
export ANDROID_NDK="$ANDROID_HOME/ndk/25.0.8775105"
}

download_jar_library() {
mkdir -p ${TORCHCHAT_ROOT}/build/android
curl "${LLAMA_JAR_URL}" -o ${TORCHCHAT_ROOT}/build/android/executorch.jar
}

download_jni_library() {
mkdir -p ${TORCHCHAT_ROOT}/build/android/arm64-v8a
mkdir -p ${TORCHCHAT_ROOT}/build/android/x86_64
if [ ! -f ${TORCHCHAT_ROOT}/build/android/arm64-v8a/libexecutorch_llama_jni.so ]; then
curl "${LLAMA_JNI_ARM64_URL}" -o ${TORCHCHAT_ROOT}/build/android/arm64-v8a/libexecutorch_llama_jni.so
fi
if [ ! -f ${TORCHCHAT_ROOT}/build/android/x86_64/libexecutorch_llama_jni.so ]; then
curl "${LLAMA_JNI_X86_64_URL}" -o ${TORCHCHAT_ROOT}/build/android/x86_64/libexecutorch_llama_jni.so
fi
}

build_app() {
pushd build/src/executorch/examples/demo-apps/android/LlamaDemo
./gradlew :app:setup
mkdir -p app/src/main/jniLibs/arm64-v8a
mkdir -p app/src/main/jniLibs/x86_64
cp ${TORCHCHAT_ROOT}/build/android/arm64-v8a/libexecutorch_llama_jni.so app/src/main/jniLibs/arm64-v8a
cp ${TORCHCHAT_ROOT}/build/android/x86_64/libexecutorch_llama_jni.so app/src/main/jniLibs/x86_64
./gradlew :app:build
popd
}

setup_avd() {
sdkmanager "emulator"
sdkmanager "system-images;android-34;google_apis;${ANDROID_ABI}"

avdmanager create avd --name "torchchat" --package "system-images;android-34;google_apis;${ANDROID_ABI}"
sdk/emulator/emulator @torchchat &
}

push_files_to_android() {
adb wait-for-device
adb shell mkdir /data/local/tmp/llama
adb shell mkdir -p /data/local/tmp/llama
adb push stories15M.pte /data/local/tmp/llama
adb push checkpoints/stories15M/tokenizer.bin /data/local/tmp/llama
adb install -t build/src/executorch/examples/demo-apps/android/LlamaDemo/app/build/outputs/apk/debug/app-debug.apk
Expand All @@ -98,7 +122,8 @@ if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
setup_android_sdk_manager
setup_android_sdk
setup_android_ndk
build_app
setup_avd
download_jni_library
build_app
push_files_to_android
fi