Update base for Update on "use --use_sdpa_with_kv_cache for 1B/3B bf16"

We should use this option during exporting 1B/3B models as bf16 because KVCache is always fp32. Otherwise, we see regressed performance for 1B/3B in bf16 format.

Differential Revision: [D63871048](https://our.internmc.facebook.com/intern/diff/D63871048/)

[ghstack-poisoned]

Author: helunwencser

backends/arm/test/ops/test_sigmoid.py

@@ -152,12 +152,32 @@ def test_sigmoid_tosa_MI(
     def test_sigmoid_tosa_BI(self, test_name: str, test_data: torch.Tensor):
         self._test_sigmoid_tosa_BI_pipeline(self.Sigmoid(), (test_data,))
 
+    def test_add_sigmoid_tosa_MI(self):
+        self._test_sigmoid_tosa_MI_pipeline(self.AddSigmoid(), (test_data_suite[0][1],))
+
+    @unittest.skip(
+        reason="Started to fails when PyTorch 2.5->2.6 https://github.com/pytorch/executorch/issues/5832"
+    )
     def test_add_sigmoid_tosa_BI(self):
         self._test_sigmoid_tosa_BI_pipeline(self.AddSigmoid(), (test_data_suite[0][1],))
 
+    def test_sigmoid_add_tosa_MI(self):
+        self._test_sigmoid_tosa_MI_pipeline(self.SigmoidAdd(), (test_data_suite[0][1],))
+
+    @unittest.skip(
+        reason="Started to fails when PyTorch 2.5->2.6 https://github.com/pytorch/executorch/issues/5832"
+    )
     def test_sigmoid_add_tosa_BI(self):
         self._test_sigmoid_tosa_BI_pipeline(self.SigmoidAdd(), (test_data_suite[0][1],))
 
+    def test_sigmoid_add_sigmoid_tosa_MI(self):
+        self._test_sigmoid_tosa_MI_pipeline(
+            self.SigmoidAddSigmoid(), (test_data_suite[4][1], test_data_suite[3][1])
+        )
+
+    @unittest.skip(
+        reason="Started to fails when PyTorch 2.5->2.6 https://github.com/pytorch/executorch/issues/5832"
+    )
     def test_sigmoid_add_sigmoid_tosa_BI(self):
         self._test_sigmoid_tosa_BI_pipeline(
             self.SigmoidAddSigmoid(), (test_data_suite[4][1], test_data_suite[3][1])

backends/mediatek/scripts/README.md

@@ -16,7 +16,7 @@ Before you begin, ensure you have the following prerequisites installed and conf
 
 ### 2. Android NDK
 
-- **Download Android NDK**: Acquire the Android NDK from the [Android developer site](https://developer.android.com/ndk/downloads).
+- **Download Android NDK**: Acquire the Android NDK version 26.3.11579264 from the [Android developer site](https://developer.android.com/ndk/downloads).
 - **Set NDK Path**: Ensure that the `$ANDROID_NDK` environment variable is set to the path where the NDK is located.
    ```bash
    export ANDROID_NDK=<path_to_android_ndk>

examples/demo-apps/android/LlamaDemo/README.md

@@ -34,6 +34,12 @@ First it’s important to note that currently ExecuTorch provides support across
 | QNN (Qualcomm AI Accelerators)  | [link](https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/LlamaDemo/docs/delegates/qualcomm_README.md) |
 | MediaTek (MediaTek AI Accelerators)  | [link](https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/LlamaDemo/docs/delegates/mediatek_README.md)  |
 
+**WARNING** NDK r27 will cause issues like:
+  ```
+  java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "_ZTVNSt6__ndk114basic_ifstreamIcNS_11char_traitsIcEEEE" referenced by "/data/app/~~F5IwquaXUZPdLpSEYA-JGA==/com.example.executorchllamademo-FSyx80gEhsQCsxz7hvS2Ew==/lib/arm64/libexecutorch.so"...
+  ```
+  Please use NDK version 26.3.11579264.
+
 ## How to Use the App
 
 This section will provide the main steps to use the app, along with a code snippet of the ExecuTorch API.

examples/demo-apps/android/LlamaDemo/docs/delegates/qualcomm_README.md

@@ -6,13 +6,13 @@ More specifically, it covers:
 2. Building and linking libraries that are required to inference on-device for Android platform using Qualcomm AI accelerators.
 3. Building the Android demo app itself.
 
-Verified on Linux CentOS, QNN SDK [v2.26](https://softwarecenter.qualcomm.com/api/download/software/qualcomm_neural_processing_sdk/v2.26.0.240828.zip), python 3.10, Android SDK r27 and r26b.
+Verified on Linux CentOS, QNN SDK [v2.26](https://softwarecenter.qualcomm.com/api/download/software/qualcomm_neural_processing_sdk/v2.26.0.240828.zip), python 3.10, Android SDK r26c.
 
 Phone verified: OnePlus 12, Samsung 24+, Samsung 23
 
 ## Prerequisites
 * Download and unzip QNN SDK [v2.26](https://softwarecenter.qualcomm.com/api/download/software/qualcomm_neural_processing_sdk/v2.26.0.240828.zip)
-* Download and unzip Android SDK [r27](https://developer.android.com/ndk/downloads)
+* Download and unzip Android SDK [r26](https://developer.android.com/ndk/downloads)
 * Android phone with Snapdragon8 Gen3 (SM8650) or Gen2 (SM8550). Gen 1 and lower SoC might be supported but not fully validated.
 * Desired Llama model weights in .PTH format. You can download them on HuggingFace ([Example](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)).
 
@@ -40,7 +40,7 @@ Install dependencies
 ## Setup QNN
 ```
 # Set these variables correctly for your environment
-export ANDROID_NDK_ROOT=$HOME/android-ndk-r27 # Download android SDK and unzip to home directory
+export ANDROID_NDK_ROOT=$HOME/android-ndk-r26 # Download android SDK and unzip to home directory
 export QNN_SDK_ROOT=$HOME/Your-SDK-Root #Folder contains lib
 export EXECUTORCH_ROOT=$HOME/repos/executorch
 export LD_LIBRARY_PATH=$QNN_SDK_ROOT/lib/x86_64-linux-clang/:$LD_LIBRARY_PATH

examples/demo-apps/android/LlamaDemo/docs/delegates/xnnpack_README.md

@@ -16,7 +16,11 @@ Phone verified: OnePlus 12, OnePlus 9 Pro. Samsung S23 (Llama only), Samsung S24
 
 ## Prerequisites
 * Install [Java 17 JDK](https://www.oracle.com/java/technologies/javase/jdk17-archive-downloads.html).
-* Install the [Android SDK API Level 34](https://developer.android.com/about/versions/15/setup-sdk) and [Android NDK 26.3.11579264](https://developer.android.com/studio/projects/install-ndk).
+* Install the [Android SDK API Level 34](https://developer.android.com/about/versions/15/setup-sdk) and [Android NDK 26.3.11579264](https://developer.android.com/studio/projects/install-ndk). **WARNING** NDK r27 will cause issues like:
+  ```
+  java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "_ZTVNSt6__ndk114basic_ifstreamIcNS_11char_traitsIcEEEE" referenced by "/data/app/~~F5IwquaXUZPdLpSEYA-JGA==/com.example.executorchllamademo-FSyx80gEhsQCsxz7hvS2Ew==/lib/arm64/libexecutorch.so"...
+  ```
+  Please downgrade to version 26.3.11579264.
 * If you have Android Studio set up, you can install them with
   * Android Studio Settings -> Language & Frameworks -> Android SDK -> SDK Platforms -> Check the row with API Level 34.
   * Android Studio Settings -> Language & Frameworks -> Android SDK -> SDK Tools -> Check NDK (Side by side) row.
@@ -135,6 +139,7 @@ export ANDROID_NDK=<path_to_android_ndk>
 export ANDROID_ABI=arm64-v8a
 ```
 *Note: <path_to_android_ndk> is the root for the NDK, which is usually under ~/Library/Android/sdk/ndk/XX.Y.ZZZZZ for macOS, and contains NOTICE and README.md. We use <path_to_android_ndk>/build/cmake/android.toolchain.cmake for CMake to cross-compile.*
+
 3. Build the Android Java extension code:
 ```
 pushd extension/android

examples/models/flamingo/__init__.py renamed to examples/models/llama3_2_mm/__init__.py

@@ -6,7 +6,7 @@
  * LICENSE file in the root directory of this source tree.
  */
 
-#include <executorch/examples/models/flamingo/cross_attention/cross_attention_mask.h>
+#include <executorch/examples/models/llama3_2_mm/cross_attention/cross_attention_mask.h>
 
 #include <algorithm>
 #include <string>

examples/models/flamingo/cross_attention/cross_attention_mask.cpp renamed to examples/models/llama3_2_mm/cross_attention/cross_attention_mask.cpp

@@ -6,7 +6,7 @@
  * LICENSE file in the root directory of this source tree.
  */
 
-#include <executorch/examples/models/flamingo/cross_attention/cross_attention_mask.h>
+#include <executorch/examples/models/llama3_2_mm/cross_attention/cross_attention_mask.h>
 
 #include <gtest/gtest.h>
 

examples/models/flamingo/cross_attention/cross_attention_mask.h renamed to examples/models/llama3_2_mm/cross_attention/cross_attention_mask.h

@@ -5,7 +5,7 @@
 # LICENSE file in the root directory of this source tree.
 
 import torch
-from executorch.examples.models.flamingo.preprocess.export_preprocess_lib import (
+from executorch.examples.models.llama3_2_mm.preprocess.export_preprocess_lib import (
     export_preprocess,
     get_example_inputs,
     lower_to_executorch_preprocess,

examples/models/flamingo/cross_attention/cross_attention_mask_test.cpp renamed to examples/models/llama3_2_mm/cross_attention/cross_attention_mask_test.cpp

@@ -6,7 +6,7 @@
  * LICENSE file in the root directory of this source tree.
  */
 
-#include "executorch/examples/models/flamingo/preprocess/preprocess.h"
+#include "executorch/examples/models/llama3_2_mm/preprocess/preprocess.h"
 
 #include <algorithm>
 #include <cassert>

examples/models/flamingo/cross_attention/targets.bzl renamed to examples/models/llama3_2_mm/cross_attention/targets.bzl

@@ -6,7 +6,7 @@
  * LICENSE file in the root directory of this source tree.
  */
 
-#include <executorch/examples/models/flamingo/preprocess/preprocess.h>
+#include <executorch/examples/models/llama3_2_mm/preprocess/preprocess.h>
 #include <gtest/gtest.h>
 
 using namespace ::testing;

examples/models/flamingo/install_requirements.sh renamed to examples/models/llama3_2_mm/install_requirements.sh

@@ -48,9 +48,9 @@ test_buck2_select_ops_in_list() {
     ${PYTHON_EXECUTABLE} -m examples.portable.scripts.export --model_name="add_mul"
 
     echo "Running selective build test"
-    # set max_kernel_num=21: 19 primops, add, mul
+    # set max_kernel_num=22: 19 primops, add, mul
     $BUCK run //examples/selective_build:selective_build_test \
-        --config=executorch.max_kernel_num=21 \
+        --config=executorch.max_kernel_num=22 \
         --config=executorch.select_ops=list \
         -- --model_path=./add_mul.pte
 
@@ -117,11 +117,11 @@ test_cmake_select_ops_in_list() {
 
     local example_dir=examples/selective_build
     local build_dir=cmake-out/${example_dir}
-    # set MAX_KERNEL_NUM=21: 19 primops, add, mul
+    # set MAX_KERNEL_NUM=22: 19 primops, add, mul
     rm -rf ${build_dir}
     retry cmake -DBUCK2="$BUCK" \
             -DCMAKE_BUILD_TYPE=Release \
-            -DMAX_KERNEL_NUM=21 \
+            -DMAX_KERNEL_NUM=22 \
             -DEXECUTORCH_SELECT_OPS_LIST="aten::convolution.out,\
 aten::_native_batch_norm_legit_no_training.out,aten::hardtanh.out,aten::add.out,\
 aten::mean.out,aten::view_copy.out,aten::permute_copy.out,aten::addmm.out,\

examples/models/flamingo/preprocess/__init__.py renamed to examples/models/llama3_2_mm/preprocess/__init__.py

@@ -21,8 +21,30 @@
 #include <executorch/runtime/executor/method.h>
 #include <executorch/runtime/executor/program.h>
 
-namespace torch {
-namespace executor {
+using ::executorch::aten::Tensor;
+using ::executorch::extension::BufferDataLoader;
+using ::executorch::runtime::ArrayRef;
+using ::executorch::runtime::Backend;
+using ::executorch::runtime::BackendExecutionContext;
+using ::executorch::runtime::BackendInitContext;
+using ::executorch::runtime::CompileSpec;
+using ::executorch::runtime::DelegateHandle;
+using ::executorch::runtime::Error;
+using ::executorch::runtime::EValue;
+using ::executorch::runtime::FreeableBuffer;
+using ::executorch::runtime::HierarchicalAllocator;
+using ::executorch::runtime::MemoryAllocator;
+using ::executorch::runtime::MemoryManager;
+using ::executorch::runtime::Method;
+using ::executorch::runtime::MethodMeta;
+using ::executorch::runtime::Program;
+using ::executorch::runtime::Result;
+using ::executorch::runtime::Span;
+using ::executorch::runtime::Tag;
+using ::executorch::runtime::internal::copy_tensor_data;
+
+namespace example {
+
 /**
  * ExecutorBackend is a backend to execute an executorch program via delegate.
  * In preprocess, the preprocesed bytes (delegate blob) is an executorch
@@ -51,8 +73,8 @@ class ExecutorBackend final : public ::executorch::runtime::BackendInterface {
     // will return the data directly without copying it.
     MemoryAllocator* runtime_allocator = context.get_runtime_allocator();
     auto loader = ET_ALLOCATE_INSTANCE_OR_RETURN_ERROR(
-        runtime_allocator, util::BufferDataLoader);
-    new (loader) util::BufferDataLoader(processed->data(), processed->size());
+        runtime_allocator, BufferDataLoader);
+    new (loader) BufferDataLoader(processed->data(), processed->size());
     // Can't free `processed` because the program will point into that memory.
 
     // Try loading the program.
@@ -150,7 +172,7 @@ class ExecutorBackend final : public ::executorch::runtime::BackendInterface {
       if (output.tag == Tag::Tensor) {
         Tensor t_src = output.toTensor();
         Tensor t_dst = args[num_inputs + i]->toTensor();
-        status = internal::copy_tensor_data(t_dst, t_src);
+        status = copy_tensor_data(t_dst, t_src);
       }
     }
 
@@ -165,12 +187,11 @@ class ExecutorBackend final : public ::executorch::runtime::BackendInterface {
   }
 };
 
-Error registerExecutorBackend() {
+Error register_executor_backend() {
   static auto cls = ExecutorBackend();
   static Backend backend{"ExecutorBackend", &cls};
   static auto success_with_compiler = register_backend(backend);
   return success_with_compiler;
 }
 
-} // namespace executor
-} // namespace torch
+} // namespace example

examples/models/flamingo/preprocess/export_preprocess.py renamed to examples/models/llama3_2_mm/preprocess/export_preprocess.py

@@ -10,10 +10,8 @@
 
 #include <executorch/runtime/core/error.h>
 
-namespace torch {
-namespace executor {
+namespace example {
 
-Error registerExecutorBackend();
+::executorch::runtime::Error register_executor_backend();
 
-} // namespace executor
-} // namespace torch
+} // namespace example

examples/models/flamingo/preprocess/export_preprocess_lib.py renamed to examples/models/llama3_2_mm/preprocess/export_preprocess_lib.py

@@ -10,10 +10,9 @@
 #include <executorch/runtime/backend/interface.h>
 #include <executorch/runtime/core/error.h>
 
-namespace torch {
-namespace executor {
+namespace example {
 namespace {
-static Error register_success = registerExecutorBackend();
+static ::executorch::runtime::Error register_success =
+    register_executor_backend();
 } // namespace
-} // namespace executor
-} // namespace torch
+} // namespace example

examples/models/flamingo/preprocess/preprocess.cpp renamed to examples/models/llama3_2_mm/preprocess/preprocess.cpp

@@ -11,7 +11,7 @@
 		03B0118E2CAC567900054791 /* DynamicTestCase.m in Sources */ = {isa = PBXBuildFile; fileRef = 03B0118C2CAC567900054791 /* DynamicTestCase.m */; };
 		03B011912CAD114E00054791 /* ResourceTestCase.m in Sources */ = {isa = PBXBuildFile; fileRef = 03B011902CAD114E00054791 /* ResourceTestCase.m */; };
 		03B2D37A2C8A515C0046936E /* GenericTests.mm in Sources */ = {isa = PBXBuildFile; fileRef = 03B2D3792C8A515C0046936E /* GenericTests.mm */; };
-		03C7FA382C8AA3EC00E6E9AE /* Models in Resources */ = {isa = PBXBuildFile; fileRef = 03C7FA322C8AA24200E6E9AE /* Models */; };
+		03C7FA382C8AA3EC00E6E9AE /* Resources in Resources */ = {isa = PBXBuildFile; fileRef = 03C7FA322C8AA24200E6E9AE /* Resources */; };
 		03DD00A92C8FE44600FE4619 /* backend_coreml.xcframework in Frameworks */ = {isa = PBXBuildFile; fileRef = 03DD00992C8FE44600FE4619 /* backend_coreml.xcframework */; };
 		03DD00AA2C8FE44600FE4619 /* kernels_custom.xcframework in Frameworks */ = {isa = PBXBuildFile; fileRef = 03DD009A2C8FE44600FE4619 /* kernels_custom.xcframework */; };
 		03DD00AF2C8FE44600FE4619 /* kernels_portable.xcframework in Frameworks */ = {isa = PBXBuildFile; fileRef = 03DD009F2C8FE44600FE4619 /* kernels_portable.xcframework */; };
@@ -50,7 +50,7 @@
 		03B0118F2CAD114E00054791 /* ResourceTestCase.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = ResourceTestCase.h; sourceTree = "<group>"; };
 		03B011902CAD114E00054791 /* ResourceTestCase.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = ResourceTestCase.m; sourceTree = "<group>"; };
 		03B2D3792C8A515C0046936E /* GenericTests.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; path = GenericTests.mm; sourceTree = "<group>"; };
-		03C7FA322C8AA24200E6E9AE /* Models */ = {isa = PBXFileReference; lastKnownFileType = folder; path = Models; sourceTree = SOURCE_ROOT; };
+		03C7FA322C8AA24200E6E9AE /* Resources */ = {isa = PBXFileReference; lastKnownFileType = folder; path = Resources; sourceTree = SOURCE_ROOT; };
 		03DD00992C8FE44600FE4619 /* backend_coreml.xcframework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.xcframework; name = backend_coreml.xcframework; path = Frameworks/backend_coreml.xcframework; sourceTree = "<group>"; };
 		03DD009A2C8FE44600FE4619 /* kernels_custom.xcframework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.xcframework; name = kernels_custom.xcframework; path = Frameworks/kernels_custom.xcframework; sourceTree = "<group>"; };
 		03DD009F2C8FE44600FE4619 /* kernels_portable.xcframework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.xcframework; name = kernels_portable.xcframework; path = Frameworks/kernels_portable.xcframework; sourceTree = "<group>"; };
@@ -105,7 +105,7 @@
 			children = (
 				03B2D3662C8A515A0046936E /* App */,
 				03ED6CEB2C8AAF5300F2D6EE /* Frameworks */,
-				03C7FA322C8AA24200E6E9AE /* Models */,
+				03C7FA322C8AA24200E6E9AE /* Resources */,
 				03B2D3782C8A515C0046936E /* Tests */,
 				03B0118D2CAC567900054791 /* TestUtils */,
 				03B2D3652C8A515A0046936E /* Products */,
@@ -261,7 +261,7 @@
 			isa = PBXResourcesBuildPhase;
 			buildActionMask = 2147483647;
 			files = (
-				03C7FA382C8AA3EC00E6E9AE /* Models in Resources */,
+				03C7FA382C8AA3EC00E6E9AE /* Resources in Resources */,
 			);
 			runOnlyForDeploymentPostprocessing = 0;
 		};

examples/models/flamingo/preprocess/preprocess.h renamed to examples/models/llama3_2_mm/preprocess/preprocess.h

@@ -0,0 +1,25 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+frameworks=(
+  "backend_coreml"
+  "backend_mps"
+  "backend_xnnpack"
+  "executorch"
+  "kernels_custom"
+  "kernels_optimized"
+  "kernels_portable"
+  "kernels_quantized"
+)
+
+for framework in "${frameworks[@]}"; do
+  rm -f "${f}-latest.zip"
+  rm -rf "${f}.xcframework"
+  curl -sSLO "https://ossci-ios.s3.amazonaws.com/executorch/${framework}-latest.zip" && \
+  unzip -q "${framework}-latest.zip" && \
+  rm "${framework}-latest.zip"
+done

examples/models/flamingo/preprocess/preprocess_test.cpp renamed to examples/models/llama3_2_mm/preprocess/preprocess_test.cpp

@@ -0,0 +1,4 @@
+# Resources
+
+This directory and all files in it will be copied to the bundle’s resources directory, accessible via `NSBundle.resourcePath`.
+Place any resource files you want to access at runtime here.

examples/models/flamingo/preprocess/targets.bzl renamed to examples/models/llama3_2_mm/preprocess/targets.bzl

@@ -29,8 +29,8 @@ @implementation GenericTests
 
 + (NSArray<NSString *> *)directories {
   return @[
-    @"Models",
-    @"aatp/data",
+    @"Resources",
+    @"aatp/data", // AWS Farm devides look at this subdirectory.
   ];
 }