Pick pybind (#6151)

* Add MethodMeta object for python visibility (#5571)

Summary:
Pull Request resolved: #5571

Some clients and consumers of the Executorch program files (.pte) were
requesting ways to access metadata like the sizes of tensors and the number
of bytes they needed.
When I told them how to access them in C++, they requested Python wrappers
since they had processing scripts written in Python.

Add some implementations of MethodMeta and TensorInfo methods.
Note that these become more expensive than in C++ because they need to
allocate python objects, but I doubt these are used in
performance-sensitive applications anyway. And dealing with
lifetimes of mixed C++/Python objects is complex, so I favored simple lifetimes.

Reviewed By: dbort

Differential Revision: D63288433

fbshipit-source-id: af775120a8ebd9bf455671a8ce1f158259aa50e6

* Add mapping from C++ program::verification to Python (#5915)

Summary:
As titled. This enables
`portable_lib._load_for_executorch[_from_buffer]` to accept `Program::Verification` argument.

See added test, now we can do something like:

```
from executorch.extension.pybindings.portable_lib import Verification
module = load_fn(
  exported_program.buffer,
  enable_etdump=False,
  debug_buffer_size=0,
  program_verification=Verification.Minimal,
)
```

Pull Request resolved: #5915

Test Plan: See unit test

Reviewed By: dbort

Differential Revision: D63987538

Pulled By: larryliu0820

fbshipit-source-id: b68d8d1149e2d46b90544679707f420179e72b19

* Find portable_lib.so in pip package during cmake build (#5961)

Summary:
* Rename `_portable_lib.cpython-3.<distribution info>.so` to `_portable_lib.so` so it can be found by CMake `find_library()`. This can be achieved by setting `SETUPTOOLS_EXT_SUFFIX`.
* Since `executorch-config.cmake` is also being used to find installed libraries such as `executorch.a`, `xnnpack_backend.a`, add a condition to tell if `executorch-config.cmake` is being used in cmake-out or site-packages.

Pull Request resolved: #5961

Reviewed By: metascroy

Differential Revision: D64014291

Pulled By: larryliu0820

fbshipit-source-id: 2757f2883d3f836e9efd45676f792c12f742e63d

* Improve pip package build (#5965)

Summary:
Addressing comments in #5961.

* Separate out `executorch-wheel-config.cmake` from `executorch-config.cmake`.
* Hardcode the envrionment variable `SETUPTOOLS_EXT_SUFFIX` in `setup.py`.

Pull Request resolved: #5965

Reviewed By: dbort

Differential Revision: D64017947

Pulled By: larryliu0820

fbshipit-source-id: 0bdff5e2d2ec5873540d1b701595c7a316e84e80

* Let find_package(executorch) find the correct include directory (#6102)

Summary:
There's a typo in `executorch-wheel-config.cmake` that points to the wrong `include` path:
```
<site-packages>/executorch/share/cmake/include
```

Where it actually should be
```
<site-packages>/executorch/include
```

Fixing this issue. Verified it on [build_torchao_ops.sh](https://github.com/pytorch/ao/blob/main/torchao/experimental/build_torchao_ops.sh)

Pull Request resolved: #6102

Reviewed By: lucylq

Differential Revision: D64189337

Pulled By: larryliu0820

fbshipit-source-id: 13033587f5499537623995b8f9457fb47d780340

* New Runtime pybind API (#6063)

Summary:
Based on this proposal: https://docs.google.com/document/d/10Q4-pt97inQQtFf-FjjwhMaDXXCfk1zGy6V6EkygNUY/edit#heading=h.fcrpnrtb6cud

Historically our pybinding APIs are not following the same C++ modeling
(Program, Method etc) and hence it's hard to use and easy to hit
footguns - for example, if we load the program and return it from a
python method, it goes out of the scope and releases the memory.

This effort is to create Pybind APIs that resembles C++ objects so it's
less confusing to the users.

Add the following python classes:
* `Runtime`: a singleton object hosting methods like `load_program`.
  Returns a `Program` object when calling `load_program`. Also exposes
  the operator registry
* `Program`: each pte file should have one `Program` object. Most
  important method is `load_method` which returns a `Method` object. It
  has a property `method_names` where we can inspect what methods are
  inside this .pte file.
* `Method`: one object per method name in a given `Program`. Exposes
  `execute` which takes in pytree flattened torch tensors as input and
  return pytree flattened output. It also exposes `MethodMeta` for users
  to inspect more information regarding input/output of this method.

Pull Request resolved: #6063

Reviewed By: dbort

Differential Revision: D64132360

Pulled By: larryliu0820

fbshipit-source-id: a2f35edc5fd8c200df0812a693e454d66d6a907e

* Lint

* Fix test_pybindings.py

---------

Co-authored-by: Riley Dulin <[email protected]>

Author: larryliu0820

build/executorch-wheel-config.cmake

@@ -0,0 +1,40 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Config defining how CMake should find ExecuTorch package. CMake will search
+# for this file and find ExecuTorch package if it is installed. Typical usage
+# is:
+#
+# find_package(executorch REQUIRED)
+# -------
+#
+# Finds the ExecuTorch library
+#
+# This will define the following variables:
+#
+#   EXECUTORCH_FOUND        -- True if the system has the ExecuTorch library
+#   EXECUTORCH_INCLUDE_DIRS -- The include directories for ExecuTorch
+#   EXECUTORCH_LIBRARIES    -- Libraries to link against
+#
+cmake_minimum_required(VERSION 3.19)
+
+# Find prebuilt _portable_lib.so. This file should be installed under
+# <site-packages>/executorch/share/cmake
+find_library(_portable_lib_LIBRARY _portable_lib.so PATHS "${CMAKE_CURRENT_LIST_DIR}/../../extension/pybindings/")
+set(EXECUTORCH_LIBRARIES)
+set(EXECUTORCH_FOUND OFF)
+if(_portable_lib_LIBRARY)
+  set(EXECUTORCH_FOUND ON)
+  message(STATUS "ExecuTorch portable library is found at ${_portable_lib_LIBRARY}")
+  list(APPEND EXECUTORCH_LIBRARIES _portable_lib)
+  add_library(_portable_lib STATIC IMPORTED)
+  set(EXECUTORCH_INCLUDE_DIRS ${CMAKE_CURRENT_LIST_DIR}/../../include)
+  set_target_properties(_portable_lib PROPERTIES
+    IMPORTED_LOCATION "${_portable_lib_LIBRARY}"
+    INTERFACE_INCLUDE_DIRECTORIES "${EXECUTORCH_INCLUDE_DIRS}"
+    CXX_STANDARD 17
+  )
+endif()

extension/pybindings/TARGETS

@@ -67,6 +67,7 @@ runtime.python_library(
     srcs = ["portable_lib.py"],
     visibility = [
         "//executorch/exir/...",
+        "//executorch/runtime/...",
         "@EXECUTORCH_CLIENTS",
     ],
     deps = [":_portable_lib"],

extension/pybindings/portable_lib.py

@@ -45,6 +45,8 @@
     _reset_profile_results,  # noqa: F401
     BundledModule,  # noqa: F401
     ExecuTorchModule,  # noqa: F401
+    MethodMeta,  # noqa: F401
+    Verification,  # noqa: F401
 )
 
 # Clean up so that `dir(portable_lib)` is the same as `dir(_portable_lib)`