Fix index.rst links to tutorials (#737)

svekars · facebook-github-bot · commit 2628c0e5aa78 · 2023-10-10T13:41:18.000-07:00
Summary: Pull Request resolved: #737 - Fix links to tutorials - Add logic to copy .md files from "tutorials_source/" to "tutorials/" so that they can be added to the build and have urls like tutorials/<tutorial_name>.html instead of tutorials_source/... * Update instructions on how to add a tutorial from a dir outside of doc build. Reviewed By: dbort Differential Revision: D50091328 fbshipit-source-id: 1632fc027fd9a91b44306c5bd697e5c7223aba99
diff --git a/docs/README.md b/docs/README.md
@@ -1,8 +1,8 @@
 # ExecuTorch Documentation
 
-Welcome to the ExecuTorch documentation! This README.md will provide an
-overview of the ExecuTorch docs and its features, as well as instructions on
-how to contribute and build locally.
+Welcome to the ExecuTorch documentation! This README.md will provide an overview
+of the ExecuTorch docs and its features, as well as instructions on how to
+contribute and build locally.
 
 All current documentation is located in the `docs/source` directory.
 
@@ -13,12 +13,11 @@ All current documentation is located in the `docs/source` directory.
 - [Using Custom Variables](#using-custom-variables)
 - [Including READMEs to the Documentation Build](#including-readmes-to-the-documentation-build)
 - [Contributing](#contributing)
+- [Adding Tutorials](#adding-tutorials)
 - [Auto-generated API documentation](#auto-generated-api-documentation)
   - [Python APIs](#python-apis)
   - [C++ APIs](#c-apis)
-
-<!-- tocstop -->
-
+  <!-- tocstop -->
 
 ## Toolchain Overview
 
@@ -73,12 +72,12 @@ To build the documentation locally:
    make html
    ```
 
-   This should build both documentation and tutorials. The build will be
-   placed in the `_build` directory.
+   This should build both documentation and tutorials. The build will be placed
+   in the `_build` directory.
 
-1. You can preview locally by using [sphinx-serve](https://pypi.org/project/sphinx-serve/).
-   To install sphinx-serve, run: `pip3 install sphinx-serve`. To serve
-   your documentation:
+1. You can preview locally by using
+   [sphinx-serve](https://pypi.org/project/sphinx-serve/). To install
+   sphinx-serve, run: `pip3 install sphinx-serve`. To serve your documentation:
 
    ```
    sphinx-serve -b _build
@@ -89,11 +88,10 @@ To build the documentation locally:
 
 ## Using Custom Variables
 
-You can use custom variables in your `.md` and `.rst` files.
-The variables take their values from the files listed in the
-`./.ci/docker/ci_commit_pins/` directory.
-For example, to insert a variable that specifies the latest Buck2 version,
-use the following syntax:
+You can use custom variables in your `.md` and `.rst` files. The variables take
+their values from the files listed in the `./.ci/docker/ci_commit_pins/`
+directory. For example, to insert a variable that specifies the latest Buck2
+version, use the following syntax:
 
 ```
 The current version of Buck2 is ${executorch_version:buck2}.
@@ -105,59 +103,120 @@ This will result in the following output:
 
 We have the following custom variables defined in this docset:
 
-* `${executorch_version:buck2}`
-* `${executorch_version:nightly}`
-* `${executorch_version:pytorch}`
-* `${executorch_version:vision}`
-* `${executorch_version:audio}`
-
+- `${executorch_version:buck2}`
+- `${executorch_version:nightly}`
+- `${executorch_version:pytorch}`
+- `${executorch_version:vision}`
+- `${executorch_version:audio}`
 
 You can use the variables in both regular text and code blocks.
 
-
 ## Including READMEs to the Documentation Build
 
-You might want to include some of the `README.md` files from various
-directories in this repositories in your documentation build.
-To do that, create an `.md` file and use the `{include}` directive to
-insert your `.md` files. Example:
+You might want to include some of the `README.md` files from various directories
+in this repositories in your documentation build. To do that, create an `.md`
+file and use the `{include}` directive to insert your `.md` files. Example:
 
-```
+````
 ```{include} ../README.md
-```
+````
 
-**NOTE:** Many `README.md` files are written as placeholders with limited information
-provided. Some of that content you might want to keep in the repository rather
-than on the website. If you still want to add it, make sure to check
-the content for accuracy, structure, and overall quality.
+**NOTE:** Many `README.md` files are written as placeholders with limited
+information provided. Some of that content you might want to keep in the
+repository rather than on the website. If you still want to add it, make sure to
+check the content for accuracy, structure, and overall quality.
 
 ## Contributing
 
-Use the [PyTorch contributing guidelines](https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md#writing-documentation)
+Use the
+[PyTorch contributing guidelines](https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md#writing-documentation)
 to contribute to the documentation.
 
-In addition to that, see [Markdown in Sphinx Tips and Tricks](https://pytorch.org/executorch/markdown-sphinx-tips-tricks.html)
+In addition to that, see
+[Markdown in Sphinx Tips and Tricks](https://pytorch.org/executorch/markdown-sphinx-tips-tricks.html)
 for tips on how to author high-quality markdown pages with Myst Parser.
 
+## Adding Tutorials
+
+You can add both interactive (`.py`) and non-interactive tutorials (`.md`) to
+this documentation. All tutorials should go to the `tutorials_source/`
+directory. Use one of the following templates:
+
+- [Python Template](https://github.com/pytorch/executorch/blob/main/docs/source/tutorials_source/template_tutorial.py)
+- [Markdown template](https://github.com/pytorch/executorch/blob/main/docs/source/tutorial-template.md)
+
+After creating a tutorial, make sure to add the corresponding path in the
+[index.rst](./source/index.rst) file in the following places:
+
+- In the
+  [tutorials torctree](https://github.com/pytorch/executorch/blob/main/docs/source/index.rst?plain=1#L183)
+- In the
+  [customcard section](https://github.com/pytorch/executorch/blob/main/docs/source/index.rst?plain=1#L201)
+
+If you want to include a Markdown tutorial that is stored in another directory
+outside of the `docs/source` directory, complete the following steps:
+
+1. Create an `.md` file under `source/tutorials_source`. Name that file after
+   your tutorial.
+2. Include the following in that file:
+
+   ````
+   ---
+   orphan: true
+   ---
+   ```{include} ../path-to-your-file/outside-of-the-docs-dir.md```
+   ````
+
+   **NOTE:** Your tutorial source file needs to follow the tutorial template.
+
+3. Add the file that you have created in **Step 1** to the `index.rst` toctree
+   and add a `customcarditem` with the link to that file.
+
+For example, if I wanted to include the `README.md` file from
+`examples/selective_build` as a tutorial under
+`pytorch.org/executorch/tutorials`, I could create a file called
+`tutorials_source/selective-build-tutorial.md` and add the following to that
+file:
+
+````
+---
+orphan: true
+---
+
+```{include} ../../../examples/selective_build/README.md
+````
+
+In the `index.rst` file, I would add `tutorials/selective-build-tutorial` in
+both the `toctree` and the `cusotmcarditem` sections.
+
 # Auto-generated API documentation
 
 On a high level (will go into detail later), we use Sphinx to generate both
 Python and C++ documentation in the form of HTML pages.
 
 ### Python APIs
 
-We generate Python API documentation through Sphinx, bootstrapping [PyTorch's
-Sphinx theme](https://github.com/pytorch/pytorch_sphinx_theme) for
-a cohesive look with the existing PyTorch API documentation.
+We generate Python API documentation through Sphinx, bootstrapping
+[PyTorch's Sphinx theme](https://github.com/pytorch/pytorch_sphinx_theme) for a
+cohesive look with the existing PyTorch API documentation.
 
 The setup for Python documentation lies within `source_py/`. To set up Sphinx, a
 `conf.py` configuration file is required. We can specify ways to generate
 documentation here. Specifically, the most important/relevant parts are:
 
-* Make sure to add a path to the directory above the directory you're trying to generate documentation for. For example, since we want to generate documentation for the `executorch/` directory. This tells Sphinx where to find the code to generate docs for.
+- Make sure to add a path to the directory above the directory you're trying to
+  generate documentation for. For example, since we want to generate
+  documentation for the `executorch/` directory. This tells Sphinx where to find
+  the code to generate docs for.
 
-* `extensions` contains extension modules. For auto-generating APIs, make sure to include `sphinx.ext.autodoc`.
-* `autodoc_mock_imports` is where you put imports that Sphinx is unable to access. Sphinx runs your code in order to autogenerate the docs, so for any libraries that it unable to access due to it being outside of the directory, or containing c++ bindings, we need to specify it in the `autodoc_mock_imports` list. You can see what modules Sphinx is confused by when you run into importing errors when generating docs.
+- `extensions` contains extension modules. For auto-generating APIs, make sure
+  to include `sphinx.ext.autodoc`.
+- `autodoc_mock_imports` is where you put imports that Sphinx is unable to
+  access. Sphinx runs your code in order to autogenerate the docs, so for any
+  libraries that it unable to access due to it being outside of the directory,
+  or containing c++ bindings, we need to specify it in the
+  `autodoc_mock_imports` list. You can see what modules Sphinx is confused by
+  when you run into importing errors when generating docs.
 
 Additionally, RST files are needed in order to specify the structure of the
 auto-generated pages and to tell Sphinx what modules to generate documentation
@@ -177,22 +236,21 @@ executorch.exir
    :show-inheritance:
 ```
 
-These separate RST files should all be linked together, with the initial
-landing page under `index.rst`. A sample of this structure can be found in
-`source_py/`.
+These separate RST files should all be linked together, with the initial landing
+page under `index.rst`. A sample of this structure can be found in `source_py/`.
 
-A diagram of how the files work together:
-![image](python_docs.png)
+A diagram of how the files work together: ![image](python_docs.png)
 
 To view your changes on the ExecuTorch website, you can follow the same steps
 listed in the "General Documentation" section.
 
 To view just the auto-generated pages:
+
 1. `cd executorch/docs/`
-2. `sphinx-build -M html source_py sphinxbuild_py`
-to build Sphinx and generate APIs any packages.
-3. `python3 -m http.server 8000 --directory sphinxbuild_py/html` to view your HTML
-files at `localhost:8000`.
+2. `sphinx-build -M html source_py sphinxbuild_py` to build Sphinx and generate
+   APIs any packages.
+3. `python3 -m http.server 8000 --directory sphinxbuild_py/html` to view your
+   HTML files at `localhost:8000`.
 
 ### C++ APIs
 
@@ -212,22 +270,21 @@ To configure Doxygen, we can run `doxygen -g` in the root of our repository (ex.
 generating c++ documentation. Specifically, the most important/relevant parts
 are:
 
-* `OUTPUT_DIRECTORY` specifies where to output the auto-generated XML files
-* `INPUT` specifies which files to generate documenation for
-* `GENERATE_XML = YES`
+- `OUTPUT_DIRECTORY` specifies where to output the auto-generated XML files
+- `INPUT` specifies which files to generate documenation for
+- `GENERATE_XML = YES`
 
 Following PyTorch's `conf.py`
 [file](https://github.com/pytorch/pytorch/blob/master/docs/cpp/source/conf.py),
 we can set up our own `conf.py` file to take in the directory in which we
 generated the XML files, and output HTML to another directory. A sample of this
 structure can be found in `source_cpp/`.
 
-A diagram of how the files work together:
-![image](cpp_docs.png)
+A diagram of how the files work together: ![image](cpp_docs.png)
 
 To view the C++ API documentation locally, run:
 
 1. `cd executorch/docs/`
 2. `sphinx-build -M html source_cpp sphinxbuild_cpp`
-3. `python3 -m http.server 8000 --directory sphinxbuild_py/html` to view your HTML
-files at `localhost:8000`.
+3. `python3 -m http.server 8000 --directory sphinxbuild_py/html` to view your
+   HTML files at `localhost:8000`.
diff --git a/docs/source/_templates/layout.html b/docs/source/_templates/layout.html
@@ -54,8 +54,6 @@
 </script>
 
 {{ super() }}
-// Patch up links to point to ExecuTorch URLs instead of to
-// pytorch/pytorch URLs.
 <script type="text/javascript">
   $(document).ready(function () {
     // Patch links on interactive tutorial pages to point
diff --git a/docs/source/build-run-xtensa.md b/docs/source/build-run-xtensa.md
@@ -41,7 +41,7 @@ On top of being able to run on the Xtensa HiFi4 DSP another goal of this tutoria
 
 ::::{grid} 2
 :::{grid-item-card}  What you will learn in this tutorial:
-:class-card: card-learn
+:class-card: card-prerequisites
 * In this tutorial you will learn how to export a quantized model with linear and batch norm ops targeted for the Xtensa HiFi4 DSP.
 * You will also learn how to compile and deploy the ExecuTorch runtime with the kernels required for running the quantized model generated in the previous step on the Xtensa HiFi4 DSP.
 :::
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -18,6 +18,8 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
+import distutils.file_util
+import glob
 import os
 import sys
 
@@ -93,6 +95,22 @@
     "first_notebook_cell": ("%matplotlib inline"),
 }
 
+assert len(sphinx_gallery_conf["examples_dirs"]) == len(
+    sphinx_gallery_conf["gallery_dirs"]
+), "Lengths of galery_dirs and examples_dir must be same."
+
+for i in range(len(sphinx_gallery_conf["examples_dirs"])):
+    gallery_dir = sphinx_gallery_conf["gallery_dirs"][i]
+    source_dir = sphinx_gallery_conf["examples_dirs"][i]
+
+    # Create gallery dirs if it doesn't exist
+    os.makedirs(gallery_dir, exist_ok=True)
+
+    # Copy .md files from source dir to gallery dir
+    for f in glob.glob(os.path.join(source_dir, "*.md")):
+
+        distutils.file_util.copy_file(f, gallery_dir, update=True)
+
 source_suffix = [".rst", ".md"]
 
 
diff --git a/docs/source/executorch_custom_versions.py b/docs/source/executorch_custom_versions.py
@@ -62,11 +62,17 @@ def replace_variables(app, doctree, docname):
         # CSS classes. Otherwise, the sphinx-gallery generated outputs are
         # formatted as regular code blocks with gray background instead of pink.
         is_sphinx_gallery = any("sphx-glr" in class_ for class_ in classes)
+
+        language = node.get("language")
+
         if is_sphinx_gallery:
             new_literal_block = nodes.literal_block(new_text, new_text, classes=classes)
         else:
             new_literal_block = nodes.literal_block(
-                new_text, new_text, classes=["highlight-none", "notranslate"]
+                new_text,
+                new_text,
+                classes=["highlight-none", "notranslate"],
+                language=language,
             )
 
         node.parent.replace(node, new_literal_block)
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -226,5 +226,12 @@ ExecuTorch tutorials.
    :image: _static/img/android_app.png
    :link: demo-apps-android.html
    :tags: Delegation,Android
+   
+.. customcarditem::
+   :header: Building and Running ExecuTorch on Xtensa HiFi4 DSP
+   :card_description: A tutorial that walks you through the process of building ExecuTorch for an Xtensa Hifi4 DSP.
+   :image: _static/img/generic-pytorch-logo.png
+   :link: build-run-xtensa.html
+   :tags: DSP
 
 .. customcardend::
