chore: rebase to master

Signed-off-by: Dheeraj Peri <[email protected]>

Author: peri044

BUILD

@@ -18,9 +18,12 @@ pkg_tar(
         "//core/conversion/var:include",
         "//core/conversion/tensorcontainer:include",
         "//core/conversion/evaluators:include",
-        "//core/plugins:include",
+        "//core/ir:include",
         "//core/lowering:include",
         "//core/lowering/passes:include",
+        "//core/partitioning:include",
+        "//core/plugins:impl_include",
+        "//core/plugins:include",
         "//core/runtime:include",
         "//core/util:include",
         "//core/util/logging:include",

CHANGELOG.md

@@ -614,4 +614,80 @@ Signed-off-by: Naren Dasan <[email protected]>
 Signed-off-by: Naren Dasan <[email protected]>
 
 
+# 0.3.0 (2021-05-13)
+
+
+### Bug Fixes
+
+* **//plugins:** Readding cuBLAS BUILD to allow linking of libnvinfer_plugin on Jetson ([a8008f4](https://github.com/NVIDIA/TRTorch/commit/a8008f4))
+* **//tests/../concat:** Concat test fix ([2432fb8](https://github.com/NVIDIA/TRTorch/commit/2432fb8))
+* **//tests/core/partitioning:** Fixing some issues with the partition ([ff89059](https://github.com/NVIDIA/TRTorch/commit/ff89059))
+* erase the repetitive nodes in dependency analysis ([80b1038](https://github.com/NVIDIA/TRTorch/commit/80b1038))
+* fix a typo for debug ([c823ebd](https://github.com/NVIDIA/TRTorch/commit/c823ebd))
+* fix typo bug ([e491bb5](https://github.com/NVIDIA/TRTorch/commit/e491bb5))
+* **aten::linear:** Fixes new issues in 1.8 that cause script based ([c5057f8](https://github.com/NVIDIA/TRTorch/commit/c5057f8))
+* register the torch_fallback attribute in Python API ([8b7919f](https://github.com/NVIDIA/TRTorch/commit/8b7919f))
+* support expand/repeat with IValue type input ([a4882c6](https://github.com/NVIDIA/TRTorch/commit/a4882c6))
+* support shape inference for add_, support non-tensor arguments for segmented graphs ([46950bb](https://github.com/NVIDIA/TRTorch/commit/46950bb))
+
+
+* feat!: Updating versions of CUDA, cuDNN, TensorRT and PyTorch ([71c4dcb](https://github.com/NVIDIA/TRTorch/commit/71c4dcb))
+* feat(WORKSPACE)!: Updating PyTorch version to 1.8.1 ([c9aa99a](https://github.com/NVIDIA/TRTorch/commit/c9aa99a))
+
+
+### Features
+
+* **//.github:** Linter throws 1 when there needs to be style changes to ([a39dea7](https://github.com/NVIDIA/TRTorch/commit/a39dea7))
+* **//core:** New API to register arbitrary TRT engines in TorchScript ([3ec836e](https://github.com/NVIDIA/TRTorch/commit/3ec836e))
+* **//core/conversion/conversionctx:** Adding logging for truncated ([96245ee](https://github.com/NVIDIA/TRTorch/commit/96245ee))
+* **//core/partitioing:** Adding ostream for Partition Info ([b3589c5](https://github.com/NVIDIA/TRTorch/commit/b3589c5))
+* **//core/partitioning:** Add an ostream implementation for ([ee536b6](https://github.com/NVIDIA/TRTorch/commit/ee536b6))
+* **//core/partitioning:** Refactor top level partitioning API, fix a bug with ([abc63f6](https://github.com/NVIDIA/TRTorch/commit/abc63f6))
+* **//core/plugins:** Gating plugin logging based on global config ([1d5a088](https://github.com/NVIDIA/TRTorch/commit/1d5a088))
+* added user level API for fallback ([f4c29b4](https://github.com/NVIDIA/TRTorch/commit/f4c29b4))
+* allow users to set fallback block size and ops ([6d3064a](https://github.com/NVIDIA/TRTorch/commit/6d3064a))
+* insert nodes by dependencies for nonTensor inputs/outputs ([4e32eff](https://github.com/NVIDIA/TRTorch/commit/4e32eff))
+* support aten::arange converter ([014e381](https://github.com/NVIDIA/TRTorch/commit/014e381))
+* support aten::transpose with negative dim ([4a1d2f3](https://github.com/NVIDIA/TRTorch/commit/4a1d2f3))
+* support Int/Bool and other constants' inputs/outputs for TensorRT segments ([54e407e](https://github.com/NVIDIA/TRTorch/commit/54e407e))
+* support prim::Param for fallback inputs ([ec2bbf2](https://github.com/NVIDIA/TRTorch/commit/ec2bbf2))
+* support prim::Param for input type after refactor ([3cebe97](https://github.com/NVIDIA/TRTorch/commit/3cebe97))
+* support Python APIs for Automatic Fallback ([100b090](https://github.com/NVIDIA/TRTorch/commit/100b090))
+* support the case when the injected node is not supported in dependency analysis ([c67d8f6](https://github.com/NVIDIA/TRTorch/commit/c67d8f6))
+* support truncate long/double to int/float with option ([740eb54](https://github.com/NVIDIA/TRTorch/commit/740eb54))
+* Try to submit review before exit ([9a9d7f0](https://github.com/NVIDIA/TRTorch/commit/9a9d7f0))
+* update truncate long/double python api ([69e49e8](https://github.com/NVIDIA/TRTorch/commit/69e49e8))
+* **//docker:** Adding Docker 21.03 ([9b326e8](https://github.com/NVIDIA/TRTorch/commit/9b326e8))
+* update truncate long/double warning message ([60dba12](https://github.com/NVIDIA/TRTorch/commit/60dba12))
+* **//docker:** Update CI container ([df63467](https://github.com/NVIDIA/TRTorch/commit/df63467))
+* **//py:** Allowing people using the PyTorch backend to use TRTorch/TRT ([6c3e0ad](https://github.com/NVIDIA/TRTorch/commit/6c3e0ad))
+* **//py:** Catch when bazel is not in path and error out when running ([1da999d](https://github.com/NVIDIA/TRTorch/commit/1da999d))
+* **//py:** Gate partial compilation from to_backend API ([bf1b2d8](https://github.com/NVIDIA/TRTorch/commit/bf1b2d8))
+* **//py:** New API to embed engine in new module ([88d07a9](https://github.com/NVIDIA/TRTorch/commit/88d07a9))
+* **aten::floor:** Adds floor.int evaluator ([a6a46e5](https://github.com/NVIDIA/TRTorch/commit/a6a46e5))
+
+
+### BREAKING CHANGES
+
+* PyTorch version has been bumped to 1.8.0
+Default CUDA version is CUDA 11.1
+TensorRT version is TensorRT 7.2.3.4
+cuDNN version is now cuDNN 8.1
+
+Signed-off-by: Naren Dasan <[email protected]>
+Signed-off-by: Naren Dasan <[email protected]>
+* Due to issues with compatability between PyTorch 1.8.0
+and 1.8.1 in the Torch Python API, TRTorch 0.3.0 compiled for 1.8.0 does not
+work with PyTorch 1.8.1 and will show an error about use_input_stats.
+If you see this error make sure the version of libtorch you are
+compiling with is PyTorch 1.8.1
+
+TRTorch 0.3.0 will target PyTorch 1.8.1. There is no backwards
+compatability with 1.8.0. If you need this specific version compile from
+source with the dependencies in WORKSPACE changed
+
+Signed-off-by: Naren Dasan <[email protected]>
+Signed-off-by: Naren Dasan <[email protected]>
+
+
 

WORKSPACE

@@ -38,6 +38,11 @@ new_local_repository(
     path = "/usr/local/cuda-11.1/",
 )
 
+new_local_repository(
+    name = "cublas",
+    build_file = "@//third_party/cublas:BUILD",
+    path = "/usr",
+)
 #############################################################################################################
 # Tarballs and fetched dependencies (default - use in cases when building from precompiled bin and tarballs)
 #############################################################################################################

core/conversion/converters/impl/activation.cpp

@@ -161,7 +161,7 @@ auto acthardtanh TRTORCH_UNUSED =
                     TRTORCH_CHECK(new_layer, "Unable to create layer for aten::elu");
                     new_layer->setAlpha(alpha);
 
-                    new_layer->setName(trtorch::core::util::node_info(n).c_str());
+                    new_layer->setName(util::node_info(n).c_str());
 
                     auto out_tensor = ctx->AssociateValueAndTensor(n->outputs()[0], new_layer->getOutput(0));
                     LOG_DEBUG("Output shape: " << out_tensor->getDimensions());
@@ -190,7 +190,7 @@ auto acthardtanh TRTORCH_UNUSED =
                     TRTORCH_CHECK(gelu_plugin, "Unable to create gelu plugin from TensorRT plugin registry" << *n);
                     auto new_layer =
                         ctx->net->addPluginV2(reinterpret_cast<nvinfer1::ITensor* const*>(&in), 1, *gelu_plugin);
-                    new_layer->setName("gelu");
+                    new_layer->setName(util::node_info(n).c_str());
                     auto out_tensor = new_layer->getOutput(0);
                     out_tensor = ctx->AssociateValueAndTensor(n->outputs()[0], out_tensor);
                     LOG_DEBUG("Output shape: " << out_tensor->getDimensions());

core/partitioning/README.md

@@ -0,0 +1,67 @@
+# TRTorch Partitioning
+
+TRTorch partitioning phase is developed to support `automatic fallback` feature in TRTorch. This phase won't run by
+default until the automatic fallback feature is enabled.
+
+On a high level, TRTorch partitioning phase does the following:
+- `Segmentation`. Go through the set of operators in order and verify if there is converter for each operator. Then, 
+roughly separate the graph into parts that TRTorch can support and parts TRTorch cannot.
+- `Dependency Analysis`. For every to be compiled operator there is a "complete dependency graph", which means that 
+every input can to traced back to an input as Tensor or TensorList. Go through all segments after segmentation then 
+  do dependency analysis to ensure that there are only Tensor/TensorList inputs and outputs for TensorRT segments.
+- `Shape Analysis`. For each segments, figure out the input and outputs shapes starting from the provided input shape 
+from the user. Shapes can be calculated by running the graphs with JIT.
+- `Conversion`. Every TensorRT segments will be converted to TensorRT engine. This part is done in compiler.cpp, but 
+  it's still a phase in our partitioning process.
+- `Stitching`. Stitch all TensorRT engines with PyTorch nodes altogether. 
+
+Test cases for each of these components could be found [here](https://github.com/NVIDIA/TRTorch/tree/master/tests/core/partitioning).
+
+Here is the brief description of functionalities of each file:
+- `PartitionInfo.h/cpp`: The automatic fallback APIs that is used for partitioning. 
+- `SegmentedBlock.h/cpp`: The main data structures that is used to maintain information for each segments after segmentation.
+- `shape_analysis.h/cpp`: Code implementation to get the shapes for each segments by running them in JIT.
+- `partitioning.h/cpp`: APIs and main code implementation for partitioning phase.
+
+### Automatic Fallback
+To enable automatic fallback feature, you can set following attributes in Python: 
+```python
+  import torch
+  import trtorch
+  
+  ...
+  model = MyModel()
+  ts_model = torch.jit.script(model)
+  trt_model = trtorch.compile(model, {
+    ...
+    "torch_fallback" : {
+      "enabled" : True,
+      "min_block_size" : 3,
+      "forced_fallback_ops": ["aten::add"],
+    } 
+  })
+```
+- `enabled`: By default automatic fallback will be off. It is enabled by setting it to True. 
+- `min_block_size`: The minimum number of consecutive operations that must satisfy to be converted to TensorRT. For 
+example, if it's set to 3, then there must be 3 consecutive supported operators then this segments will be converted.
+- `forced_fallback_ops`: A list of strings that will be the names of operations that the user explicitly want to be in 
+PyTorch nodes.
+
+To enable automatic fallback feature in C++, following APIs could be uses:
+
+```c++
+#include "torch/script.h"
+#include "trtorch/trtorch.h"
+
+...
+auto in = torch::randn({1, 3, 224, 224}, {torch::kCUDA});
+
+auto mod = trtorch::jit::load("trt_ts_module.ts");
+auto input_sizes =  std::vector<trtorch::CompileSpec::InputRange>{{in.sizes()}};
+trtorch::CompileSpec cfg(input_sizes);
+cfg.torch_fallback = trtorch::CompileSpec::TorchFallback(true);
+cfg.torch_fallback.min_block_size = 2;
+cfg.torch_fallback.forced_fallback_ops.push_back("aten::relu");
+auto trt_mod = trtorch::CompileGraph(mod, cfg);
+auto out = trt_mod.forward({in});
+```

core/plugins/BUILD

@@ -42,7 +42,12 @@ load("@rules_pkg//:pkg.bzl", "pkg_tar")
 pkg_tar(
     name = "include",
     package_dir = "core/plugins/",
-    srcs = ["plugins.h",
-            "impl/interpolate_plugin.h",
-            "impl/normalize_plugin.h"],
+    srcs = ["plugins.h"],
 )
+
+pkg_tar(
+    name = "impl_include",
+    package_dir = "core/plugins/impl",
+    srcs = ["impl/interpolate_plugin.h",
+            "impl/normalize_plugin.h"],
+)

core/plugins/register_plugins.cpp

@@ -14,36 +14,37 @@ namespace impl {
 class TRTorchPluginRegistry {
  public:
   TRTorchPluginRegistry() {
-    trtorch_logger.log(util::logging::LogLevel::kINFO, "Instatiated the TRTorch plugin registry class");
     // register libNvInferPlugins and TRTorch plugins
     // trtorch_logger logging level is set to kERROR and reset back to kDEBUG.
     // This is because initLibNvInferPlugins initializes only a subset of plugins and logs them.
     // Plugins outside this subset in TensorRT are not being logged in this. So temporarily we disable this to prevent
     // multiple logging of same plugins. To provide a clear list of all plugins, we iterate through getPluginRegistry()
     // where it prints the list of all the plugins registered in TensorRT with their namespaces.
-    trtorch_logger.set_reportable_log_level(util::logging::LogLevel::kERROR);
-    initLibNvInferPlugins(&trtorch_logger, "");
-    trtorch_logger.set_reportable_log_level(util::logging::LogLevel::kDEBUG);
+    plugin_logger.set_reportable_log_level(util::logging::LogLevel::kERROR);
+    initLibNvInferPlugins(&plugin_logger, "");
+    plugin_logger.set_reportable_log_level(util::logging::get_logger().get_reportable_log_level());
 
     int numCreators = 0;
     auto pluginsList = getPluginRegistry()->getPluginCreatorList(&numCreators);
     for (int k = 0; k < numCreators; ++k) {
       if (!pluginsList[k]) {
-        trtorch_logger.log(util::logging::LogLevel::kDEBUG, "Plugin creator for plugin " + str(k) + " is a nullptr");
+        plugin_logger.log(util::logging::LogLevel::kDEBUG, "Plugin creator for plugin " + str(k) + " is a nullptr");
         continue;
       }
       std::string pluginNamespace = pluginsList[k]->getPluginNamespace();
-      trtorch_logger.log(
+      plugin_logger.log(
           util::logging::LogLevel::kDEBUG,
           "Registered plugin creator - " + std::string(pluginsList[k]->getPluginName()) +
               ", Namespace: " + pluginNamespace);
     }
-    trtorch_logger.log(util::logging::LogLevel::kDEBUG, "Total number of plugins registered: " + str(numCreators));
+    plugin_logger.log(util::logging::LogLevel::kDEBUG, "Total number of plugins registered: " + str(numCreators));
   }
 
  public:
-  util::logging::TRTorchLogger trtorch_logger =
-      util::logging::TRTorchLogger("[TRTorch Plugins Context] - ", util::logging::LogLevel::kDEBUG, true);
+  util::logging::TRTorchLogger plugin_logger = util::logging::TRTorchLogger(
+      "[TRTorch Plugins Context] - ",
+      util::logging::get_logger().get_reportable_log_level(),
+      util::logging::get_logger().get_is_colored_output_on());
 };
 
 namespace {

cpp/api/include/trtorch/macros.h

@@ -20,7 +20,7 @@
 #define STR(x) XSTR(x)
 
 #define TRTORCH_MAJOR_VERSION 0
-#define TRTORCH_MINOR_VERSION 3
+#define TRTORCH_MINOR_VERSION 4
 #define TRTORCH_PATCH_VERSION 0
 #define TRTORCH_VERSION      \
   STR(TRTORCH_MAJOR_VERSION) \

docker/Dockerfile.21.03

@@ -0,0 +1,41 @@
+FROM nvcr.io/nvidia/pytorch:21.03-py3
+
+RUN apt-get update && apt-get install -y curl gnupg  && rm -rf /var/lib/apt/lists/*
+
+RUN curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
+RUN mv bazel.gpg /etc/apt/trusted.gpg.d/
+RUN echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list
+
+RUN apt-get update && apt-get install -y bazel-4.0.0 && rm -rf /var/lib/apt/lists/*
+RUN ln -s /usr/bin/bazel-4.0.0 /usr/bin/bazel
+
+RUN pip install notebook
+
+COPY . /opt/trtorch
+RUN rm /opt/trtorch/WORKSPACE
+COPY ./docker/WORKSPACE.cu.docker  /opt/trtorch/WORKSPACE
+
+# Workaround for bazel expecting both static and shared versions, we only use shared libraries inside container
+RUN cp /usr/lib/x86_64-linux-gnu/libnvinfer.so /usr/lib/x86_64-linux-gnu/libnvinfer_static.a
+
+WORKDIR /opt/trtorch
+RUN bazel build //:libtrtorch --compilation_mode opt
+
+WORKDIR /opt/trtorch/py
+
+RUN pip install ipywidgets --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host=files.pythonhosted.org
+RUN jupyter nbextension enable --py widgetsnbextension
+
+# Locale is not set by default
+RUN apt-get update && apt-get install -y locales ninja-build && rm -rf /var/lib/apt/lists/* && locale-gen en_US.UTF-8
+ENV LANG en_US.UTF-8
+ENV LANGUAGE en_US:en
+ENV LC_ALL en_US.UTF-8
+RUN python3 setup.py install --use-cxx11-abi
+
+RUN conda init bash
+
+ENV LD_LIBRARY_PATH /opt/conda/lib/python3.8/site-packages/torch/lib:$LD_LIBRARY_PATH
+
+WORKDIR /opt/trtorch/
+CMD /bin/bash

docs/_cpp_api/class_view_hierarchy.html

@@ -152,6 +152,9 @@
        <a href="https://nvidia.github.io/TRTorch/" title="master">
         master
        </a>
+       <a href="https://nvidia.github.io/TRTorch/v0.3.0/" title="v0.3.0">
+        v0.3.0
+       </a>
        <a href="https://nvidia.github.io/TRTorch/v0.2.0/" title="v0.2.0">
         v0.2.0
        </a>

docs/_cpp_api/classtrtorch_1_1CompileSpec_1_1DataType.html

@@ -154,6 +154,9 @@
        <a href="https://nvidia.github.io/TRTorch/" title="master">
         master
        </a>
+       <a href="https://nvidia.github.io/TRTorch/v0.3.0/" title="v0.3.0">
+        v0.3.0
+       </a>
        <a href="https://nvidia.github.io/TRTorch/v0.2.0/" title="v0.2.0">
         v0.2.0
        </a>

docs/_cpp_api/classtrtorch_1_1CompileSpec_1_1Device_1_1DeviceType.html

@@ -154,6 +154,9 @@
        <a href="https://nvidia.github.io/TRTorch/" title="master">
         master
        </a>
+       <a href="https://nvidia.github.io/TRTorch/v0.3.0/" title="v0.3.0">
+        v0.3.0
+       </a>
        <a href="https://nvidia.github.io/TRTorch/v0.2.0/" title="v0.2.0">
         v0.2.0
        </a>

docs/_cpp_api/classtrtorch_1_1ptq_1_1Int8CacheCalibrator.html

@@ -154,6 +154,9 @@
        <a href="https://nvidia.github.io/TRTorch/" title="master">
         master
        </a>
+       <a href="https://nvidia.github.io/TRTorch/v0.3.0/" title="v0.3.0">
+        v0.3.0
+       </a>
        <a href="https://nvidia.github.io/TRTorch/v0.2.0/" title="v0.2.0">
         v0.2.0
        </a>

docs/_cpp_api/classtrtorch_1_1ptq_1_1Int8Calibrator.html

@@ -154,6 +154,9 @@
        <a href="https://nvidia.github.io/TRTorch/" title="master">
         master
        </a>
+       <a href="https://nvidia.github.io/TRTorch/v0.3.0/" title="v0.3.0">
+        v0.3.0
+       </a>
        <a href="https://nvidia.github.io/TRTorch/v0.2.0/" title="v0.2.0">
         v0.2.0
        </a>

docs/_cpp_api/define_macros_8h_1a18d295a837ac71add5578860b55e5502.html

@@ -154,6 +154,9 @@
        <a href="https://nvidia.github.io/TRTorch/" title="master">
         master
        </a>
+       <a href="https://nvidia.github.io/TRTorch/v0.3.0/" title="v0.3.0">
+        v0.3.0
+       </a>
        <a href="https://nvidia.github.io/TRTorch/v0.2.0/" title="v0.2.0">
         v0.2.0
        </a>

docs/_cpp_api/define_macros_8h_1a20c1fbeb21757871c52299dc52351b5f.html

@@ -154,6 +154,9 @@
        <a href="https://nvidia.github.io/TRTorch/" title="master">
         master
        </a>
+       <a href="https://nvidia.github.io/TRTorch/v0.3.0/" title="v0.3.0">
+        v0.3.0
+       </a>
        <a href="https://nvidia.github.io/TRTorch/v0.2.0/" title="v0.2.0">
         v0.2.0
        </a>

docs/_cpp_api/define_macros_8h_1a25ee153c325dfc7466a33cbd5c1ff055.html

@@ -154,6 +154,9 @@
        <a href="https://nvidia.github.io/TRTorch/" title="master">
         master
        </a>
+       <a href="https://nvidia.github.io/TRTorch/v0.3.0/" title="v0.3.0">
+        v0.3.0
+       </a>
        <a href="https://nvidia.github.io/TRTorch/v0.2.0/" title="v0.2.0">
         v0.2.0
        </a>

docs/_cpp_api/define_macros_8h_1a48d6029a45583a06848891cb0e86f7ba.html

@@ -154,6 +154,9 @@
        <a href="https://nvidia.github.io/TRTorch/" title="master">
         master
        </a>
+       <a href="https://nvidia.github.io/TRTorch/v0.3.0/" title="v0.3.0">
+        v0.3.0
+       </a>
        <a href="https://nvidia.github.io/TRTorch/v0.2.0/" title="v0.2.0">
         v0.2.0
        </a>