Skip to content

Commit 1c051a6

Browse files
narendasannarendasan
committed
Initial open sourcing of the library
- C++ API is functional and all current tests pass Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
0 parents  commit 1c051a6

File tree

114 files changed

+7065
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

114 files changed

+7065
-0
lines changed

‎.bazelrc

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
##########################################################################
2+
# Copyright (c) 2018-2019 NVIDIA Corporation. All rights reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
# File: DL4AGX/.bazelrc
17+
# Description: Default bazel settings and toolchain configuration
18+
##########################################################################
19+
20+
# +------------------------------------------------------------+
21+
# | Build Configurations |
22+
# +------------------------------------------------------------+
23+
# Enable colorful output of GCC
24+
build --cxxopt="-fdiagnostics-color=always"
25+
build --cxxopt='-std=c++14'
26+
#build --linkopt="-Wl,--no-as-needed"

‎.gitignore

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
bazel
2+
bazel-bazel-test
3+
bazel-bin
4+
bazel-genfiles
5+
bazel-out
6+
bazel-testlogs
7+
bazel-TRTorch
8+
third_party/pytorch
9+
*.jit
10+
*.jit.pt
11+
.\#*
12+
experiments/
13+
py/build/
14+
py/tmp/
15+
py/.eggs
16+

‎BUILD

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
load("@rules_pkg//:pkg.bzl", "pkg_tar")
2+
3+
pkg_tar(
4+
name = "include_core",
5+
package_dir = "include/trtorch",
6+
deps = [
7+
"//core:include",
8+
"//core/conversion:include",
9+
"//core/conversion/conversionctx:include",
10+
"//core/conversion/converters:include",
11+
"//core/conversion/evaluators:include",
12+
"//core/execution:include",
13+
"//core/lowering:include",
14+
"//core/lowering/irfusers:include",
15+
"//core/util:include",
16+
"//core/util/logging:include"
17+
],
18+
)
19+
20+
pkg_tar(
21+
name = "include",
22+
package_dir = "include/trtorch/",
23+
srcs = [
24+
"//cpp/api:api_headers",
25+
],
26+
)
27+
28+
pkg_tar(
29+
name = "lib",
30+
package_dir = "lib/",
31+
srcs = [
32+
"//cpp/api/lib:libtrtorch.so",
33+
],
34+
mode = "0755",
35+
)
36+
37+
38+
39+
40+
pkg_tar(
41+
name = "libtrtorch",
42+
extension = "tar.gz",
43+
package_dir = "trtorch",
44+
srcs = [
45+
"//:LICENSE"
46+
],
47+
deps = [
48+
":lib",
49+
":include",
50+
":include_core",
51+
],
52+
)

‎LICENSE

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
Copyright (c) 2020-present, NVIDIA CORPORATION. All rights reserved.
2+
3+
Redistribution and use in source and binary forms, with or without
4+
modification, are permitted provided that the following conditions
5+
are met:
6+
7+
1. Redistributions of source code must retain the above copyright
8+
notice, this list of conditions and the following disclaimer.
9+
10+
2. Redistributions in binary form must reproduce the above copyright
11+
notice, this list of conditions and the following disclaimer in the
12+
documentation and/or other materials provided with the distribution.
13+
14+
3. Neither the name of NVIDIA CORPORATION nor the names of its
15+
contributors may be used to endorse or promote products derived
16+
from this software without specific prior written permission.
17+
18+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
19+
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
20+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
21+
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
22+
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
23+
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
24+
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
25+
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
26+
OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
27+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
28+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎README.md

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# TRTorch
2+
3+
> Ahead of Time (AOT) compiling for PyTorch JIT
4+
5+
## Compiling TRTorch
6+
7+
### Dependencies
8+
9+
- Libtorch 1.4.0
10+
- CUDA 10.1
11+
- cuDNN 7.6
12+
- TensorRT 6.0.1.5
13+
14+
Install TensorRT, CUDA and cuDNN on the system before starting to compile.
15+
16+
17+
``` shell
18+
bazel build //:libtrtorch --cxxopt="-DNDEBUG"
19+
```
20+
21+
### Debug build
22+
``` shell
23+
bazel build //:libtrtorch --compilation_mode=dbg
24+
```
25+
26+
A tarball with the include files and library can then be found in bazel-bin
27+
28+
### Running TRTorch on a JIT Graph
29+
30+
> Make sure to add LibTorch's version of CUDA 10.1 to your LD_LIBRARY_PATH `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/bazel-TRTorch/external/libtorch/lib`
31+
32+
33+
``` shell
34+
bazel run //cpp/trtorchexec -- $(realpath <PATH TO GRAPH>) <input-size>
35+
```
36+
37+
## How do I add support for a new op...
38+
39+
### In TRTorch?
40+
41+
Thanks for wanting to contribute! There are two main ways to handle supporting a new op. Either you can write a converter for the op from scratch and register it in the NodeConverterRegistry or if you can map the op to a set of ops that already have converters you can write a graph rewrite pass which will replace your new op with an equivalent subgraph of supported ops. Its preferred to use graph rewriting because then we do not need to maintain a large library of op converters.
42+
43+
### In my application?
44+
45+
> The Node Converter Registry is not exposed currently in the public API but you can try using internal headers.
46+
47+
You can register a converter for your op using the NodeConverterRegistry inside your application.
48+
49+
## Structure of the repo
50+
51+
| Component | Description |
52+
| ------------- | ------------------------------------------------------------ |
53+
| [**core**]() | Main JIT ingest, lowering, conversion and execution implementations |
54+
| [**cpp**]() | C++ API for TRTorch |
55+
| [**tests**]() | Unit test for TRTorch |
56+
57+
## License
58+
59+
The TRTorch license can be found in the LICENSE file.

‎WORKSPACE

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
workspace(name = "TRTorch")
2+
3+
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
4+
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
5+
6+
7+
http_archive(
8+
name = "rules_python",
9+
url = "https://github.com/bazelbuild/rules_python/releases/download/0.0.1/rules_python-0.0.1.tar.gz",
10+
sha256 = "aa96a691d3a8177f3215b14b0edc9641787abaaa30363a080165d06ab65e1161",
11+
)
12+
13+
load("@rules_python//python:repositories.bzl", "py_repositories")
14+
py_repositories()
15+
# Only needed if using the packaging rules.
16+
load("@rules_python//python:pip.bzl", "pip_repositories", "pip_import")
17+
pip_repositories()
18+
19+
http_archive(
20+
name = "libtorch",
21+
build_file = "@//third_party/libtorch:BUILD",
22+
strip_prefix = "libtorch",
23+
urls = ["https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-shared-with-deps-1.4.0.zip"],
24+
sha256 = "f214bfde532877aa5d4e0803e51a28fa8edd97b6a44b6615f75a70352b6b542e"
25+
)
26+
27+
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
28+
http_archive(
29+
name = "rules_pkg",
30+
url = "https://github.com/bazelbuild/rules_pkg/releases/download/0.2.4/rules_pkg-0.2.4.tar.gz",
31+
sha256 = "4ba8f4ab0ff85f2484287ab06c0d871dcb31cc54d439457d28fd4ae14b18450a",
32+
)
33+
34+
load("@rules_pkg//:deps.bzl", "rules_pkg_dependencies")
35+
rules_pkg_dependencies()
36+
37+
new_local_repository(
38+
name = "cuda",
39+
path = "/usr/local/cuda-10.1/targets/x86_64-linux/",
40+
build_file = "@//third_party/cuda:BUILD",
41+
)
42+
43+
new_local_repository(
44+
name = "cudnn",
45+
path = "/usr/",
46+
build_file = "@//third_party/cudnn:BUILD"
47+
)
48+
49+
new_local_repository(
50+
name = "tensorrt",
51+
path = "/usr/",
52+
build_file = "@//third_party/tensorrt:BUILD"
53+
)
54+
55+
git_repository(
56+
name = "googletest",
57+
remote = "https://github.com/google/googletest",
58+
commit = "703bd9caab50b139428cea1aaff9974ebee5742e",
59+
shallow_since = "1570114335 -0400"
60+
)

‎core/BUILD

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
package(default_visibility = ["//visibility:public"])
2+
3+
cc_library(
4+
name = "core",
5+
hdrs = [
6+
"compiler.h",
7+
],
8+
srcs = [
9+
"compiler.cpp",
10+
],
11+
deps = [
12+
"//core/conversion",
13+
"//core/execution",
14+
"//core/lowering",
15+
"//core/util/logging",
16+
"@libtorch//:libtorch",
17+
"@tensorrt//:nvinfer"
18+
]
19+
)
20+
21+
22+
load("@rules_pkg//:pkg.bzl", "pkg_tar")
23+
24+
pkg_tar(
25+
name = "include",
26+
package_dir = "core/",
27+
srcs = ["compiler.h"],
28+
)

‎core/README.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# TRTorch Core
2+
The TRTorch Core is the main graph analysis library, it processes a TorchScript Module, converting method graphs to engines and returning a new equivalent module which when run will run inputs through a TensorRT engine
3+
4+
## Stages
5+
6+
> Basic rule of thumb for organization, if the the output of the component is a modified block then it is in lowering, if the output is a TRT engine block then its in conversion
7+
8+
## Lowering Passes
9+
10+
There are a set of passes over the IR that will be made to lower the graph into a block of convertible nodes.
11+
12+
### PyTorch JIT Lowering Passes
13+
Firstly the graph will go through the lowering passes used in LibTorch, this will lower it to a graph where all attributes accesses are replaced with explicit inputs to the graph (i.e. graph parameters vs. prim::GetAttr)
14+
15+
#### Call Method Insertions
16+
17+
Graphs from prim::CallMethods need to be inserted into the graph or used to segment the graph into convertible subgraphs.
18+
19+
### TRTorch Lowering
20+
21+
To simplify conversion we can use the PyTorch JIT Subgraph Rewriter to simplify the set of subgraphs that need explicit TensorRT converters. This means we could aim for closer to 1->1 op conversion vs looking for applicable subgraphs, limit the number of converters and reduce the size of each converter.
22+
23+
24+
## Conversion Phase
25+
26+
Once the graph has be simplified to a form thats easy to convert, we then set up a conversion context to manage the construction of a TensorRT INetworkDefinition from the blocks nodes. The conversion context records the set of converted nodes, block inputs and outputs and other information about the conversion of the graph. This data is then used to help converters link together layers and also hold build time information like weights required to construct the engine. After the context is created, the block converter starts iterating through the list of nodes, for each node, the converter will look at its inputs and assemble a dictionary of resources to pass to the converter. Inputs can be in a couple of states:
27+
- The input is a block parameter
28+
In this case the input should have already been stored in as an IValue in the conversion context evaluated_value_map. The conversion stage will add the IValue to the list of args for the converter
29+
- The input is an output of a node that has already been converted
30+
In this case the ITensor of the output has added to the to the value_tensor_map, The conversion stage will add the ITensor to the list of args for the converter
31+
- The input is from a node that produces a static value
32+
There are nodes that produce static values, typically used to store parameters for operators, we need to evaluate these nodes at conversion time to be able to convert a op. The converter will look for a node evaluator in the evaluator registry and run it on the node. The IValue produced will be entered in the conversion context evaluated_value_map and added to the list of args for the converter.
33+
- The input is from a node that has not been converted
34+
TRTorch will error here
35+
36+
### Node Evaluation
37+
38+
There are some nodes that contain static data and are resources for operations. These can be evaluated at conversion time so that you can use those values when doing node conversion. In theory any node kind can have a conversion time evaluator as long as it produces a static IValue, This IValue will be stored in the conversion context so it can be consumed by any node that takes the evaluated node as an input.
39+
40+
### Converters
41+
42+
See the README in //core/conversion/converters for more information

0 commit comments

Comments
 (0)