You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There are a set of configurations that would influence the performance of PyTorch inference running on Intel(R) Xeon(R) Scalable Processors.
4
+
There are several configuration options that can impact the performance of PyTorch inference when executed on Intel(R) Xeon(R) Scalable Processors.
5
5
To get peak performance, the ``torch.backends.xeon.run_cpu`` script is provided that optimizes the configuration of thread and memory management.
6
6
For thread management, the script configures thread affinity and the preload of Intel(R) OMP library.
7
-
For memory management, it configures NUMA binding and preloads optimized memory allocation libraries (e.g. TCMalloc, JeMalloc).
7
+
For memory management, it configures NUMA binding and preloads optimized memory allocation libraries, such as TCMalloc and JeMalloc.
8
8
In addition, the script provides tunable parameters for compute resource allocation in both single instance and multiple instance scenarios,
9
9
helping the users try out an optimal coordination of resource utilization for the specific workloads.
10
10
11
-
Prerequisites
12
-
-------------
11
+
What you will learn
12
+
-------------------
13
13
14
-
NUMA Access Control
15
-
~~~~~~~~~~~~~~~~~~~
14
+
* How to utilize tools like ``numactl``, ``taskset``, Intel(R) OpenMP Runtime Library and optimized memory allocators such as TCMalloc and JeMalloc for enhanced performance.
15
+
* How to configure CPU cores and memory management to maximize PyTorch inference performance on Intel(R) Xeon(R) processors.
16
16
17
-
It is a good thing that more and more CPU cores are provided to users in one socket, as it brings in more computation resources.
18
-
However, this also brings memory access competitions. Program can stall because memory is busy to visit.
17
+
Introduction for Optimizations
18
+
------------------------------
19
+
20
+
Applying NUMA Access Control
21
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
22
+
23
+
It is beneficial that an increasing number of CPU cores are being provided to users within a single socket, as this offers greater computational resources.
24
+
However, this also leads to competition for memory access, which can cause programs to stall due to busy memory.
19
25
To address this problem, Non-Uniform Memory Access (NUMA) was introduced.
20
-
Comparing to Uniform Memory Access (UMA), in which scenario all the memories are connected to all cores equally,
21
-
NUMA tells memories into multiple groups. Certain number of memories are directly attached to one socket's integrated memory controller to become local memory of this socket.
26
+
Unlike Uniform Memory Access (UMA), where all memories are equally accessible to all cores,
27
+
NUMA organizes memory into multiple groups. Certain number of memories are directly attached to one socket's integrated memory controller to become local memory of this socket.
22
28
Local memory access is much faster than remote memory access.
23
29
24
-
Users can get CPU information with ``lscpu`` command on Linux to learn how many cores, sockets are there on the machine.
25
-
Also, NUMA information like how CPU cores are distributed can also be retrieved.
26
-
The following is an example of ``lscpu`` execution on a machine with Intel (R) Xeon (R) CPU Max 9480.
27
-
2 sockets were detected. Each socket has 56 physical cores onboard. Since Hyper-Threading is enabled, each core can run 2 threads.
28
-
i.e. each socket has another 56 logical cores. Thus, there are 224 CPU cores on service.
29
-
When indexing CPU cores, usually physical cores are indexed before logical core.
30
-
In this case, the first 56 cores (0-55) are physical cores on the first NUMA socket (node), the second 56 cores (56-111) are physical cores on the second NUMA socket (node).
31
-
Logical cores are indexed afterward. 112-167 are 56 logical cores on the first NUMA socket,
32
-
168-223 are the second 56 logical cores on the second NUMA socket.
33
-
Typically, running PyTorch programs with compute intense workloads should avoid using logical cores to get good performance.
30
+
Users can get CPU information with ``lscpu`` command on Linux to learn how many cores and sockets are there on the machine.
31
+
Additionally, this command provides NUMA information, such as the distribution of CPU cores.
32
+
Below is an example of executing ``lscpu`` on a machine equipped with an Intel(R) Xeon(R) CPU Max 9480:
34
33
35
34
.. code-block:: console
36
35
@@ -52,46 +51,48 @@ Typically, running PyTorch programs with compute intense workloads should avoid
52
51
NUMA node1 CPU(s): 56-111,168-223
53
52
...
54
53
55
-
Linux provides a tool, ``numactl``, that allows user control of NUMA policy for processes or shared memory.
54
+
* Two sockets were detected, each containing 56 physical cores. With Hyper-Threading enabled, each core can handle 2 threads, resulting in 56 logical cores per socket. Therefore, the machine has a total of 224 CPU cores in service.
55
+
* Typically, physical cores are indexed before logical cores. In this scenario, cores 0-55 are the physical cores on the first NUMA node, and cores 56-111 are the physical cores on the second NUMA node.
56
+
* Logical cores are indexed subsequently: cores 112-167 correspond to the logical cores on the first NUMA node, and cores 168-223 to those on the second NUMA node.
57
+
58
+
Typically, running PyTorch programs with compute intense workloads should avoid using logical cores to get good performance.
59
+
60
+
Linux provides a tool called ``numactl`` that allows user control of NUMA policy for processes or shared memory.
56
61
It runs processes with a specific NUMA scheduling or memory placement policy.
57
62
As described above, cores share high-speed cache in one socket, thus it is a good idea to avoid cross socket computations.
58
63
From a memory access perspective, bounding memory access locally is much faster than accessing remote memories.
59
-
``numactl`` command should have been installed in recent Linux distributions. In case it is missing, we can install it manually with the installation command, like
64
+
``numactl`` command should have been installed in recent Linux distributions. In case it is missing, you can install it manually with the installation command, like on Ubuntu:
60
65
61
66
.. code-block:: console
62
67
63
68
$ apt-get install numactl
64
69
65
-
on Ubuntu, or
70
+
on CentOS you can run the following command:
66
71
67
72
.. code-block:: console
68
73
69
74
$ yum install numactl
70
75
71
-
on CentOS.
72
-
73
76
The ``taskset`` command in Linux is another powerful utility that allows you to set or retrieve the CPU affinity of a running process.
74
-
``taskset`` are pre-installed in most Linux distributions and in case it's not, we can install it with command
77
+
``taskset`` are pre-installed in most Linux distributions and in case it's not, on Ubuntu you can install it with the command:
75
78
76
79
.. code-block:: console
77
80
78
81
$ apt-get install util-linux
79
82
80
-
on Ubuntu, or
83
+
on CentOS you can run the following command:
81
84
82
85
.. code-block:: console
83
86
84
87
$ yum install util-linux
85
88
86
-
on CentOS.
87
-
88
-
OpenMP
89
-
~~~~~~
89
+
Using Intel(R) OpenMP Runtime Library
90
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
90
91
91
92
OpenMP is an implementation of multithreading, a method of parallelizing where a primary thread (a series of instructions executed consecutively) forks a specified number of sub-threads and the system divides a task among them. The threads then run concurrently, with the runtime environment allocating threads to different processors.
92
93
Users can control OpenMP behaviors with some environment variable settings to fit for their workloads, the settings are read and executed by OMP libraries. By default, PyTorch uses GNU OpenMP Library (GNU libgomp) for parallel computation. On Intel(R) platforms, Intel(R) OpenMP Runtime Library (libiomp) provides OpenMP API specification support. It usually brings more performance benefits compared to libgomp.
93
94
94
-
The Intel(R) OpenMP Runtime Library can be installed via the command
95
+
The Intel(R) OpenMP Runtime Library can be installed using one of these commands:
95
96
96
97
.. code-block:: console
97
98
@@ -103,74 +104,69 @@ or
103
104
104
105
$ conda install mkl
105
106
106
-
Memory Allocator
107
-
~~~~~~~~~~~~~~~~
107
+
Choosing an Optimized Memory Allocator
108
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
108
109
109
110
Memory allocator plays an important role from performance perspective as well. A more efficient memory usage reduces overhead on unnecessary memory allocations or destructions, and thus results in a faster execution. From practical experiences, for deep learning workloads, JeMalloc or TCMalloc can get better performance by reusing memory as much as possible than default malloc function.
110
111
111
-
TCMalloc can be installed by
112
+
You can install TCMalloc by running the following command on Ubuntu:
112
113
113
114
.. code-block:: console
114
115
115
116
$ apt-get install google-perftools
116
117
117
-
on Ubuntu, or
118
+
On CentOS, you can install it by running:
118
119
119
120
.. code-block:: console
120
121
121
122
$ yum install gperftools
122
123
123
-
on CentOS.
124
-
125
-
In conda environment, it can also be installed by
124
+
In a conda environment, it can also be installed by running:
126
125
127
126
.. code-block:: console
128
127
129
128
$ conda install conda-forge::gperftools
130
129
131
-
JeMalloc can be installed by
130
+
On Ubuntu JeMalloc can be installed by this command:
132
131
133
132
.. code-block:: console
134
133
135
134
$ apt-get install libjemalloc2
136
135
137
-
on Ubuntu, or
136
+
On CentOS it can be installed by running:
138
137
139
138
.. code-block:: console
140
139
141
140
$ yum install jemalloc
142
141
143
-
on CentOS, or
142
+
In a conda environment, it can also be installed by running:
144
143
145
144
.. code-block:: console
146
145
147
146
$ conda install conda-forge::jemalloc
148
147
149
-
in conda environment.
150
-
151
-
152
148
Quick Start Example Commands
153
149
----------------------------
154
150
155
-
1. To run single-instance inference with 1 thread on 1 CPU core (only Core #0 would be used)
151
+
1. To run single-instance inference with 1 thread on 1 CPU core (only Core #0 would be used):
4. To run inference in throughput mode, in which all the cores in each CPU node set up an instance
169
+
4. To run inference in throughput mode, in which all the cores in each CPU node set up an instance:
174
170
175
171
.. code-block:: console
176
172
@@ -180,18 +176,17 @@ Quick Start Example Commands
180
176
181
177
Term "instance" here doesn't refer to a cloud instance. This script is executed as a single process which invokes multiple "instances" which are formed from multiple threads. "Instance" is kind of group of threads in this context.
182
178
183
-
Usage of torch.backends.xeon.run_cpu
184
-
------------------------------------
179
+
Using ``torch.backends.xeon.run_cpu``
180
+
-------------------------------------
185
181
186
-
The argument list and usage guidance can be shown with
182
+
The argument list and usage guidance can be shown with the following command:
Please note that the script respects environment variables set preliminarily. i.e. If you have set the environment variables mentioned above before running the script, the values of the variables will not overwritten by the script.
286
+
Please note that the script respects environment variables set preliminarily. For example, if you have set the environment variables mentioned above before running the script, the values of the variables will not be overwritten by the script.
287
+
288
+
Conclusion
289
+
----------
290
+
291
+
In this tutorial, we explored a variety of advanced configurations and tools designed to optimize PyTorch inference performance on Intel(R) Xeon(R) Scalable Processors.
292
+
By leveraging the ``torch.backends.xeon.run_cpu script``, we demonstrated how to fine-tune thread and memory management to achieve peak performance.
293
+
We covered essential concepts such as NUMA access control, optimized memory allocators like TCMalloc and JeMalloc, and the use of Intel(R) OpenMP for efficient multithreading.
294
+
295
+
Additionally, we provided practical command-line examples to guide you through setting up single and multiple instance scenarios, ensuring optimal resource utilization tailored to specific workloads.
296
+
By understanding and applying these techniques, users can significantly enhance the efficiency and speed of their PyTorch applications on Intel(R) Xeon(R) platforms.
0 commit comments