[SYCL] Persistent program cache #3391

vladimirlaz · 2021-03-22T14:36:01Z

First part of implementation for persistent cache of device programs following sycl/doc/KernelProgramCache.md:

Programs built for target device are stored in on-disk cache and used next time when the same application is executed.
What is done:

storing and loading files when built program is not found in in-memory cache;
file system concurrent access management (lock file is created per cache item while data is being written, read operation does not lock the cache item);
thread safety for persistent device code cache is not needed as soon as build/in-memory cache part is managed by program_manager and persisten_device_code_cache does not store data shared between threads.
Implement support for environment variables affecting cache:
SYCL_CACHE_DIR
SYCL_CACHE_TRACE
SYCL_CACHE_DISABLE_PERSISTENT
SYCL_CACHE_MIN_DEVICE_IMAGE_SIZE
SYCL_CACHE_MAX_DEVICE_IMAGE_SIZE
SYCL_CACHE_MIN_DEVICE_IMAGE_SIZE
Errors happening during filesystem operation cause corresponding cache operation silently ends:
- for read operation, cache miss happens (without SYCL error), message is send to std::cerr if SYCL_CACHE_TRACE is set;
- for write operation, no cache item stored (without SYCL error), message is send to std::cerr if SYCL_CACHE_TRACE is set;
Added on-device tests for basic cache functionality and unit tests for concurrent access to FS and error management.

To be done in next patches:

implement rest of environment variables and extend them to both in-memory and persistent cache;
implement kernel eviction mechanism for on-device and in-memory caches.

After 86b0e8d patch extra operations with device images happen before check if it is present in in-memory cache. For applications with small kernels which are executed multiple times noticeable performance degradation is seen for host code. That was done to get build options stored in the kernel image and use them as in-memory cache key. At the same time kernel image (where these options are taken from) is used in cache key so it is reasonable to use only build options which are specified in SYCL API and/or environment variables as separate cache key. Getting build options from kernel image is moved back to build operation which happens only if built program is missed in in-memory cache.

vladimirlaz · 2021-03-22T17:55:00Z

/summary:run

vladimirlaz · 2021-03-23T16:09:37Z

/summary:run

sycl/doc/KernelProgramCache.md

sycl/doc/EnvironmentVariables.md

sycl/include/CL/sycl/detail/os_util.hpp

sycl/source/detail/config.def

sycl/test/on-device/cache/basic.cpp

sycl/test/on-device/cache/basic.hpp

sycl/doc/KernelProgramCache.md

vladimirlaz · 2021-03-25T18:29:56Z

/summary:run

vladimirlaz · 2021-04-05T15:33:01Z

/summary:run

vladimirlaz · 2021-04-06T10:08:19Z

/summary:run

sycl/source/detail/persistent_device_code_cache.cpp

sycl/source/detail/persistent_device_code_cache.hpp

sycl/unittests/kernel-and-program/PersistentDeviceCodeCache.cpp

sycl/source/detail/persistent_device_code_cache.cpp

sycl/source/detail/persistent_device_code_cache.hpp

vladimirlaz · 2021-04-07T19:07:02Z

/summary:run

sycl/source/detail/persistent_device_code_cache.cpp

sycl/source/detail/persistent_device_code_cache.hpp

sycl/unittests/kernel-and-program/PersistentDeviceCodeCache.cpp

sycl/source/detail/program_manager/program_manager.cpp

sycl/source/detail/persistent_device_code_cache.hpp

sycl/source/detail/persistent_device_code_cache.cpp

…istent cache

sycl/doc/EnvironmentVariables.md

Co-authored-by: sergei <[email protected]>

vladimirlaz · 2021-04-08T11:17:29Z

/summary:run

s-kanaev

LGTM

pvchupin · 2021-04-08T20:19:23Z

@kbobrovs to approve

kbobrovs

Thanks Vladimir, LGTM.

bader · 2021-04-11T11:26:00Z

sycl/source/detail/persistent_device_code_cache.cpp

+std::vector<std::vector<char>> PersistentDeviceCodeCache::getItemFromDisc(
+    const device &Device, const RTDeviceBinaryImage &Img,
+    const SerializedObj &SpecConsts, const std::string &BuildOptionsString,
+    RT::PiProgram &NativePrg) {


@vladimirlaz, post-commit build fails with

/home/runner/work/llvm/llvm/src/sycl/source/detail/persistent_device_code_cache.cpp:130:20: error: unused parameter 'NativePrg' [-Werror,-Wunused-parameter] RT::PiProgram &NativePrg) {

Please, fix ASAP.

vladimirlaz added 5 commits March 10, 2021 09:12

Initial implementation

7d98ed8

Raw implementation

4db44a5

Merge remote-tracking branch 'intel_llvm/sycl' into jit_cache_2

b7fa3f6

Fix LIT tests

95f4e99

Fix CI issues

3f1ab43

vladimirlaz force-pushed the jit_cache_2 branch from 8125561 to 3f1ab43 Compare March 23, 2021 07:39

vladimirlaz added 3 commits March 23, 2021 13:09

Enable Windows and brush up code

4d847f3

Fix failure

f776006

Fix issues

414780c

vladimirlaz added 2 commits March 24, 2021 22:36

Move on-disk cache to separate module

d3283f3

revert unnecessary change

5f94856

vladimirlaz marked this pull request as ready for review March 24, 2021 19:39

vladimirlaz requested review from kbobrovs, pvchupin and a team as code owners March 24, 2021 19:39

vladimirlaz requested a review from rbegam March 24, 2021 19:39

pvchupin suggested changes Mar 25, 2021

View reviewed changes

sycl/doc/KernelProgramCache.md Outdated Show resolved Hide resolved

sycl/doc/EnvironmentVariables.md Outdated Show resolved Hide resolved

Apply review comments and fix CI issues

eece33e

s-kanaev reviewed Mar 25, 2021

View reviewed changes

sycl/include/CL/sycl/detail/os_util.hpp Show resolved Hide resolved

sycl/source/detail/config.def Outdated Show resolved Hide resolved

sycl/test/on-device/cache/basic.cpp Outdated Show resolved Hide resolved

sycl/test/on-device/cache/basic.hpp Outdated Show resolved Hide resolved

Apply review comments and remove redundant change

5b26cd6

vladimirlaz requested review from pvchupin and s-kanaev March 25, 2021 13:53

pvchupin reviewed Mar 25, 2021

View reviewed changes

sycl/doc/KernelProgramCache.md Outdated Show resolved Hide resolved

vladimirlaz commented Mar 25, 2021

View reviewed changes

sycl/doc/KernelProgramCache.md Outdated Show resolved Hide resolved

Update sycl/doc/KernelProgramCache.md

09d3ee7

vladimirlaz requested a review from pvchupin March 25, 2021 15:56

vladimirlaz requested a review from kbobrovs April 2, 2021 15:28

Remove duplicated and recursive functions

76fbc2c

vladimirlaz added 2 commits April 6, 2021 11:19

Save code with obsoleting .lock files

349920c

Implement non-nlocking lock files

cdda42e

Fix comments

2c95c35

kbobrovs reviewed Apr 6, 2021

View reviewed changes

s-kanaev reviewed Apr 7, 2021

View reviewed changes

sycl/source/detail/persistent_device_code_cache.cpp Outdated Show resolved Hide resolved

sycl/source/detail/persistent_device_code_cache.cpp Outdated Show resolved Hide resolved

sycl/source/detail/persistent_device_code_cache.hpp Outdated Show resolved Hide resolved

vladimirlaz added 2 commits April 7, 2021 18:02

Apply review remarks

8dad336

Added image sized thresholds

9775e50

vladimirlaz requested review from kbobrovs and s-kanaev April 7, 2021 19:03

kbobrovs reviewed Apr 7, 2021

View reviewed changes

vladimirlaz added 3 commits April 8, 2021 10:21

Aply review comments

b0c6a6d

Merge remote-tracking branch sycl into jit_cache_2

b395ed7

Disable test on CUDA because non-SPIRV target are not covered by pers…

740408b

…istent cache

s-kanaev reviewed Apr 8, 2021

View reviewed changes

sycl/doc/EnvironmentVariables.md Outdated Show resolved Hide resolved

sycl/doc/EnvironmentVariables.md Show resolved Hide resolved

Apply suggestions from code review

163fb07

Co-authored-by: sergei <[email protected]>

vladimirlaz requested review from s-kanaev and kbobrovs April 8, 2021 12:01

s-kanaev approved these changes Apr 8, 2021

View reviewed changes

kbobrovs approved these changes Apr 9, 2021

View reviewed changes

vladimirlaz merged commit ec97165 into intel:sycl Apr 9, 2021

bader reviewed Apr 11, 2021

View reviewed changes

vladimirlaz mentioned this pull request Apr 12, 2021

[SYCL] Fix post-commit failure #3532

Merged

vladimirlaz deleted the jit_cache_2 branch April 23, 2021 06:26

[SYCL] Persistent program cache #3391

[SYCL] Persistent program cache #3391

Uh oh!

Conversation

vladimirlaz commented Mar 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vladimirlaz commented Mar 22, 2021

Uh oh!

vladimirlaz commented Mar 23, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vladimirlaz commented Mar 25, 2021

Uh oh!

vladimirlaz commented Apr 5, 2021

Uh oh!

vladimirlaz commented Apr 6, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vladimirlaz commented Apr 7, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vladimirlaz commented Apr 8, 2021

Uh oh!

s-kanaev left a comment

Choose a reason for hiding this comment

Uh oh!

pvchupin commented Apr 8, 2021

Uh oh!

kbobrovs left a comment

Choose a reason for hiding this comment

Uh oh!

bader Apr 11, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vladimirlaz commented Mar 22, 2021 •

edited

Loading