New changes.

Signed-off-by: Garima Gupta <[email protected]>

Author: garimagu

sycl/doc/PluginInterface.md

@@ -1,13 +1,13 @@
 # The DPC++ Runtime Plugin Interface.
 
 ## Overview
-The DPC++ Runtime Plugin Interface (PI) is the interface layer between
-device-agnostic part of the DPC++ runtime and the device-specific runtime layers
+The DPC++ Runtime Plugin Interface (PI) is an interface layer between the
+device-agnostic part of DPC++ runtime and the device-specific runtime layers
 which control execution on devices. It employs the “plugin” mechanism to bind
-to the device specific runtime layers similarly to what is used by libomptarget
+to the device specific runtime layers similar to what is used by libomptarget
 or OpenCL.
 
-The picture below illustrates the placement of the PI within the overall DPC++
+The picture below illustrates the placement of PI within the overall DPC++
 runtime stack. Dotted lines show components or paths which are not yet available
 in the runtime, but are likely to be developed.
 ![PI in DPC++ runtime architecture](images/RuntimeArchitecture.svg)
@@ -21,29 +21,30 @@ points implementing the PI interface. The DPC++ runtime collects those function
 pointers into a PI interface dispatch table - one per plugin - and uses this
 table to dispatch to the device(s) covered by the corresponding plugin.
 
-PI is based on a subset of OpenCL 1.2 runtime specification, it follows its
-platform, execution and memory models in all aspects except those explicitly
-mentioned in this document. A part of PI API types and functions have exact
+PI is based on a subset of OpenCL 1.2 runtime specification, it follows OpenCL's
+platform, execution and memory models in all aspects except for those explicitly
+mentioned in this document. Some of PI API types and functions have exact
 matches in OpenCL. Whenever there is such a match, the semantics also fully
-matches unless the differences are explicitly specified in this document. While
+match unless the differences are explicitly specified in this document. While
 PI has roots in OpenCL, it does have many differences, and the gap is likely
-to grow, for example in the areas of memory model and management, program
+to grow, for example in areas of memory model and management, program
 management.
 
 ## Discovery and linkage of PI implementations
 
 ![PI implementation discovery](images/PluginDiscovery.svg)
 
 Device discovery phase enumerates all available devices and their features by
-querying underlying plugins found in the system. This process is only performed
-once before any actual offload is attempted.
+querying underlying plugins found in the system. This process is performed when
+all attached platforms or devices are queried in an application; for example,
+during device selection.
 
 ### Plugin discovery
 
 Plugins are physically dynamic libraries or shared objects.
-The process to discover plugins will follow the following guidelines.
+The process to discover plugins follows the following guidelines.
 
-The SYCL Runtime will read the names of the plugins from a configuration file 
+The DPC++ Runtime reads the names of the plugins from a configuration file 
 at a predetermined location (TBD - Add this location). These plugins are
 searched at locations in env LD_LIBRARY_PATH on Linux and env PATH on Windows.
 (TBD - Extend to search the plugins at a path relative to the SYCL Runtime
@@ -53,44 +54,52 @@ of LD_LIBRARY_PATH.)
 To avoid any issues with read-only access, an environment variable SYCL_PI_CONFIG
 can be set to point to the configuration file which lists the Plugin names. The
 enviroment variable if set overrides the predetermined location's config file.
-These Plugins will then be searched in LD_LIBRARY_PATH locations.
+These Plugins are then be searched in LD_LIBRARY_PATH locations.
 It is the developer's responsibility to include the plugin names from the
 predetermined location's config file to enable discovery of all plugins.
 (TBD - Extend to support search in DT_RPATH as above.)
-A trace mechanism is provided to log the discovery/ binding/ device
-enumeration process. Eg: Display all the plugins being discovered, their
-information and supported PI version. List attached devices and their properties.
-
- TBD - design and describe the process in detail.
+In the current implementation the plugin names are hardcoded in the library.
+Configuration file or env SYCL_PI_CONFIG is currently not being considered.
+
+A trace mechanism is provided using env SYCL_PI_TRACE to log the discovery/
+binding/ device enumeration process. Different levels of tracing can be achieved
+with different values of SYCL_PI_TRACE.
+SYCL_PI_TRACE=0x01 provides basic trace of plugins discovered and bound. It also
+lists the device selector's selected device information.
+SYCL_PI_TRACE=0x02 provides trace of all PI calls made from the DPC++ runtime
+with arguments and returned values.
+SYCL_PI_TRACE=-1 lists all PI Traces above and more debug messages.
 
 #### Plugin binary interface
 Plugins should implement all the Interface APIs required for the PI Version
-it supports. It will export a function that will return the function pointer
-table that contains the list of implemented Interface Function pointers in a
-predetermined order defined in pi.h.
+it supports. There is [pi.def](../include/CL/sycl/detail/pi.def)/ 
+[pi.h](../include/CL/sycl/detail/pi.h) file listing all PI API names that can be
+called by the specific version of Plugin Interface.
+It exports a function - "piPluginInit" that returns the plugin details and function pointer
+table containing the list of pointers to implemented Interface Functions defined in pi.h.
 In the future, this document will list the minimum set of Interface APIs
 to be supported by Plugins. This will also require adding functionality to SYCL
 Runtime to work with such limited functionality plugins.
 
- TBD - list and describe the symbols that a plugin must implement in order to
- be picked up by the SYCL runtime for offload.
+(TBD - list and describe the symbols that a plugin must implement in order to
+be picked up by the DPC++ runtime for offload.)
 
 #### Binding a Plugin
-Plugins expose the information of supported PI API version.
-The Plugin Interface queries the plugins on the supported PI version and checks
-for compatibility.(TBD - Extend to support version compatibility checks without
-loading the library. Eg: Changing the plugin name to reflect the supported
-Plugin Interface version.)
-The Plugin Loader then queries each plugin for the Function Pointer Table
-and populates a list of the PI API Function Pointers for each plugin.
-The user can select/disable a specific plugin with an environment variable,
-SYCL_PI_USE. (TBD - Describe the semantics in a separate section for EV and
-trace.)
+The DPC++ Runtime loads all discovered Plugins and tries to bind them by calling
+piPluginInit API for each loaded Plugin. The Plugins return the information of 
+supported PI version and the list of implemented PI API Function pointers.
+(TBD - Use the PI API Version information and check for compatibility.
+Extend to support version compatibility checks without loading the library.
+Eg:Changing the plugin name to reflect the supported Plugin Interface version.)
 The information of compatible plugins (with the Function Pointer Table) is
 stored in the associated platforms during platform object construction.
-The PI API calls are forwarded using this information.
-There is pi.def/pi.h file that lists all PI API names that can be called by the
-Plugin Interface.
+The PI API calls are later forwarded using this information.
+A plugin is said to "bind" after this process completes with no errors.
+During device selection, the user can prefer selection of a device from a
+specific Plugin or Backend using the env SYCL_BE. The correspondence between
+a plugin and a SYCL_BE value is currently hardcoded in the runtime. 
+( TBD: Make this a part of configuration file). 
+Eg: SYCL_BE=PI_OPENCL corresponds to OpenCL Plugin.
 
 #### OpenCL plugin
 
@@ -100,18 +109,16 @@ OpenCL implementations. They can be installed either in the standard Khronos
 ICD-compatible way (e.g. listed in files under /etc/OpenCL/vendors on
 Linux) or not, and the OpenCL plugin can hook up with both.
 
-TBD - describe the nested OpenCL implementation discovery process performed by
+TBD - implement and describe the nested OpenCL implementation discovery process performed by
 the OpenCL plugin
 
 ### Device enumeration by plugins
-After the compatible plugins are loaded, the trace will show all available
-devices from each plugin. Similarly, the trace can be extended to show the
-underlying API calls that each PI plugin call is being directed to.
-
-TBD - Describe the exact API calls to enable device enumeration feature.
+Devices from all bound plugins are queried and listed as and when required, eg: during device selection in device_selector.
+The trace shows the PI API calls made when using SYCL_PI_TRACE=-1.
+(TBD - Add the trace to list all available devices when plugins are successfully bound.)
 
 ### Plugin Unloading
-The plugins not chosen to be connected to will be unloaded.
+The plugins not chosen to be connected to should be unloaded.
 
 TBD - Unloading a bound plugin.
 
@@ -123,11 +130,11 @@ able to operate on the corresponding device. The core API further breaks down
 into
   - **OpenCL-based** APIs which have OpenCL origin and semantics
   - **Extension** APIs which don't have counterparts in the OpenCL
-- **Interoperability API** which allows interoperability with underlying APIs
+- **Interoperability API** which allows interoperability with underlying runtimes
 such as OpenCL.
 
 See [pi.h](../include/CL/sycl/detail/pi.h) header for the full list and
-descriptions of PI APIs. [TBD: link to pi.h doxygen here]
+descriptions of PI APIs.
 
 ### The Core OpenCL-based PI APIs
 
@@ -166,10 +173,10 @@ in a data section.
 ### The Interoperability PI APIs
 
 These are APIs needed to implement DPC++ runtime interoperability with
-underlying "native" device runtimes such as OpenCL. Currently there are only
-OpenCL interoperability APIs, which is to be implemented by the OpenCL PI
-plugin only.  These APIs match semantics of the corresponding OpenCL APIs
-exactly.
+underlying "native" device runtimes such as OpenCL.
+There are some OpenCL interoperability APIs, which are to be implemented
+by the OpenCL PI plugin only. These APIs match semantics of the corresponding
+OpenCL APIs exactly.
 For example:
 
 ```
@@ -181,6 +188,21 @@ pi_result piclProgramCreateWithSource(
   pi_program *      ret_program);
 ```
 
+Some interoperability extension APIs have been added to get native runtime
+handles from the backend-agnostic PI Objects or to create PI Objects using the
+native handles. Eg:
+
+```
+pi_result piextDeviceGetNativeHandle(
+  pi_device device,
+  pi_native_handle *nativeHandle);
+
+pi_result piextDeviceCreateWithNativeHandle(
+  pi_native_handle nativeHandle,
+  pi_device *device);
+
+```
+
 ### PI Extension mechanism
 
 TBD This section describes a mechanism for DPC++ or other runtimes to detect