Input/output data plug-in engine (#423)

Signed-off-by: Victor Chang <[email protected]>

Author: mocsharp

guidelines/diagrams/mig-export.png

@@ -6,6 +6,7 @@ title Export Results
 
 participantgroup #00CC66 Informatics\nGateway
 participant "SCU Export Watcher" as EW
+participant "Data Export Plugin Engine" as DEPE
 participant "SCU Export" as SCU
 participant "FHIR Export" as FHIR
 participant "DICOMweb Export" as DICOMWEB
@@ -36,6 +37,16 @@ PS->EW: Notify new export task event
 activate EW
 
 EW->DISK: Load results
+EW->DEPE: Execute plugins (DicomFile, ExportRequestEventDetails)
+activate DEPE
+loop plugin in plugins
+DEPE->DEPE: execute plugin
+activate DEPE
+DEPE<<--DEPE: return DicomFile
+deactivate DEPE
+end
+EW<--DEPE: return DicomFile
+deactivate DEPE
 
 alt case DICOM
 EW->SCU: Queue DICOM export task
@@ -62,6 +73,3 @@ EA->DB: Update task state
 EW<--EA: Response
 deactivate EA
 deactivate EW
-
-
-

guidelines/diagrams/mig-export.txt

@@ -8,6 +8,7 @@ fontawesome f0f8 "Hospital\nDICOM Devices" as DICOM
 
 participantgroup #00CC66 Informatics\nGateway
 participant "DICOM SCP" as SCP
+participant "Input Data Plugin Engine" as IDPE
 participant "Payload Assembler Service" as PA
 participant "Payload Notification Service" as PN
 participant "Object Upload Service" as OUS
@@ -45,6 +46,17 @@ SCP->DB: Query for \nconfigured\nsources
 SCP<--DB: Response
 SCP->SCP: Validate
 SCP->SCP: Extract DICOM headers
+SCP->IDPE: Execute Plugins (DicomFile, DicomFileStorageMetadata)
+activate IDPE
+loop plugin in plugins
+IDPE->IDPE: execute plugin
+activate IDPE
+IDPE<<--IDPE: return DicomFileStorageMetadata
+deactivate IDPE
+end
+SCP<<--IDPE: return DicomFileStorageMetadata
+deactivate IDPE
+
 SCP->OUS: Queue for upload
 activate OUS
 OUS->DISK: Upload to temporary bucket path
@@ -67,4 +79,4 @@ deactivate PA
 PN->DISK: Move files associated with the payload to payload directory
 PN->PS: Publish new workflow request
 PN<--PS: Response
-end
+end

guidelines/diagrams/mig-scp.png

@@ -139,6 +139,7 @@ MIG Storage SCP service can be configured to accept all incoming association req
 | Save to disk | Sliding: 250ms - 1000ms | 3               |
 | Notify MWM   | Sliding: 250ms - 1000ms | 3               |
 
+
 ---
 
 ### DICOM SCU Service
@@ -279,6 +280,18 @@ The following APIs are supported to interact with the ACR-DSI API:
 
 ---
 
+#### DICOMWeb STOW-RS API
+
+[DICOMWeb STOW-RS API](../docs/api/rest/dicomweb-stow.md)
+
+---
+
+#### FHIR API
+
+[FHIR API](../docs/api/rest/fhir.md)
+
+---
+
 ### Health API
 
 [Health API](../docs/api/rest/health.md)
@@ -291,6 +304,37 @@ The following APIs are supported to interact with the ACR-DSI API:
 
 ---
 
+### Data Plug-in Engines
+
+[REQ-FNC-06] MIG Data Plug-in Engines provide a plug-in architecture to enable customization of zero or more plug-ins
+to be executed on the inbound and outbound data pipelines.
+
+#### Inbound Data Plug-in Engine
+
+When data arrives at one of the supported data-receiving services, the MIG passes the raw data through the plug-in engine to enable data manipulation.
+
+- For DICOM DIMSE, a list of plug-ins may be configured for each AE Title
+- For DICOMWeb STOW-RS, a list of plug-ins may be configured for each virtual AE Title and a single list for the default API endpoint
+- For the ACR Inference API, a single list of plug-ins may be defined
+- For FHIR, a single list of plug-ins may be defined
+- For HL7, a single list of plug-ins may be defined
+- Each plug-in is executed in the order that it is saved in the list
+- If any plug-in in the configured list fails, the data is dropped and may return an error to the sender, depending on the design of the receiving service.
+- Plug-ins MUST be lightweight and not hinder the upload process
+- Plug-ins SHALL not accumulate files in memory or storage for bulk processing
+
+
+#### Outbound Data Plug-in Engine
+
+When the export service receives an export request and downloads the files included in the request, the export data pipeline passes each file through the plug-in engine for data manipulation.
+
+- A list of plug-ins can be included with each export request
+- Each plug-in is executed in the order that it is saved in the list
+- Plug-ins MUST be lightweight and not hinder the export process
+- Plug-ins SHALL not accumulate files in memory or storage for bulk processing
+- 
+---
+
 ### Storage & Subsystems
 
 #### Data Storage
@@ -327,54 +371,18 @@ Each event is associated with a specific type and is serialized to JSON before s
 
 - Workflow Request event
   - Routing Key: `md.workflow.request`
-  - Class name: `WorkflowRequestMessage`
-
-  | Property       | Type               | Description                                                                                                                                                                                     |
-  | -------------  | -------------      | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-  | Bucket         | string             | Name of the bucket/directory name, which can be used to locate the payload on the shared storage.                                                                                               |
-  | PayloadId      | string (UUID)      | A UUID generated by Informatics Gateway for internal use.                                                                                                                                       |
-  | Workflows      | string[]           | Workflows associated with the payload, if any.                                                                                                                                                  |
-  | FileCount      | int                | Number of files in the payload.                                                                                                                                                                 |
-  | CorrelationId  | string             | For DIMSE, the correlation ID is the UUID associated with the first DICOM association received. For an ACR inference request, the correlation ID is the Transaction ID in the original request. |
-  | CallingAeTitle | string             | For DIMSE, the sender/calling AE Title of the DICOM dataset. For ACR requests, this is the transaction ID.                                                                                      |
-  | CalledAeTitle  | string             | For DIMSE, the receiving AE Title of the DICOM dataset. For ACR requests, this field is empty.                                                                                                  |
-  | Timestamp      | DateTime           | Date & time in, UTC, when the payload was created.                                                                                                                                              |
-  | Payload        | BlockStorageInfo[] | List of files in the payload                     .                                                                                                                                              |
-
-  - Definition of *BlockStorageInfo*:
-
-  | Property | Type   | Description                                                                                |
-  | -------- | ------ | ------------------------------------------------------------------------------------------ |
-  | Path     | string | Path to the file located relatively to the root of the bucket.                             |
-  | Metadata | string | Path to the metadata file located relatively to the root of the bucket. *See notes below.* |
-
-  Notes:
-    For DICOM files, the metadata file contains the serialized representation of a [DICOM JSON Model](https://dicom.nema.org/dicom/2013/output/chtml/part18/sect_F.2.html).
-    The serialized file may or may not contain any binary blob depending on user's configuration as defined in [DicomConfiguration](../src/Configuration/DicomConfiguration.cs).
+  - Class name: [WorkflowRequestEvent](https://github.com/Project-MONAI/monai-deploy-messaging/blob/develop/src/Messaging/Events/WorkflowRequestEvent.cs)
 
 - Export Complete event
   - Routing Key: `md.export.complete`
-  - Class name: `ExportCompleteMessage`
-
-   | Property     | Type          | Description                                                   |
-   | ------------ | ------------- | ------------------------------------------------------------- |
-   | WorkflowId   | string (UUID) | A UUID generated by the Workflow Manager.                     |
-   | ExportTaskId | string (UUID) | A UUID generated by the Workflow Manager for the export task. |
-   | Status       | string (enum) | Success (0), Failure (1), PartialFailure(2), Unknown(3)       |
-   | Message      | string        | Optional for error messages.                                  |
+  - Class name: [ExportCompleteEvent](https://github.com/Project-MONAI/monai-deploy-messaging/blob/develop/src/Messaging/Events/ExportCompleteEvent.cs)
 
 ##### Subscribed Events
 
 - Export task event
   - Routing Key: `md.export.request.[agent name]`
     - DIMSE SCU Export Service: `md.export.request.monaiscu`
     - DICOMweb Export Service: `md.export.request.monaidicomweb`
-  - Class name: `ExportRequestMessage`
-
-    | Property      | Type          | Description                                                                                                                                                                   |
-    | ------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-    | WorkflowId    | string (UUID) | A UUID generated by the Workflow Manager.                                                                                                                                     |
-    | ExportTaskId  | string (UUID) | A UUID generated by the Workflow Manager for the export task.                                                                                                                 |
-    | Files         | string[]      | A list of files to be exported.                                                                                                                                               |
-    | Destination   | string        | Export destination. For DICOM C-STORE, a named DICOM destination.  For ACR request, use the Transaction ID in the original request.                                           |
-    | CorrelationId | string        | For DIMSE, the correlation ID is the UUID associated with the first DICOM association received. For an ACR inference request, use the Transaction ID in the original request. |
+    - FHIR: TBD
+    - HL7: TBD
+  - Class name: [ExportRequestEvent](https://github.com/Project-MONAI/monai-deploy-messaging/blob/develop/src/Messaging/Events/ExportRequestEvent.cs)

guidelines/diagrams/mig-scp.txt

@@ -484,4 +484,23 @@ Setup notification service, make one of the dependencies unavailable, and expect
 
 #### Target Release
 
-MONAI Deploy Informatics Gateway R2
+TBD
+
+### [REQ-FNC-06] MIG SHALL allow minimum data manipulation of incoming and outgoing data while data is in memory
+
+#### Background
+
+Accessing and managing large-scale medical data between storage devices or services has posed significant bottlenecks
+in medical systems. This requirement aims to address these challenges by enabling users to effortlessly manipulate data
+as it flows into the Informatics Gateway and just before it is saved to a designated storage service. Moreover, it
+empowers users to perform data manipulation at the moment the Informatics Gateway exports the data, ensuring a seamless
+and efficient data processing experience.
+
+#### Verification Strategy
+
+Configure suported inbound and export services with one or moreplug-ins and ensure the plug-ins are called in the automated testing.
+
+#### Target Release
+
+MONAI Deploy Informatics Gateway R4
+