remove the need for a double copy to minio

[neildsouth]

Description

on a store request the gateway currently
saves to file locally,
saves to Minio temp folder
moved from temp folder to final destination

This branch skips the second step and saves directly to the final folder.

Status

Work in progress

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• All tests passed locally.
• Documentation comments included/updated.
• User guide updated.
• I have updated the changelog

[codecov]

Codecov Report

Merging #395 (556d2c3) into develop (d27c8e3) will increase coverage by 0.03848%.
The diff coverage is 100.00000%.

@@                 Coverage Diff                 @@
##             develop        #395         +/-   ##
===================================================
+ Coverage   79.63869%   79.67717%   +0.03848%     
===================================================
  Files            328         328                 
  Lines          20647       20568         -79     
  Branches         955         950          -5     
===================================================
- Hits           16443       16388         -55     
+ Misses          3876        3858         -18     
+ Partials         328         322          -6     

Flag	Coverage Δ
unittests	79.67717% <100.00000%> (+0.03848%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/Api/Storage/FileStorageMetadata.cs	100.00000% <ø> (ø)
...ay/Services/Connectors/PayloadMoveActionHandler.cs	64.17910% <ø> (-6.47958%)	⬇️
src/Configuration/ConfigurationValidator.cs	96.96970% <100.00000%> (ø)
...work/Test/SourceApplicationEntityRepositoryTest.cs	98.73418% <100.00000%> (+0.01622%)	⬆️
...tion.Test/SourceApplicationEntityRepositoryTest.cs	98.78049% <100.00000%> (+0.01506%)	⬆️
...ateway/Services/Connectors/DataRetrievalService.cs	81.56028% <100.00000%> (+0.91512%)	⬆️
...icsGateway/Services/Connectors/PayloadAssembler.cs	84.14634% <100.00000%> (+0.19571%)	⬆️
...rmaticsGateway/Services/DicomWeb/IStreamsWriter.cs	85.85859% <100.00000%> (+0.14430%)	⬆️
...rc/InformaticsGateway/Services/Fhir/FhirService.cs	78.94737% <100.00000%> (+0.56898%)	⬆️
...maticsGateway/Services/HealthLevel7/MllpService.cs	77.41935% <100.00000%> (+0.24543%)	⬆️
... and 4 more

... and 2 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d27c8e3...556d2c3. Read the comment docs.

[lillie-dae]

would it be worth declaring a ctor that can take GUID?

example

public DicomFileStorageMetadata(Guid associationId, Guid identifier, Guid studyInstanceUid, Guid seriesInstanceUid, Guid sopInstanceUid) : this(
            associationId.ToString(),
            identifier.ToString(),
            studyInstanceUid.ToString(),
            seriesInstanceUid.ToString(),
            sopInstanceUid.ToString())
        { }

[neildsouth]

looking at the code the only benefit for the above change would be the tests. so ....

[mocsharp]

This PR removes the ability to have the same instance in multiple payloads.
There was a discussion (some time ago) to have rule-based instance grouping which could result in sending the same instance to multiple payloads.

[JoeBatt1989]

We have ran some performance tests on this version and seen some significant improvements when sening the same data through (CT 326 slices). We are also seeing MinIO not reporting that we need to reduce our request rate which does sometimes leave the MIG is a broken state.

Previous baseline tests for MIG:

• Avg = 34.4
• Max = 36.2

New baseline tests for MIG:

• Avg = 14
• Max = 16.2

2023-06-08 09:51:14.1811|712|INFO|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationActionHandler||Workflow request published to md.workflow.request, message ID=33e05f11-c2bb-4492-ab29-25fd38a2135e. Payload took 00:00:13.1329433 to complete. 2023-06-08 09:52:46.3083|712|INFO|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationActionHandler||Workflow request published to md.workflow.request, message ID=01f9c934-d8aa-49f4-8f69-4bda2894e53a. Payload took 00:00:14.0552913 to complete. 2023-06-08 09:53:23.4085|712|INFO|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationActionHandler||Workflow request published to md.workflow.request, message ID=a8fc9cf1-c441-4ed5-96c8-9f6fdd5e0207. Payload took 00:00:13.9584748 to complete. 2023-06-08 09:53:50.4880|712|INFO|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationActionHandler||Workflow request published to md.workflow.request, message ID=06f49a60-4afe-4c85-a840-bf57b709cfea. Payload took 00:00:12.7193439 to complete. 2023-06-08 09:54:12.5633|712|INFO|Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationActionHandler||Workflow request published to md.workflow.request, message ID=0d65548e-921e-488e-876e-00d420cc6b7d. Payload took 00:00:16.2113721 to complete.

[woodheadio]

@mocsharp Based on the above findings and that this appears to have increased the stability of the MIG when sending 300-500 slice payloads in quick succession, I'm going to merge this change in. We can revisit this again if the requirement you mention is asked for again.

[github-actions]

🎉 This issue has been resolved in version 0.3.21 🎉

The release is available on:

• GitHub Release