WIP: Attempt to provide a proxy for gzip file

[bpinsard]

My hard drive is full but some of our codes/interfaces uses matlab/spm.
So I coded an FileProxy extension to Node that uncompress the .nii.gzip files before running the interfaces, then compress the output nii files to nii.gz (though only if the output_name is specified).
It maybe not the best design, I have been thinking after that Node instances could be added a list of file preprocessor/postprocessors object to handle certain types of files.
I push it not to be included but to discuss about solution to handle spm .nii.gz files problem and maybe other interfaces I/O that I am not aware of.
What do you think?

[chrisgorgo]

This is something that we should finally address. The general testcase is on the fly conversion of files into compatible formats (not only nii -> nii.gz, but also nrrd to nii maybe even dicom to nii). There are some use cases we need to address:

1. could we make it implicit? For example we know that SPM interfaces are not compatible with nii.gz. Therefore if ones connects .nii file to an SPM Interface it should be converted. MAybe we should put compatible filetypes in traits?
2. File conversion can be made by many different external pieces of software as well as python libraries. Maybe we should make some flexible framework for those converters?

[bpinsard]

I completely agree there will be other cases of file format incompatibility and we need to find a as generic as possible way to handle this without having to add nodes to do the pipeline. I tried to handle this at the Node level but there could be information in the interface traits, it is more of a quick hack to solve my full disk problem.

[chrisgorgo]

The issue is that hard disk space and format incompatibilities are sometimes not the same problem. For example - imagine this pipeline:

nii.gz -> A(FSL) -> B(FSL) -> C(SPM) -> D(SPM) -> E(FSL)

if you only care about data format conversion nii.gz will be converted to .nii only once (B->C). Since FSL can read .nii there is no need to do anything at the D->E step.

On the other side if you want to save space you would like to compress outputs of SPM each time.

[chrisgorgo]

Ok men. I don;t want to hold you back with this any longer. Just please add some code example in the docstrings (like a super simple SPM/FSL pipeline) and a mention in the changelog and we are ready to merge.

[bpinsard]

Yes I left this a bit aside as to my eyes it seems more a quick hack to solve my disk space problem, maybe it should not be merge now.
There would be better way to do the job, maybe being able to register multiple proxies for a single node to handle multiple file format changes.
The on-the-fly conversion with interfaces specifying accepted file formats would be great and then having a collection of standard file-proxies that can be extended with new ones for exotic cases.
Btw, is there any kind of roadmap for nipype, the "0.4 brainstorm" is a bit old now and there are multiple ideas showing up for enhancements.

[oesteban]

I find this proxy very interesting. Not only for the compression management itself, it would be a great advance for those interfaces in python (I'm thinking of dipy right now), that handle very large nifti files. If they are not compressed, then they are not fully loaded into memory if it's not absolutely necessary.

[codecov-io]

Codecov Report

Merging #454 into master will decrease coverage by 1.57%.
The diff coverage is 20%.

@@            Coverage Diff             @@
##           master     #454      +/-   ##
==========================================
- Coverage   72.16%   70.58%   -1.58%     
==========================================
  Files        1144     1029     -115     
  Lines       57606    50721    -6885     
  Branches     8251     7366     -885     
==========================================
- Hits        41572    35802    -5770     
+ Misses      14733    13805     -928     
+ Partials     1301     1114     -187

Flag	Coverage Δ
#smoketests	52.37% <ø> (-19.79%)	⬇️
#unittests	70.34% <20%> (+0.54%)	⬆️

Impacted Files	Coverage Δ
nipype/pipeline/file_proxy.py	20% <20%> (ø)
nipype/interfaces/freesurfer/tests/test_utils.py	14.03% <0%> (-82.84%)	⬇️
nipype/workflows/smri/freesurfer/ba_maps.py	11.11% <0%> (-79.17%)	⬇️
nipype/interfaces/freesurfer/tests/test_model.py	25% <0%> (-75%)	⬇️
nipype/workflows/rsfmri/fsl/resting.py	14.75% <0%> (-70.73%)	⬇️
...ype/interfaces/freesurfer/tests/test_preprocess.py	18.96% <0%> (-70.65%)	⬇️
nipype/workflows/smri/freesurfer/autorecon2.py	2.3% <0%> (-68.96%)	⬇️
nipype/workflows/smri/freesurfer/autorecon3.py	2.09% <0%> (-67.01%)	⬇️
nipype/workflows/smri/freesurfer/bem.py	38.46% <0%> (-61.54%)	⬇️
nipype/workflows/smri/freesurfer/recon.py	8.57% <0%> (-57.15%)	⬇️
... and 946 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ee584b...2f6a944. Read the comment docs.

[effigies]

This pull request is "orphaned," which means it has been deemed to be abandoned by its original author. Orphaned pull requests have not been rejected, and we hope that if a user sees one that will meet their needs with a little work, that they will fork it and open a new pull request (or, in the case of the original author, reopen the original PR).

We ask that all adopted PRs be updated to merge or rebase the current master. If you would like to adopt a PR and need help getting started, any of a number of contributors will be happy to help.