Child pages
  • Disk cleanup after a registration
Skip to end of metadata
Go to start of metadata
You can ignore the "No such file or directory" error that prints if you do delete things. 
The files will indeed be deleted!

Quick and Thorough Cleanup

If you are absolutely happy with your results then you can copy and paste the following two code blocks and delete all temporary files. Cd into the directory of your pipeline and run the following command.

PIPELINEPATH=$(find . -maxdepth 1 -name "*_processed" -type d) \
echo $PIPELINEPATH

Confirm that this is indeed the pipeline for which you would like to get rid of all temporary files, then you may copy and paste the following code block and delete all temporary files associated with this pipeline.

find $PIPELINEPATH -name tmp -type d -exec rm -rf \{} \;
for file in *_processed/*/resampled/*{__concat_lsq6_N_I_lsq6_lsq12_and_nlin-resampled.mnc,_N_I_lsq6_avg_lsq12-resampled.mnc}; do if [ -f $file ]; then rm -f $file; fi; done
for file in *_processed/*/transforms/*{_mt_to_,_N_I_lsq6_lsq12_and_nlin_inverted,__concat_lsq6_N_I_lsq6_lsq12_and_nlin}*; do if [ -f $file ]; then rm -f $file; fi; done
for file in *_processed/*/transforms/*N_I_lsq6_lsq12_and_nlin__concat_inverted_linear_part*; do if [ -f $file ]; then rm -f $file; fi; done
for file in *_processed/*/masking/transforms/*{mt_to_Dorr_2008_Steadman_2013_Ullmann_2013_on_NRXN1a_v1_average_lsq12,mt_to_Dorr_2008_Steadman_2013_Ullmann_2013_on_NRXN1a_v1_average_nlin_5}*; do if [ -f $file ]; then rm -f $file; fi; done

Recover Pipeline Names

MICe-build-model / MBM.py puts lots of data into a tmp directory for each of the input files. These files can safely be deleted once the pipeline has finished and the results look good. If you are performing a retrospective clean up, and forgot where you've been running all your pipelines, this command will indicate all the directories you're interested in:

# find any pipeline. Run this from a "root" directory.
> find . -name "*_processed" -type d

The above command is fun to run regardless. Who knows what'll show up!

The most basic clean up

Removing only the tmp directories after your pipeline finished and you've verified that the registration went well, and all files are properly aligned

# Using the output from the command above you can run the following which only prints all the tmp directories:
> find /path/to/pipeline_processed -name tmp -type d

# and actually remove them
> find /path/to/pipeline_processed -name tmp -type d -exec rm -rfv \{} \;

Another way to find all possible pipeline tmp files and delete them:

# remove all possible tmp directories:
> find /main/path/ -path '*_processed/*/tmp' -type d -exec rm -rfv \{} \;
# on HPF, that could be one of the following:
> find /hpf/largeprojects/MICe/yourusername/ -path '*_processed/*/tmp' -type d -exec rm -rfv \{} \;

Deep Cleaning

#
# *_processed/*/resampled

# if you only want to keep the lsq6 and final nlin file:
for file in *_processed/*/resampled/*{__concat_lsq6_N_I_lsq6_lsq12_and_nlin-resampled.mnc,_N_I_lsq6_avg_lsq12-resampled.mnc}; do if [ -f $file ]; then rm -fv $file; fi; done 
#
# *_processed/*/transforms

# From all the transformations that are produced you only need to keep the:
# pure non linear ("*N_I_lsq6_lsq12_and_nlin__concat_inverted_linear_part.xfm") - used to calculate relative Jacobians
# full linear and non linear ("*_N_I_lsq6_lsq12_and_nlin") - used to calculate absolute Jacobians
# the following can all safely be removed:
for file in *_processed/*/transforms/*{_mt_to_,_N_I_lsq6_lsq12_and_nlin_inverted,__concat_lsq6_N_I_lsq6_lsq12_and_nlin}*; do if [ -f $file ]; then rm -fv $file; fi; done

# And if you want to remove the transforms used to create relative jacobians
for file in *_processed/*/transforms/*N_I_lsq6_lsq12_and_nlin__concat_inverted_linear_part*; do if [ -f $file ]; then rm -fv $file; fi; done
#
# *_processed/*/masking/transforms

# This removes all the transforms used to register to the atlas for initial masking purposes
for file in *_processed/*/masking/transforms/*{mt_to_Dorr_2008_Steadman_2013_Ullmann_2013_on_NRXN1a_v1_average_lsq12,mt_to_Dorr_2008_Steadman_2013_Ullmann_2013_on_NRXN1a_v1_average_nlin_5}*; do if [ -f $file ]; then rm -fv $file; fi; done

Other cleaning:

Once your files are reconned/dist corrected, you can delete the original fid files. You can also delete the phase calibration folder, but given the small size of the procpar and global files, it's worth it to keep them in case we need them in the future.

Moreover, if multiple registrations have been run at different stages of a project, earlier registrations can be deleted. For example, if a registration was run on half the group for a MICe meeting presentation, and then later a registration was run on the full data set, the earlier registration can be discarded. An exception to this is when a registration was used for publication purposes and more data was later added. These registrations are worth keeping to be able to reproduce figures from papers, etc.

Once your data has been published, there's only three files you should be keeping for each subject: the final resampled nlin, the absolute jacobians, and the relative jacobians. You should also keep your atlas. You can delete everything else registration-related.