Child pages
  • Pydpiper Virtual Machine

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

Table of Contents

Downloading the Virtual Machine

Download a tarball of the Virtual Machine (VM) here (last updated: June 24, 2015, PydPiper version 1.14-beta, https://github.com/Mouse-Imaging-Centre/pydpiper/releases). Alternatively, you can download it using wget:

Code Block
languagebash
wget --tries=3 --retry-connrefused -c http://repo.phenogenomics.ca/repo/MICe-software/MICe-Pydpiper-v1.14-beta.tar.gz -O MICe-Pydpiper-v1.14-beta.tar.gz

# unpack the tarball
tar xzvf MICe-Pydpiper-v1.14-beta.tar.gz

# using Oracle VirtualBox you can now add the machine using the .vbox file (MICe-Pydpiper.vbox)
Note
 The Virtual Machine is originally set to use 1 processor. Prior to running the VM, you can increase the number of processors in order to speed up the test cases below.

Test data

The virtual machine contains a folder on the Desktop with test data for the 4 core applications: MAGet.py, MBM.py, registration_chain.py and twolevel_model_building.py. The following sections describe how to run each of these and provides information to verify that the tests ran successfully.

Testing the MBM.py application

Open a shell, and run the following:

Code Block
languagebash
cd /home/mice-pydpiper/Desktop/Pydpiper_test_data/
mkdir test_MBM
cd test_MBM/
#
# The following command will run MBM.py on a single processor using 2GB of memory. It will sequentially 
# perform a linear 6 parameter, a linear 12 parameter and a non-linear registration stage. Finally a set
# of files will be created that can be used for statistical analysis
#
MBM.py  \
--num-executors=1 \
--proc=1 \
--mem=2 \
--pipeline-name=striatum_mutants \
--registration-method=mincANTS \
--nlin-protocol=/home/mice-pydpiper/Desktop/Pydpiper_test_data/MBM/mincANTS_protocol_224_micron_pydpiper.csv \
--init-model=/home/mice-pydpiper/Desktop/Pydpiper_test_data/MBM/initial-model/test-data-atlas.mnc \
--lsq6-large-rotations \
--lsq6-large-rotations-parameters=mousebrain \
--lsq12-subject-matter=mousebrain \
--calc-stats \
/home/mice-pydpiper/Desktop/Pydpiper_test_data/MBM/test_files_striatum_mutants/mouse_*mnc

When the application starts it will provide information about the total amount of stages in the pipeline, and how many of these stages are duplicates (redundant). Then it will launch the server and the executor. The output should be similar to this:

Code Block
languagebash
816 stages skipped as redundant. 1209 stages to run.
Daemon is running on port: 7766
The object's uri is: PYRO://10.0.2.15:7766/7f000101088120c0e3d65ae2750c770e58
Client registered (banzai!): PYRO://10.0.2.15:7767/7f000101088720c0e3d733000000a76047
Info

Using a single core (processor), this will take about 10 hours (speed up the computation time significantly by adding either additional executors, or by increasing the number of processors per executor!)

Next, use R and RMINC to determine the difference between the two input data sets. First, create a csv file that points to the absolute Jacobian determinants.

Code Block
languagebash
echo absolute_jacobians,genotype > absolute_jacobians_and_genotypes.csv; \
for file in striatum_mutants_processed/*; \
do base=`basename $file`; if [[ $base =~ "mutant" ]]; \
then type="mutant"; else type="wt"; fi; pwd=`pwd`; \
echo ${pwd}/${file}/stats-volumes/${base}-final-nlin_with_additional_inverted_absolute_log_determinant.mnc,$type; \
done >> absolute_jacobians_and_genotypes.csv

#
# Run R, and load RMINC
#

R
# from here on, the character ">" indicates the command prompt in R
> library(RMINC)
# now load the data mapping
> gf <- read.csv("absolute_jacobians_and_genotypes.csv")
# extract volume information for all structures
> volume_striatum <- anatGetAll(gf$absolute_jacobians, atlas="/home/mice-pydpiper/Desktop/Pydpiper_test_data/MBM/rough_striatum_segmentation.mnc", defs="/home/mice-pydpiper/Desktop/Pydpiper_test_data/MBM/mapping_for_striatum.csv")
# the volumes for the striatum
> volume_striatum
       striatum
 [1,]  7.254975
 [2,]  5.922732
 [3,]  7.267388
 [4,]  5.706052
 [5,]  7.421839
 [6,]  5.565661
 [7,]  5.647604
 [8,]  4.606988
 [9,]  6.649315
[10,]  5.332636
[11,]  6.738569
[12,]  5.487864
attr(,"atlas")
[1] "/home/mice-pydpiper/Desktop/Pydpiper_test_data/MBM/rough_striatum_segmentation.mnc"
attr(,"class")
[1] "anatUnilateral" "anatMatrix"     "matrix"        
attr(,"anatIDs")
[1] "1"
# plot the difference between the two genotypes:
> plot(volume_striatum[,1] ~ gf$genotype)

Code Block
languagebash
# finally we can look at the percentage difference:
> tapply(volume_striatum[,1], gf$genotype, mean)
  mutant       wt 
5.436989 6.829948 
> (tapply(volume_striatum[,1], gf$genotype, mean)[1] - tapply(volume_striatum[,1], gf$genotype, mean)[2] ) / tapply(volume_striatum[,1], gf$genotype, mean)[2] * 100
   mutant 
-20.39488 

 

Testing the MAGeT.py application

Open a shell, and run the following:

Code Block
languagebash
cd /home/mice-pydpiper/Desktop/Pydpiper_test_data/
mkdir test_MAGeT
cd test_MAGeT/

MAGeT.py  \
--create-graph \
--num-executors=1 \
--proc=1 \
--mem=2 \
--registration-method=minctracc \
--atlas-library=/home/mice-pydpiper/Desktop/Pydpiper_test_data/MAGeT/library/ \
--lsq12-protocol=/home/mice-pydpiper/Desktop/Pydpiper_test_data/MAGeT/default_linear_MAGeT_prot.csv \
--nlin-protocol=/home/mice-pydpiper/Desktop/Pydpiper_test_data/MAGeT/default_nlin_MAGeT_minctracc_prot.csv  \
/home/mice-pydpiper/Desktop/Pydpiper_test_data/MAGeT/input-files/input_*mnc

When the application starts it will provide information about the total amount of stages in the pipeline, and how many of these stages are duplicates (redundant). Then it will launch the server and the executor. The output should be similar to this:

Code Block
80 stages skipped as redundant.  68 stages to run.
Daemon is running on port: 7766
The object's uri is: PYRO://10.0.2.15:7766/7f00010151b920c0e6f789e2e537423cf3
Client   registered   (banzai!): PYRO://10.0.2.15:7767/7f00010151bf20c0e6f7a7000000b19955
Info

Using a single core (processor), this will take about 20 minutes (time for a snack!)

Verify the label files using Display:

Code Block
languagebash
Display  /home/mice-pydpiper/Desktop/Pydpiper_test_data/MAGeT/input-files/input_1.mnc -label /home/mice-pydpiper/Desktop/Pydpiper_test_data/test_MAGeT/input_1/labels/input_1_votedlabels.mnc

After changing the background colour to gray, press space bar, then d, and then d again (see more details here: http://en.wikibooks.org/wiki/MINC/VisualTools/Display), the labels will look something like this:

Code Block
languagebash
# the same thing can be done for the other file
Display  /home/mice-pydpiper/Desktop/Pydpiper_test_data/MAGeT/input-files/input_5.mnc -label /home/mice-pydpiper/Desktop/Pydpiper_test_data/test_MAGeT/input_5/labels/input_5_votedlabels.mnc

Testing the twolevel_model_building.pl application

Open a shell, and run the following:

Code Block
languagebash
cd /home/mice-pydpiper/Desktop/Pydpiper_test_data/
mkdir test_twolevel_model_building
cd test_twolevel_model_building/

twolevel_model_building.py \
--num-executors=1 \
--proc=1 \
--mem=2 \
--init-model=/home/mice-pydpiper/Desktop/Pydpiper_test_data/twolevel_model_building/init-model/file_1_timepoint_1.mnc \
--nlin-protocol=/home/mice-pydpiper/Desktop/Pydpiper_test_data/twolevel_model_building/mincANTS_example_nlin_protocol.csv  \
--stats-kernels=0.4,0.3 \
--registration-method=mincANTS \
--pipeline-name=test-two-level \
--create-graph \
/home/mice-pydpiper/Desktop/Pydpiper_test_data/twolevel_model_building/filelist.csv 

When the application starts it will provide information about the total amount of stages in the pipeline, and how many of these stages are duplicates (redundant). Then it will launch the server and the executor. The output should be similar to this:

Code Block
65 stages skipped as redundant.  334 stages to run.
Daemon is running on port: 7766
The object's uri is: PYRO://10.0.2.15:7766/7f000101525120c0ea68f609fe483a2a49
Client   registered   (banzai!): PYRO://10.0.2.15:7767/7f000101525720c0ea6934000000850a1a
Info

Using a single core (processor), this will take about 12 hours.

Testing the registration_chain.py application

Open a shell, and run the following:

Code Block
languagebash
cd /home/mice-pydpiper/Desktop/Pydpiper_test_data/
mkdir test_registration_chain
cd test_registration_chain/

registration_chain.py \
--avg-time-point=2 \
--init-model=/home/mice-pydpiper/Desktop/Pydpiper_test_data/twolevel_model_building/init-model/file_1_timepoint_1.mnc \
--nlin-protocol=/home/mice-pydpiper/Desktop/Pydpiper_test_data/twolevel_model_building/mincANTS_example_nlin_protocol.csv \
--registration-method=mincANTS \
--stats-kernels=0.5,1.0 \
--num-executors=1 --proc=1 \
/home/mice-pydpiper/Desktop/Pydpiper_test_data/twolevel_model_building/filelist.csv

When the application starts it will provide information about the total amount of stages in the pipeline, and how many of these stages are duplicates (redundant). Then it will launch the server and the executor. The output should be similar to this:

Code Block
254 stages skipped as redundant.  279 stages to run.
Daemon is running on port: 7766
The object's uri is: PYRO://10.0.2.15:7766/7f000101712d20c0eab2310a07ced16572
Client   registered   (banzai!): PYRO://10.0.2.15:7767/7f000101713320c0eab25a0000001d0f53
Info

Using a single core (processor), this will take about 2 hours (more than enough time for a special lunch, and some good coffee!)

Accessing the Virtual Machine through ssh

The virtual machine has been set up such that you can ssh into it from the parent operating system:

Code Block
languagebash
# ssh using the VM's username: "mice-pydpiper", and credentials: "MICe":
ssh -p 3022 mice-pydpiper@127.0.01

# an example of copying a file into the VM's Desktop (notice here the capital P!):
scp -P 3022 file.mnc mice-pydpiper@127.0.01:/home/mice-pydpiper/Desktop
# or with rsync
rsync -cav -e 'ssh -p 3022' --progress file.mnc mice-pydpiper@127.0.01:/home/mice-pydpiper/Desktop/

Information on how this was setup:

http://stackoverflow.com/questions/5906441/how-to-ssh-to-a-virtualbox-guest-externally-through-a-host