How to use Container
You can pull the container image from the public/private repository in campus and use it for calculation.
Compute Job with container
"Container system" is useful to setup your computing environment on computing server. Notice that "docker" container image should be convert into "Singularity" image before executing compute job, because our servers supports only "Singularity" container system.
To convert the image, refer the following contents.
1. Interactive Job
For trial use or some kind of program which requires interactive operation, you can run your container interactively .
Example:
"cowsay hello" command is executed in "whalesay.sif" container.
Without GPU:
[sentan@kagayaki]% qsub -q TINY -l select=1 -I
[sentan@spcc-XXX]% source /etc/profile.d/modules.csh
[sentan@spcc-XXX]% module load singularity
[sentan@spcc-XXX]% singularity shell whalesay.sif
Singularity > cowsay hello
With GPU:
[sentan@kagayaki]% qsub -q GPU-1 -l select=1 -I
[sentan@spcc-XXX]% source /etc/profile.d/modules.csh
[sentan@spcc-XXX]% module load singularity
[sentan@spcc-a40gXX]% singularity shell --nv whalesay.sif
Singularity > cowsay hello
*The container image file "whalesay.sif" is required to run. Please copy the image file from /Samples/Contaner_Images to your current directory in advance.
2. Batch File Job
Using batch file, the multiple calculation can be executed automatically.
Example batch file:
"cowsay hello" command is executed in container designed by "whalesay.sif" image file.
*The container image file "whalesay.sif" is required to run. Please copy the image file from /work/Samples/Images to your current directory in advance.
Without GPU:
PBS_singularity.csh:
#PBS -N test_singularity
#PBS -j oe -l select=1
cd $PBS_O_WORKDIR
singularity exec ./whalesay.sif cowsay hello > result
Batch file should be submitted to job management system as follows.See the page of each computing server for more detail information(CPU , memory resource and so on..)
[sentan@vpcc]% qsub -q TINY -l select=1 PBS_singularity.csh
Without GPU:
PBS_singularity.csh:
#PBS -N test_singularity
#PBS -j oe -l select=1
cd $PBS_O_WORKDIR
singularity exec --nv ./whalesay.sif cowsay hello > result
Batch file should be submitted to job management system as follows.See <How to submit a job to system> for more detail information(CPU, memory resource and so on..)
[sentan@vpcc]% qsub -q GPU-1 -l select=1 PBS_singularity.csh
Image converting for Singularity
1. Convert from "Docker hub" image
Notice :
The container Images in the Docker Hub isn't safe to use because they might includes insecure library, malicious programs . To reduce such risk, select Official image or Velified Publisher image to use.
1-2. Convert Docker image into Singularity image.
Firstly, submit the job interactively as follows .
[sentan@kagayaki]% qsub -q TINY -l select=1
[sentan@spcc-XXX]% source /etc/profile.d/modules.csh
[sentan@spcc-XXX]% module load singularity
Change the sentence "docker pull" in "pull command" copied in the above step to "docker://".
See below to download & convert the docker image .
Example: To make "python.sif" file from the image which is called by "docker pull python" command .
[sentan@spcc-XXX]% singularity pull python.sif docker://python
"python.sif" file will be created on current directory.
[sentan@spcc-XXX]% ls
python.sif
Exit from the computing node.
[sentan@spcc-XXX]% exit
Use qsub command to login computing node interactively.
[sentan@vpcc]% qsub -q TINY -l select=1:ncpus=1
Load module for Singularity :
[sentan@spcc-XXX]% source /etc/profile.d/modules.csh
[sentan@spcc-XXX]% module load /singularity
Build image file:
[sentan@spcc-xxx]% singularity pull --docker-login XXXXX.sif<file name> docker://moby21.jaist.ac.jp/<username>/<project>:<tag>
Enter Docker Username: < username
Enter Docker Password: < password
Close Interactive job after creating sif file.
[sentan@spcc-xxx]% exit