Submitting a job

[] Cray XC30 job queue is managed by a scheduler system．

Please check the parallel servers user manual．

[]

• Node / Core = Number of available nodes / Number of cores
• Execution time = the maximum time that a job submitted to the queue is in execution state
• Number of concurrently executed jobs = Number of jobs in the same queue that can be executed simultaneously
• Number of concurrently executed jobs / user = the number of jobs from the same user in the same queue that can be executed at the same time
• Number of simultaneous inputs / user = the number of jobs that one user can simultaneously have in the same queue (including running jobs)

■ 1 Node is 2CPU that is 24 cores．
■ Job allocation is done per-node. That is, single job of a single thread occupies 1 node = 24 cores.．
■ 4 nodes are exclusively allocated for the TINY queue. Thus the XLARGE queue is limitedd to 8544 Cores. The LARGE-L class is unlimited.

＊＊ For more details about job classes please check the link

[] XC30 uses the PBS standard qstat command, the number of nodes in use may not be displayed correctly.

The following command can be used to check the currently running jobs.

nqstat

[]

MPI program

#!/bin/csh
#PBS -q <QUEUE>
#PBS -j oe
#PBS -l mppwidth=48　<- No. of MPI processes to use　<required>
#PBS -l mppnppn=24   <- No. of processes per node <required: max. 24>
#PBS -l mppdepth=1　 <- Should be 1 for MPI programs

#PBS -N BATCH-JOB　　<- Arbitary job name

cd $PBS_O_WORKDIR　<- To move to the working directory

aprun -n 48 -N 24 -d 1  ./a.out

XC30 uses the aprun command to execute programs．

aprun [-n num/-N num/ -d num ] <ELF>

-n: no. of MPI processes　（according to the mppwidth of the PBS）
-N: no. of MPI processes per node　（according to the mppnppn of the PBS）
-d: no. of threads per MPI process　（according to the mppdepth of the PBS）

MPI/OpenMP hybrid

#!/bin/csh
#PBS -l mppwidth=4 <- number of MPI processes to run. 2 process x 2 node = 4, in this case two node utilization
#PBS -l mppnppn=2 <- two processes within a node. 12 threads x 2 process = 24, using all 24 cores in the node.
#PBS -l mppdepth=12　<- 12 threads using 12 cores in the CPU

$PBS_O_WORKDIR　<- to move to the working directory

aprun -n 4 -N 2 -d 12 ./a.out

Running multuple scripts

In XC30, the number of jobs that one user can have simultaneously has been limited since 2013/7. To execute more , you need to list multiple executable files in the job script.

An example is as follows.

* The execution results of some programs may be overwritten depending on how you write the script. Be careful about the storage location of the results.

#!/bin/csh
#PBS -q SMALL
#PBS -j oe
#PBS -l mppwidth=48　<- no. of MPI processes　<required>
#PBS -l mppnppn=24   <- no. of processes per node <required:max. 24>
#PBS -l mppdepth=1　 <- 1 for MPI programs

#PBS -N BATCH-JOB　　<- Job arbitary name

cd /work/xxxxxxxx/job1
aprun -n 48 -N 24 -d 1  ./job1.exe

cd /work/xxxxxxxxx/job2
aprun -n 48 -N 24 -d 1  ./job2.exe

cd /work/xxxxxxxx/job3
aprun -n 48 -N 24 -d 1  ./job3.exe

[] If you can apply in advance and obtain consent, you can occupy the whole or some nodes for a limited period.

A period-limited queue is called an application queue．

[]

1. Apply at least 2 week ahead for LONG-L queue by contacting mpc-admin。
2. mpc-admin group will inform the applicant with the schedule to use the LONG-L queue and stop submitting jobs according to that schedule.
3. The applicant submits to the LONG-L queue at the specified schedule when the resources are available.
4. LONG-L queue will finish and the results should be reported.