PBS Professional
From Yale HPC Wiki
To efficiently and fairly use HPC compute resources, we use scheduling software to run programs. A user must submit a "job" to the scheduler, specifying the resources (eg. memory, number of processors, program runtime) they need. Submitted jobs are placed in a "queue" and are executed when requested resources become available.
Contents |
PBS is our primary queuing and accounting system. The PBS model is very simple, users create scripts to execute a series of commands, the script gets submitted to a queue, and PBS decides where and when to execute the script. As a user, you can alter the resource requirements (number of CPUs, nodes, or memory etc. ) to affect how PBS decides where and when to execute the scripts.
Useful PBS Commands
Most users are extremely productive armed with the commands below. All of the commands below also have man pages.
Frequently used PBS Commands
Getting Status on the System, Queues, and Jobs
| Command | Short Description | |
| qstat | list information about queues and jobs | |
| qstat -q | list all queues on system | |
| qstat -Q | list queue limits for all queues | |
| qstat -a | list all jobs on system | |
| qstat -au userid | list all jobs owned by user userid | |
| qstat -s | list all jobs with status comments | |
| qstat -r | list all running jobs | |
| qstat -f jobid | list all information known about specified job | |
| qstat -Qf queue | list all information known about specified queue | |
| qstat -B | list summary information about the PBS server |
qsub - Submitting a PBS Script as a Job
qsub is the command to submit jobs to the queue to be considered to run. The simpliest form of using qsub is "qsub myscript.sh". A succesful qsub will return a unique job identification number, which you can use to query and alter the job in the queue. Most users at some point will want to invoke qsub with some flags as options. One of the most common option is the interactive qsub, where you can work interactively with the compute node.
For extensive examples of using qsub see Writing a PBS Script.
qdel - Canceling or Deleting a Job
qdel is the command to cancel, stop, and delete a job from the queue or running state. qdel requires the job id number, for example "qdel 2346" will delete the job 2346.
Note: It will take approx. 15min for your job to be terminated after you enter a qdel command.
qstat - Checking the Status of a Job
qstat is the command to query the status of queues and/or jobs. Without any arguments, it lists Job id, Job Name, Job Owner, Time Use(CPU time), Status, and Queue for each job.
A more useful "qstat -a" also includes # of nodes requested, # of CPUs requested, Memory Requirements, Time Requirements, and Time Running(real time).
For more information, "qstat -n" includes the hosts where the job is running. This can be useful information for monitoring your job via ganglia.
qstat also accepts the "-u [username]" flag. This flag allows you to limit the qstat output to just this user.
The "S" or status column of these reports is where the job state is indicated. "R" is for running, B is for an array which is running, and "Q" is for queued. Just about any other state indicates an error, including "E". If you submitted a job and it's not in the R or B state, you can query why the job is held up using:
qstat -f [jobid]|grep comment
For example, you might see
"comment = Not Running: No available resources on nodes".
or
"comment = Job run at Mon Aug 04 at 16:08 on ..."
or stricter PBS scheduling like on bulldogj might show
"comment = Job will run at Mon Aug 04 at 16:08"
To query a queue status and configuration, "qstat -Q -f [queue name]" is helpful. For example:
# qstat -Q -f general
Queue: general
queue_type = Execution
Priority = 100
total_jobs = 12
state_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:12 Exiting:0 Begun:0
resources_assigned.ncpus = 105
resources_assigned.nodect = 25
max_user_res.ncpus = 220
enabled = True
started = True
Indicates that the queue general is running 12 jobs over 25 nodes using 105 CPUs.
qmove - Moving a Job Between Queues
qmove allows users to move jobs between queues. On most systems, users have access to a group queue and the general queue. If you submitted a job to your group queue, but now decide to run it now in the general queue, you can simply move it using "qmove [destination queue] [job id]".
qorder - Changing the Order of Jobs
qorder allows users to rearrange their queued jobs. PBS by default uses a strict first in first out scheduling on each queue, so if you want the last job you submitted to run next you can use the qorder command to change the order. The command "qorder 642 646" will swap job 642's position with job 646's.
PBS Components
Queue names
Your queue_name should be given to you when your account was created. You may be able to guess your queue_name from a list. To print the queue names run the command:
qstat -q
Interactive PBS
Some clusters provide interactive jobs as an option. Interactive jobs allow you to "login" via pbs to a compute node to run jobs interactively. This is particularly useful if you're compiling, debugging, or want to view stderr/stdout as it's produced. To login interactively add the -I flag to qsub and exclude a script to run, for example:
qsub -I -l nodes=4:ppn=8 -q <queue name>
Will give you four nodes on the <queue name> queue.
An interactive job will define useful environment variables such as PBS_NODEFILE which will list the compute nodes reserved for your interactive job.
Note: using "qsub -V" command-line flag will pass your current environment variables to your interactive job:
qsub -V -I -l nodes=4:ppn=8 -q <queue name>
With a normal batch job, the input to the job is the script supplied via the qsub command and the output and error streams are spooled to disk files and delivered after the job completes. PBS provides support for scheduling and running jobs that require interactive user access to the input and output of the job during run time. This access is often required to run debuggers or other programs requiring feedback as part of a job that must have scheduled access to scarce hardware.
A PBS interactive job is submitted with the qsub command. If the -I option is specified on the qsub command line or in a #PBS directive in the script file [or if -W interactive=true is specified on the command line or in a directive] the job is a interactive job. A job may determine that it is an interactive batch job by the value of the environment variable PBS_ENVIRONMENT. Instead of the normal value of PBS_BATCH, it will have a value of PBS_INTERACTIVE. [Sessions not created by PBS do not have the PBS_ENVIRONMENT variable set.]
After submitting the job to the batch system, qsub remains active waiting for the job to connect over the network. When the job starts, data written to standard output and error is sent to qsub which displays it on its terminal (qsub's standard output). The qsub command reads its standard input and passes the data to the job as the job's standard input. The job is connected via a pseudo tty, so job control signals and special characters are processed by the job, not the local qsub session. Since the job's standard input is from the terminal, via qsub, the script is not executed as is a normal script. However, any #PBS directives in the script are processed by qsub and sent with the job. Therefore, job attributes and resources requirements may be specified in the script as with a normal batch job.
While qsub is waiting for the job to start, it will recognize the interrupt signal typically generated by entering CNTL-C on the keyboard. Qsub will ask if the user wishes to exit. If the user responds with yes (or any string starting with a 'y') qsub will send a request to the server asking that the batch job be aborted. Qsub will also periodically check on the batch job. If qsub finds that the job has been deleted, qsub will inform the user and exit.
When the job starts execution, qsub will inform the user and begin to relay the input, output and error streams. Control keys are passed to the job. Thus a CNTL-Z will suspend the job, not the qsub command, and CNTL-C will cause a interrupt signal to the job, not to qsub. During this time, qsub will accept three escape commands, if the line begins with:
~. Qsub will exit. This will end the job
~^Z (CNTL-Z) Qsub will suspend itself, the job remains running. Neither input nor output is transferred. The user may issue commands to the local shell.
~^Y (CNTL-Y) Qsub will suspend the part of itself that sends input to the job. This allows the user to issue commands to the local shell while still receiving the output of the job.
Note, the last two escape lines do not work if the local shell does not support job control, e.g. is the Bourne shell sh.
When the job terminates, qsub will exit returning control to the local shell.
PBS Resource Flags
When users submit a job, the job contains two things:
- a script to run and
- Information regarding resources required
The smallest request just requires the queue name in the form "-q [queue name]". PBS options can be set in your script using the #PBS notation (see Writing a PBS Script) or with the qsub command. For example,
qsub -q general myscript.sh
Further Info
For a complete list of resources and options see the PBS manpages and our PBS documentation page
For current information, we recommend reading manpages available on the cluster login servers.
PBS and MPI Jobs
PBS does not automatically paralelize an application or in any way manage the parallel processes of an application. PBS, in its most basic configuration, only manages node and CPU resources for mpi applications. i.e. PBS makes sure your job has X Nodes and Y CPUs dedicated to it, but does nothing to ensure you're using them.
For MPI jobs, PBS supplies the environment variable within the compute node environment called $PBS_NODEFILE. $PBS_NODEFILE lists all of the dedicated CPUs or hosts for the MPI execution. Our Writing a PBS Script page describes how to run mpi jobs for OpenMPI and MPICH2, two common MPI packages. If you're using another package you can email hpc@yale.edu with questions.
PBS select and MPI
If you are using the PBS select notation, you will need to include the "mpiprocs" flag.
Determining your queue name
You may be able to guess your queue_name from a list. To print the queue names run the command:
system_usage
Additional PBS information
See our PBS Pro Documentation page.
