Why my highp job did not start in 24 hours?

A highp job will start in 24 hours provided that your group does not overuse purchased resources (see also http://hpc.ucla.edu/hoffman2/computing/policies.php#highp).
The common reasons a highp job did not start in 24 hours are:

• 1. You did not specify the highp option in your job script.
  Check your job script, look for a line that starts with #$ -l. highp should be one of a parameter. For example, the line should look like:
  #$ -l h_data=1G,h_rt=48:00:00,highp
• 2. The pending job in question does not have highp option. (See below about how to check this.)
• 3. Members of your group are already running long jobs on the purchased compute nodes.
  In this case, your highp job will be queued until resources become available. (You still need to add "highp" to the job script described above.)
• 4. Your research group is not a Hoffman2 shared cluster program participant.
  Consider join the program and enjoy the benefits.
• 5. The of h_data and number of slots is greater than the per-node memory size of your group nodes.
  For example, you have h_data=8G and -pe shared 7. This means you are requesting a node with 56 GB (=8G*7) of memory. If each of your group's nodes has, say, 32GB of memory, your highp job will not start.

To check whether your pending job has the highp option, use the following commands and steps:

• 1. Find out job ID (of the pending job): qstat -s p -u $USER
• 2. Check if highp is specified for the job in question: qstat -j job_id | grep ^'hard resource_list' |grep highp If you see no output from the command above, it means that job does not have highp option. You need to specify highp. See below about how to use qalter command to fix this.
  If you see something like: hard resource_list: h_data=1024M,h_rt=259200,highp=TRUE
  This means the job does have highp option specified.
  

To alter (without re-submitting it) a already-pending job from non-highp to highp, use following steps:

• 1. Get the "hard resource" parameter list:
  qstat -j job_id | grep ^'hard resource_list' For example, you have hard resource_list: h_data=1024M,h_rt=259200
  You will use the list beyond the colon (":") in the "hard resource_list" output above in the next step.
• 2. Add the highp option to the hard resource_list using the qaltercommand:
  qalter -l h_data=1024M,h_rt=259200,highp job_id where job_id is replaced by the actual job ID (number). For more information about qalter, try the command: man qalter .