Why cannot I submit too many individual jobs?
When there are too many pending jobs, the scheduler may fail to process all them, causing scheduling problems. Therefore, to maintain stability, the system has a limit on how many jobs a user can submit. This limit is usually in the hundreds, and may vary depending on the system's load.
Most users who submit a huge number of individual jobs should consider using job arrays for one obvious benefit: one job-array job can hold thousands of "tasks" (or individual "runs") and consume only one (1) job out of the user's number of jobs limit. A user can then submit hundreds of job arrays (each containing thousands of "runs"). This usually can cover some of the largest "through-put" runs on the cluster.
If each individual task is very short (e.g. finish in a few minutes), users should pack several tasks into one run to increase throughput efficiency. See this FAQ for more details. Running a large number of short jobs is a severe waste of the cluster's computing power.
For more information about job array, see this page.
Most users who submit a huge number of individual jobs should consider using job arrays for one obvious benefit: one job-array job can hold thousands of "tasks" (or individual "runs") and consume only one (1) job out of the user's number of jobs limit. A user can then submit hundreds of job arrays (each containing thousands of "runs"). This usually can cover some of the largest "through-put" runs on the cluster.
If each individual task is very short (e.g. finish in a few minutes), users should pack several tasks into one run to increase throughput efficiency. See this FAQ for more details. Running a large number of short jobs is a severe waste of the cluster's computing power.
For more information about job array, see this page.