How do I request large memory to run sequential (1-core) program?

If you are requesting less than 64 GB, use the h_data to specify the requested memory size, e.g.
qsub -l h_data=32G ... 
You can also put -l h_data=32G in your job script file.
In this case, you are requesting a single core (slot), so you should not specify any -pe option.
If you are requesting more than 64GB, please contact us.