Guide for Installing TensorFlow and PyTorch
This guide provides step-by-step instructions for installing TensorFlow and PyTorch on the Hoffman2 cluster, suitable for both CPU and GPU setups.
Installing on CPU
1. Request Resources:
Request a compute node with sufficient memory, as the installation might not run successfully on the login node.
qrsh -l h_data=15G
2. Load MiniForge Module:
Load the miniforge module to manage the Python environment.
module load miniforge
3 . Create Conda Environment:
Set the Python version and create a new conda environment. You can name the environment as per your preference.
export PYTHON_VER=3.9
conda create -n tf_torch_cpu python=${PYTHON_VER} pandas -c anaconda -c conda-forge -y
4. Activate Environment:
Activate your newly created environment:
conda activate th_torch_cpu
5. Install TensorFlow and PyTorch:
Install TensorFlow and PyTorch with their CPU-optimized versions.
python3 -m pip install -U tensorflow
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
Installing on GPU
1. Request Resources:
Request a compute node, specifying GPU requirements. This step is necessary even for installing GPU versions.
qrsh -l h_data=10G,gpu,V100
2. Set Environment Variables:
Set the Python and CUDA Toolkit versions.
export PYTHON_VER=3.9
export CUDA_TK_VER=11.8
3. Create Conda Environment:
Create a new conda environmnet with the necessary CUDA tools.
conda create -n tf_torch_gpu python=${PYTHON_VER} cudatoolkit=${CUDA_TK_VER} pandas cudnn -c anaconda -c conda-forge -c nvidia -y
4. Activate Environment:
Activate the environment for GPU usage:
conda activate tf_torch_gpu
5. Install TensorFlow and PytOrch:
Install TensorFlow with GPU support and the relevant NVIDIA libraries, followed by PyTorch:
python3 -m pip install -U tensorflow[and-cuda]
pip3 install tensorrt tensorrt-bindings tensorrt-libs --extra-index-url https://pypi.nvidia.com
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu${CUDA_TK_VER//./}
Installing more packages:
You can install more packages using `conda install` or `pip install` that can be added to the conda environment. Just remember that since your have an activate conda environment, you will NOT use the `--user` flag if you use `pip install`
Testing the Installation:
To verify the TensorFlow installation, execute the following command. Note: GPU tests will only work if you are on a GPU-enabled node.
TensofFlow Test:
python -c "import tensorflow as tf; print('TensorFlow is using:', 'GPU' if tf.test.is_gpu_available(cuda_only=False, min_cuda_compute_capability=None) else 'CPU')"
PyTorch Test:
python -c "import torch; print('PyTorch is using:', 'GPU' if torch.cuda.is_available() else 'CPU')"