Guide for Installing TensorFlow and PyTorch

This guide provides step-by-step instructions for installing TensorFlow and PyTorch on the Hoffman2 cluster, suitable for both CPU and GPU setups.

Installing on CPU

1. Request Resources:

Request a compute node with sufficient memory, as the installation might not run successfully on the login node.

qrsh -l h_data=15G

2. Load Anaconda Module:

Load the Anaconda3 module to manage the Python environment.

module load anaconda3

3 . Create Conda Environment:

Set the Python version and create a new conda environment. You can name the environment as per your preference.

export PYTHON_VER=3.9

conda create -n tf_torch_cpu python=${PYTHON_VER} pandas -c anaconda -c conda-forge -y

4. Activate Environment:

Activate your newly created environment:

conda activate th_torch_cpu

5. Install TensorFlow and PyTorch:

Install TensorFlow and PyTorch with their CPU-optimized versions.

python3 -m pip install -U tensorflow

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

Installing on GPU

1. Request Resources:

Request a compute node, specifying GPU requirements. This step is necessary even for installing GPU versions.

qrsh -l h_data=10G,gpu,V100

2. Set Environment Variables:

Set the Python and CUDA Toolkit versions.

export PYTHON_VER=3.9

export CUDA_TK_VER=11.8

3. Create Conda Environment:

Create a new conda environmnet with the necessary CUDA tools.

conda create -n tf_torch_gpu python=${PYTHON_VER} cudatoolkit=${CUDA_TK_VER} pandas cudnn -c anaconda -c conda-forge -c nvidia -y

4. Activate Environment:

Activate the environment for GPU usage:

conda activate tf_torch_gpu

5. Install TensorFlow and PytOrch:

Install TensorFlow with GPU support and the relevant NVIDIA libraries, followed by PyTorch:

python3 -m pip install -U tensorflow[and-cuda]

pip3 install tensorrt tensorrt-bindings tensorrt-libs --extra-index-url https://pypi.nvidia.com

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu${CUDA_TK_VER//./}

Installing more packages:

You can install more packages using `conda install` or `pip install` that can be added to the conda environment. Just remember that since your have an activate conda environment, you will NOT use the `--user` flag if you use `pip install`

Testing the Installation:

To verify the TensorFlow installation, execute the following command. Note: GPU tests will only work if you are on a GPU-enabled node.

TensofFlow Test:

python -c "import tensorflow as tf; print('TensorFlow is using:', 'GPU' if tf.test.is_gpu_available(cuda_only=False, min_cuda_compute_capability=None) else 'CPU')"

PyTorch Test:

python -c "import torch; print('PyTorch is using:', 'GPU' if torch.cuda.is_available() else 'CPU')"

code

Creation date: 12/19/2023 5:03 PM Updated: 3/19/2024 10:34 AM