SSH port forwarding

In the context of the Hoffman2 cluster SSH port forwarding allows the cluster computing nodes (within the cluster private network) to connect to a port on a remote server and, for example, check out a license for a given software to be run on the cluster. This connection is made possible by forwarding the needed TCP/UDP port(s) over a secure channel via one of the cluster login nodes (which have a public address).

The reverse connection is also possible: authorized users can connect to the cluster internal license manager and check out their private licenses to run on their local machines.

To minimize security risks we recommend remote license servers to implement ways of controlling to whom or where are they allowing licenses to be checked out. This can be done at the license server software level by specifying users, networks allowed to check out licenses (generally with an option file read by the license server at start time), and at the system level by running the license server behind an ssh firewall (which will require user authentication); other ways to protect a license server from unauthorized usage are also possible but not considered in this article.

To connect to a remote license server that does NOT sit behind an SSH server and that listens on two ports we suggest the following procedure:

ssh -g -L $port1:$remotelicenseserver:$port1 \
-g -L $port2:$remotelicenseserver:$port2 \
-f $login -N

where:



$port1 is the default port on which the license server listen to
$port2 is the second port needed by the license server (see below)

$remotelicenseserver is the address of your license server

$login is one of the Hoffman2 login nodes (currently: login1, login2, login3, and login4)

To test the connection (after you have issued an ssh command similar to what shown above) you can try to see whether a telnet session to $remotelicenseserver can be established on port $port1:

telnet $remotelicenseserver $port1

if the port forwarding is working you should be getting:

Trying $remotelicenseserver...
Connected to $remotelicenseserver.
Escape character is '^]'

you can exit this connection typing:

^]

To connect to a remote license server that DOES sit behind an SSH server and that listens on two ports we suggest the following procedure:

ssh -g -L $port1:$remotelicenseserver:$port1 -g -L $port2:$remotelicenseserver:$port2 -f
$usernameonremotesshserver@$remotesshserver -N

you will receive a login prompt from the remote ssh server ($remotesshserver) to which you are connecting, for the sole purpose of port forwarding, as user $usernameonremotesshserver, you will need to enter your password (on $remotesshserver) for the ssh port forwarding to be established.

To test the connection you can try to estabilish a telnet session to your localhost on $port1:

telnet localhost $port1

if the port forwarding is working you should be getting:

Trying $remotelicenseserver...
Connected to $remotelicenseserver.
Escape character is '^]'

you can exit this connection typing:

^]

Once you have established an SSH port forwarding you will generally need to specify where the software should look for the license server. In most cases this can be accomplished by setting the value of an environmental variable. In FlexLM the generic environmental variable is LM_LICENSE_FILE, check the manual of your application for its specific name. Below we assume that the name of the environmental variable is LM_LICENSE_FILE. To instruct the software on where to look for the license file enter in your job submission bash script (or in your interactive bash session):

export LM_LICENSE_FILE=$port1@localhost

in the case of an application that needs to run across many nodes you will need to set the value of this variable to:

export LM_LICENSE_FILE=$port1@$masterhost

where the $masterhost is the name of the master node where the simulation will be running. Within a scheduler submission script (or within an interactive session, after having sourced the file /u/local/bin/set_qrsh_env.sh) the master host can be found as follows:

export LM_LICENSE_FILE=$port1@`cat $PE_HOSTFILE | head -1 | awk '{print $1}'

Note: Most license servers need to communicate over more than the one default port advertised by the license server manual. In many cases two ports are needed: the default port and a second port with a number given by the sum of the default port + 1. In case of job failure due to licensing issues you most likely will need to find out the ports that your software needs to communicate over and perform the SSH port forwarding, as shown above, across any needed port.

Also Note: before the SSH port forwarding can be established you need to make sure that the local ports, to which you are trying to forward the traffic to the remote ports on the remote host, are available. To test the availability of these ports on the local host you can run the following command:

netstat -tan | grep $localport{1,2}

this command should give you an empty output. You can programmatically test for availability of ports for example by using the following bash function:

################################
function isportavailable {
port=$1
testport=`netstat -tan | grep $port`
if [ "$testport" != "" ]; then
exit 99
fi 
}
################################

which you can invoke in your submission bash script with:

################################
isportavailable $localport1
isportavailable $localport2
################################

the script will reschedule the job in case any of the needed ports are busy on the master host.

At the end of the interactive session or the batch job you should terminate the SSH port forwarding, you can do so in bash by issuing:

running_pid=`ps aux | grep $USER | grep "ssh -g -L $port1:$remotelicenseserver:$port1 -g -L 
$port2:$remotelicenseserver:$port2 -f $login -N" | grep -v grep | awk '{print $2}' `

kill $running_pid