Configure ssh for MPI

To be able to run a parallel computation on a network of computers via MPI, one has to be able to log in to any of the machines without having to enter a password. This can be achieved easily using secure shell key authentication. This article describes the methodology to setup quickly ssh key authentication for MPI applications.

Principle of key authentication

The principle of key authentication is the following. A key-pair is generated which consists of private key and a public key. To generate the key-pair, a pass-phrase is used which associates the public key to the private key. A copy of the private key is stored on machine1 and the public key on machine2. When logging into machine2 from machine1, the secure shell program asks for the pass-phrase that matches the public key to the private key. The user is then authentified and fully logged into machine2.

This two stage process is a lot more secure than simply using a password as you also need to "own" the private key. This is a bit like credit card authentication; the system would be a lot less secure if you could just enter your PIN, without actually showing the card in the shop.

The private key is private (!). It needs to be known to you and only you. So be careful! You don't really care who can see the public key as only the private key fits with the public key. On wikipedia, the the public key is compared to a padlock. You don't care if people see the padlock as long as only you has the key to open it.

In the rest of this article, we assume that the user want to run parallel jobs on a network of machines called master, slave1, slave2 etc... master is the logging node, i.e. master is likely to be open to the Internet or to other machines.

Key generation on master

To generate a pair of public and private keys, use the following command:

master $ ssh-keygen -t rsa

If your network of computers for the parallel computations is on a safe private network and if no sensitive data is stored on the computing nodes, you should consider using a blank pass-phrase. Remember, this is different from a blank password, you still need to own the private key to be able to log in.

However, using a non blank and long pass-phrase is obviously more secure... but it is then slightly more complicated to setup MPI. In particular, the pass-phrase caching program ssh-agent will be required. This is described briefly in the "Advanced features" section at the end of this article.

After creating the key-pair, you will be left with two extra files in the .ssh/ folder:

master $ ls .ssh/
id_rsa  id_rsa.pub

id_rsa is your private key. It should be visible and writable to you only. id_rsa.pub is the public key. When you are at it, check the properties of the .ssh/ folder and id_rsa files. You should have something like:

master $ ls -la .ssh
drwx------  2 login login 4096 Aug 18 08:32 .
drwx------ 89 login login 4096 Aug 21 09:58 ..
-rw-------  1 login login  744 Mar 30 15:10 id_rsa
-rw-r--r--  1 login login  598 Mar 30 15:10 id_rsa.pub

If you have used a blank key or if you are likely to use other keys, rename these keys to something a bit more explicit, for instance:

mv id_rsa id_rsa.mpi
mb id_rsa.pub id_rsa.mpi.pub

Copy the public key to the slave1

The public key now needs to be copied into a file called authorized_keys in the .ssh/ folder on each slave. The methods is slightly different if your home directory is shared between the master/slaves or not.

Attention, the first time you log in in slave1, you will be asked to recognise slave1 as a "known host". This is a security feature which aims to avoid man in the middle attacks; i.e. another computer which would pretend to be slave1. "Accepting" the new host builds an entry for slave1 known_hosts file in the .ssh/ folder on master. This operation is done only once. If the hardware changes on slave1, secure shell will detect a mismatch between the new machine called slave1 and its entry in the known_hosts and will refuse to connect. The entry in known_hosts will need to be deleted before being able to log in again.

Different home directories on each slave

If the home directory is separate on each slave, the following commands should do the job nicely for slave1:

master $ scp .ssh/id_rsa.mpi.pub login@slave1:
master $ ssh login@slave1
login@slave1 password:
slave1 $ mkdir .ssh
slave1 $ chmod 0600 .ssh
slave1 $ cat id_rsa.mpi.pub >> .ssh/authorized_keys
slave1 $ rm -f id_rsa.mpi.pub
slave1 $ exit
master $

Shared home directory between the master and the slaves

If your home directory is shared you have an identical .ssh folder on all slaves and the master. You can then create authorized_keys from the master:

master $ cat .ssh/id_rsa.mpi.pub >> .ssh/authorized_keys

Specification of the key to use

As we have renamed the key, we need to tell the ssh program that we intent to use this non-default key. This is done by creating a file named configin the .ssh folder on master which contains the following lines:

IdentityFile ~/.ssh/id_rsa
IdentityFile ~/.ssh/id_rsa.mpi

Now try to log into slave1 from master. You should be asked for the pass-phrase this time:

master$ ssh login@slave1
Enter passphrase for key '.ssh/id_rsa.mpi':

Note that if you are using a blank pass-phrase, you will not be asked for a pass-phrase at all and will be logged in automatically. If you have not used a blank pass-phrase you will need to set-up ssh-agent.

Also, if your home directory is shared between the master/nodes, this might be the first time you log into slave1 and you will need to recognise it as a "known host". See the note at the top of this section.

If this work, you can then copy the public key to all slaves and try to log into them. If all went well, you should be able to log in each slave using the pass-phrase.

Advanced features

ssh-agent

ssh-agent running ssh-add

master$ ssh login@slave1 slave01$

List of keys to use

from "master, slave01, slave02" sdvkms;dflmk