Configure ssh for MPI
To be able to run a parallel computation on a network of computers via MPI, one has to be able to log in to any of the machines without having to enter a password. This can be achieved easily using secure shell key authentication. This article describes the methodology to setup quickly ssh key authentication for MPI applications.
Principle of key authentication
The principle of key authentication is the following. A key-pair is generated which consists of private key and a public key. To generate the key-pair, a pass-phrase is used which associates the public key to the private key. A copy of the private key is stored on machine1 and the public key on machine2. When logging into machine2 from machine1, the secure shell program asks for the pass-phrase that matches the public key to the private key. The user is then authentified and fully logged into machine2.
This two stage process is a lot more secure than simply using a password as you also need to "own" the private key. This is a bit like credit card authentication; the system would be a lot less secure if you could just enter your PIN, without actually showing the card in the shop.
The private key is private (!). It needs to be known to you and only you. So be careful! You don't really care who can see the public key as only the private key fits with the public key. On wikipedia, the public key is compared to a padlock. You don't care if people see the padlock as long as you own the only key to open it.
In the rest of this article, we assume that the user want to run parallel jobs on a network of machines called master, slave1, slave2 etc... master is the logging node, i.e. master is likely to be open to the Internet or to other machines.
Key generation on master
To generate a pair of public and private keys, use the following command:
master $ ssh-keygen -t rsa
If your network of computers for the parallel computations is on a safe private network and if no sensitive data is stored on the computing nodes, you should consider using a blank pass-phrase. Remember, this is different from a blank password, you still need to own the private key to be able to log in.
However, using a non blank and long pass-phrase is obviously more secure... but it is then slightly more complicated to setup MPI. In particular, the pass-phrase caching program ssh-agent will be required. This is described briefly in the "Advanced features" section at the end of this article.
After creating the key-pair, you will be left with two extra files in the .ssh/ folder:
master $ ls .ssh/
id_rsa id_rsa.pub
id_rsa is your private key. It should be visible and writable to you only. id_rsa.pub is the public key. When you are at it, check the properties of the .ssh/ folder and id_rsa files. You should have something like:
master $ ls -la .ssh
drwx------ 2 login login 4096 Aug 18 08:32 .
drwx------ 89 login login 4096 Aug 21 09:58 ..
-rw------- 1 login login 744 Mar 30 15:10 id_rsa
-rw-r--r-- 1 login login 598 Mar 30 15:10 id_rsa.pub
If you have used a blank key or if you are likely to use other keys, rename these keys to something a bit more explicit, for instance:
mv id_rsa id_rsa.mpi
mb id_rsa.pub id_rsa.mpi.pub
Copy the public key to the slaves
The public key now needs to be copied into a file called authorized_keys in the .ssh/ folder on each slave. The methods is slightly different if your home directory is shared between the master/slaves or not.
Attention, the first time you log in in slave1, you will be asked to recognise slave1 as a "known host". This is a security feature which aims to avoid man in the middle attacks; i.e. another computer which would pretend to be slave1. "Accepting" the new host builds an entry for slave1 known_hosts file in the .ssh/ folder on master. This operation is done only once. If the hardware changes on slave1, secure shell will detect a mismatch between the new machine called slave1 and its entry in the known_hosts and will refuse to connect. The entry in known_hosts will need to be deleted before being able to log in again.
Different home directories on each slave
If the home directory is separate on each slave, the following commands should do the job nicely for slave1:
master $ scp .ssh/id_rsa.mpi.pub login@slave1:
master $ ssh login@slave1
login@slave1 password:
slave1 $ mkdir .ssh
slave1 $ chmod 0600 .ssh
slave1 $ cat id_rsa.mpi.pub >> .ssh/authorized_keys
slave1 $ rm -f id_rsa.mpi.pub
slave1 $ exit
master $
If your home directory is shared you have an identical .ssh folder on all slaves and the master. You can then create authorized_keys from the master:
master $ cat .ssh/id_rsa.mpi.pub >> .ssh/authorized_keys
This copying operation is required for all the slaves but do it first on slave1 and test before carrying one.
Specification of the key to use
As we have renamed the key, we need to tell the ssh program that we intent to use this non-default key. This is done by creating a file named config in the .ssh folder on master which contains the following lines:
IdentityFile ~/.ssh/id_rsa
IdentityFile ~/.ssh/id_rsa.mpi
Now try to log into slave1 from master. You should be asked for the pass-phrase this time:
master$ ssh login@slave1
Enter passphrase for key '.ssh/id_rsa.mpi':
Note that if you are using a blank pass-phrase, you will not be asked for a pass-phrase at all and will be logged in automatically. Also, if your home directory is shared between the master/nodes, this might be the first time you log into slave1 and you will need to recognise it as a "known host". See the note at the top of this section.
Adding a bit of security
If you know that you will only be using master to start your MPI jobs, you can tighten a bit the configuration by requestion that the key id_rsa.pub is only an authorised key when logging in from master. This is done by editing the authorised_keys on each slave as shown below:
from="master.full.domain" ssh-rsa adjAWDJSDFJawdihsdlfihlsdfhisAKSUDawdoj ...
Adding the "from="master.full.domain"" ensures that the public key is only used if the request comes from the master. This is particularly important when the home directory is shared between systems and you are using a blank pass-phrase. Obviously, replace "full.domain" by the real domain name for your machine.
Copying the authorised key on each host
Now that slave1 is fully configured, you can copy the configuration to each host. Simply copy the authorized_keys file in <code.ssh/ on slave1 to the same location on master, slave2, slave3, etc... Note that if your home directory is shared between master/slaves, you don't have anything to do here.
To check: do we really need the authorized_keys entry on master for MPI purposes??
Try to log into each slave, accepting it as a "know host" if required. You should be asked for a pass-phrase (or nothing if blank) for each host.
- If you are using a blank pass-phrase and you can log into each slave without user input, that's it! You don't need to do anything else regarding ssh.
- if you are not using a blank pass-phrase, you will need to use a pass-phrase caching program called
ssh-agent, see below.
ssh-agent
ssh-agent is a program which keeps pass-phrases in memory and re-uses them automatically when required by the secure shell. ssh-agent just outputs some variables and to start it, you need to "evaluate" it:
master $ eval `ssh-agent`
Agent pid 7760
Then, to cache a pass-phrase in memory, you need to add an "identity" (or private key) into ssh-agent. This is done by inputting:
master $ ssh-add ~/.ssh/id_rsa.mpi
Enter passphrase for ~/.ssh/id_rsa.mpi:
Identity added: ~/.ssh/id_rsa.mpi (~/.ssh/id_dsa)
Now try to login into slave1, slave2 etc... you should not have to enter your pass-phrase anymore.
To remove the pass-phrase, use:
ssh-add -d ~/.ssh/id_rsa.mpi
To have ssh-agent started at boot etc, refer to the documentation for your Linux distribution.