View source for Configure ssh for MPI

To be able to run a parallel computation on a network of computers via MPI, one has to be able to log in to any of the machines without having to enter a password. This can be achieved easily using secure shell key authentication. This article describes the methodology to setup quickly ssh key authentication for MPI applications.

= Principle of key authentication=
The principle of key authentication is the following. A key-pair is generated which consists of private key and a public key. To generate the key-pair, a pass-phrase is used which associates the public key to the private key. A copy of the private key is stored on <code>machine1</code> and the public key on <code>machine2</code>. When logging into <code>machine2</code> from <code>machine1</code>, the secure shell program asks for the pass-phrase that matches the public key to the private key. The user is then authentified and fully logged into machine2.

This two stage process is a lot more secure than simply using a password as you also need to "own" the private key. This is a bit like credit card authentication; the system would be a lot less secure if you could just enter your PIN, without actually showing the card in the shop. 

The private key is '''private''' (!). It needs to be known to you and only you. So be careful! You don't really care who can see the public key as only the private key fits with the public key. On [http://en.wikipedia.org/wiki/Secure_Shell wikipedia], the public key is compared to a padlock. You don't care if people see the padlock as long as only you has the key to open it.

In the rest of this article, we assume that the user want to run parallel jobs on a network of machines called <code>master</code>, <code>slave1</code>, <code>slave2</code> etc... <code>master</code> is the logging node, i.e. <code>master</code> is likely to be open to the Internet or to other machines.

= Key generation on master =
To generate a pair of public and private keys, use the following command:
<code><pre>
master $ ssh-keygen -t rsa
</pre></code>

If your network of computers for the parallel computations is on a safe private network and if no sensitive data is stored on the computing nodes, you should consider using a blank pass-phrase. Remember, this is different from a blank password, you still need to own the private key to be able to log in.

However, using a non blank and long pass-phrase is obviously more secure... but it is then slightly more complicated to setup MPI. In particular, the pass-phrase caching program ssh-agent will be required. This is described briefly in the "Advanced features" section at the end of this article.

After creating the key-pair, you will be left with two extra files in the <code>.ssh/</code> folder:
<code><pre>
master $ ls .ssh/
id_rsa  id_rsa.pub
</pre></code>

<code>id_rsa</code> is your private key. It should be visible and writable to you only. <code>id_rsa.pub</code> is the public key. When you are at it, check the properties of the <code>.ssh/</code> folder and <code>id_rsa</code> files. You should have something like:
<code><pre>
master $ ls -la .ssh
drwx------  2 login login 4096 Aug 18 08:32 .
drwx------ 89 login login 4096 Aug 21 09:58 ..
-rw-------  1 login login  744 Mar 30 15:10 id_rsa
-rw-r--r--  1 login login  598 Mar 30 15:10 id_rsa.pub
</pre></code>

If you have used a blank key or if you are likely to use other keys, rename these keys to something a bit more explicit, for instance:
<code><pre>
mv id_rsa id_rsa.mpi
mb id_rsa.pub id_rsa.mpi.pub
</pre></code>

= Copy the public key to the slaves =
The public key now needs to be copied into a file called <code>authorized_keys</code> in the <code>.ssh/</code> folder on each slave. The methods is slightly different if your home directory is shared between the master/slaves or not.

Attention, the first time you log in in slave1, you will be asked to recognise slave1 as a "known host". This is a security feature which aims to avoid man in the middle attacks; i.e. another computer which would pretend to be slave1. "Accepting" the new host builds an entry for slave1 <code>known_hosts</code> file in the <code>.ssh/</code> folder on master. This operation is done only once. If the hardware changes on slave1, secure shell will detect a mismatch between the new machine called slave1 and its entry in the <code>known_hosts</code> and will refuse to connect. The entry in <code>known_hosts</code> will need to be deleted before being able to log in again. 

== Different home directories on each slave ==
If the home directory is separate on each slave, the following commands should do the job nicely for slave1:
<code><pre>
master $ scp .ssh/id_rsa.mpi.pub login@slave1:
master $ ssh login@slave1
login@slave1 password:
slave1 $ mkdir .ssh
slave1 $ chmod 0600 .ssh
slave1 $ cat id_rsa.mpi.pub >> .ssh/authorized_keys
slave1 $ rm -f id_rsa.mpi.pub
slave1 $ exit
master $
</pre></code>

== Shared home directory between the master and the slaves ==
If your home directory is shared you have an identical <code>.ssh</code> folder on all slaves and the master. You can then create <code>authorized_keys</code> from the master:

<code><pre>
master $ cat .ssh/id_rsa.mpi.pub >> .ssh/authorized_keys
</pre></code>

This copying operation is required for all the slaves but do it first on slave1 and test before carrying one.

= Specification of the key to use =
As we have renamed the key, we need to tell the ssh program that we intent to use this non-default key. This is done by creating a file named <code>config</code> in the <code>.ssh</code> folder on master which contains the following lines:
<code><pre>
IdentityFile ~/.ssh/id_rsa
IdentityFile ~/.ssh/id_rsa.mpi
</pre></code>

Now try to log into slave1 from master. You should be asked for the pass-phrase this time:

<code><pre>
master$ ssh login@slave1
Enter passphrase for key '.ssh/id_rsa.mpi':
</pre></code>

Note that if you are using a blank pass-phrase, you will not be asked for a pass-phrase at all and will be logged in automatically. Also, if your home directory is shared between the master/nodes, this might be the first time you log into slave1 and you will need to recognise it as a "known host". See the note at the top of this section.

= Adding a bit of security =
If you know that you will only be using master to start your MPI jobs, you can tighten a bit the configuration by requestion that the key id_rsa.pub is only an authorised key when logging in from master. This is done by editing the <code>authorised_keys</code> on each slave as shown below:

<code><pre>
from="master.full.domain" ssh-rsa adjAWDJSDFJawdihsdlfihlsdfhisAKSUDawdoj ...
</pre></code>

Adding the "<code>from="master.full.domain"</code>" ensures that the public key is only used if the request comes from the master. This is particularly important when the home directory is shared between systems and you are using a blank pass-phrase. Obviously, replace "full.domain" by the real domain name for your machine.

= Copying the authorised key on each host =

Now that slave1 is fully configured, you can copy the configuration to each host. Simply copy the <code>authorized_keys</code> file in <code.ssh/</code> on slave1 to the same location on master, slave2, slave3, etc... Note that if your home directory is shared between master/slaves, you don't have anything to do here.

'''To check: do we really need the authorized_keys entry on master for MPI purposes??'''

Try to log into each slave, accepting it as a "know host" if required. You should be asked for a pass-phrase (or nothing if blank) for each host.

* If you are using a blank pass-phrase and you can log into each slave without user input, that's it! You don't need to do anything else regarding ssh.
* if you are not using a blank pass-phrase, you will need to use a pass-phrase caching program called <code>ssh-agent</code>, see below.

= ssh-agent =
<code>ssh-agent</code> is a program which keeps pass-phrases in memory and re-uses them automatically when required by the secure shell. <code>ssh-agent</code> just outputs some variables and to start it, you need to "evaluate" it:

<code><pre>
master $ eval `ssh-agent`
Agent pid 7760
</pre></code>

Then, to cache a pass-phrase in memory, you need to add an "identity" (or private key) into ssh-agent. This is done by inputting:
<code><pre>
master $ ssh-add ~/.ssh/id_rsa.mpi
Enter passphrase for ~/.ssh/id_rsa.mpi:
Identity added: ~/.ssh/id_rsa.mpi (~/.ssh/id_dsa)
</pre></code>

Now try to login into slave1, slave2 etc... you should not have to enter your pass-phrase anymore.

To remove the pass-phrase, use:
<code><pre>
ssh-add -d ~/.ssh/id_rsa.mpi
</pre></code>

To have <code>ssh-agent</code> started at boot etc, refer to the documentation for your Linux distribution.

= Links =
http://sial.org/howto/openssh/publickey-auth/#s2

http://kimmo.suominen.com/docs/ssh/