DCC ssh connection and remote development
Connecting to dcc
DCC (Duke Compute Cluster)
Link to DCC website for more detailed documentation: https://dcc.duke.edu/
Log in with ssh
The easiest way to log into dcc is through ssh on terminal.
ssh username@dcc-login.oit.duke.edu
The username is basically your NetID.
Once connected, enter you password, and then perform MFA. You will successfully log onto one of the login nodes of dcc.
Generate ssh keys for password-free log in
It is very inconvenient that every login requires typing passwords and doing 2-factor authentication on your phone.
The tip is to set up ssh keys on your local computer and dcc. Once the setup is finished, login is simple and easy without any password input or authentication.
First, go back to your local terminal, and run
ssh-keygen -t rsa
This creates a pair of keys (private + public) using the rsa algorithm. The -t flag specifies the type of algorithm used.
If you want to specify where to save the keys, use
ssh-keygen -t rsa -f /the/file/name/to/save/keys
Now you should have id_rsa and id_rsa.pub in ~/.ssh if you do not specify the filenames.
id_rsa is your private key and id_rsa.pub is the public key which you will upload to the server.
Upload public key to DCC
On terminal, do
less ~/.ssh/id_rsa.pub
to show your public key and then copy-paste to the following duke oit webpage to add your key.
https://idms-web-selfservice.oit.duke.edu/publicSshKeyUpdate
config ssh
Create or modify the ssh connection config file ~/.ssh/config, and add
Host dcc
User YOUR_USER_NAME
IdentityFile ~/.ssh/id_rsa
HostName dcc-login.oit.duke.edu
Make sure to change your username in the entry above.
Once the setting is done, run
ssh dcc
for future logins.
Jupyter kernel and remote development on DCC
In the following, I will show how to start a Jupyter backend kernel and remote development using VS Code. This could also be translated into default terminal or your favorite IDE.
If you'd like to follow the demo here with VS Code, make sure to install VS Code and several extensions (Jupyter and Remote Development) inside VS code before you start.
SSH Log in
Click the green area at the bottom left corner to connect to dcc, or cmd+shift+P and then search [SSH connect to host].
The hosts configured in ~/.ssh/config are listed here. Choose dcc.
Once logged in, you could open a project folder if you wish. It makes file editing and GitHub version control a lot easier.
ctrl+` to open terminal or cmd+J to toggle the bottom panel that contains the terminal.
Run jobs on nodes and Create jupyter backend kernel
The login nodes are for basic file transfer and editing only. Computing tasks should be run on nodes with slurm. To start an interactive job on one of the cosmology node, run
srun --partition cosmology --pty bash -i
The bash terminal prompt indicates the node you're currently on. In this case, it's running on dcc-cosmology-5 node.
Before we initiate jupyter kernels, activate the conda environment you wish to run your jupyter.
jupyter kernelspec list
This command lists the available kernels available. New kernels using your customized environment can be installed by
python -m ipykernel install --user --name your-kernel-name
This adds your customized kernel in ~/.local/share/jupyter/kernels. You could modify kernel.json in the directory to change the name of the kernel or how the kernel starts up.
Next, start a jupyter backend with ip and port on the computing node
jupyter notebook --no-browser --port=7777 --ip=$(hostname -s)
ip has to be the node name (dcc-cosmology-05 in this case) and port could be any number you like. Usually you avoid default ports like 8888 to avoid conflict with other users.
Now, we can connect to this kernel from the login node with the ip, port and token (http://dcc-cosmology-05:7777/?token=5248f6043558ff53c69ef9e7d0d1a49e45e596d00e0b85e5).
Let's leave the terminal there and try to run a python notebook on that kernel.
Switch kernels by clicking the kernel name on the upper right corner. It lists all the python found in the local conda env.
Now, connect to the remote kernel we just created with the link.
You could also access this kernel from your PC browser. First, establish ssh link
ssh -N -f -L 1234:dcc-cosmology-05:7777 dcc
Now port 1234 of your PC listens to dcc-cosmology-05:7777.
Use http://localhost:1234 on your PC to connect to the kernel.