DCC ssh connection and remote development

Owned by Dr Chien-Hao Lin
Last updated: Sept 01, 2022
4 min read

Connecting to dcc

DCC (Duke Compute Cluster)

Link to DCC website for more detailed documentation: https://dcc.duke.edu/

Log in with ssh

The easiest way to log into dcc is through ssh on terminal. 

ssh username@dcc-login.oit.duke.edu

The username is basically your NetID.

Once connected, enter you password, and then perform MFA. You will successfully log onto one of the login nodes of dcc.


Generate ssh keys for password-free log in

It is very inconvenient that every login requires typing passwords and doing 2-factor authentication on your phone.

The tip is to set up ssh keys on your local computer and dcc. Once the setup is finished, login is simple and easy without any password input or authentication.


First, go back to your local terminal, and run

ssh-keygen -t rsa

This creates a pair of keys (private + public) using the rsa algorithm. The -t flag specifies the type of algorithm used.

If you want to specify where to save the keys, use

ssh-keygen -t rsa -f /the/file/name/to/save/keys

Now you should have id_rsa and id_rsa.pub in ~/.ssh if you do not specify the filenames. 

id_rsa is your private key and id_rsa.pub is the public key which you will upload to the server.


Upload public key to DCC

On terminal, do

less ~/.ssh/id_rsa.pub

to show your public key and then copy-paste to the following duke oit webpage to add your key.

https://idms-web-selfservice.oit.duke.edu/publicSshKeyUpdate


config ssh

Create or modify the ssh connection config file ~/.ssh/config, and add

Host dcc
         User YOUR_USER_NAME
         IdentityFile ~/.ssh/id_rsa
         HostName dcc-login.oit.duke.edu

Make sure to change your username in the entry above. 


Once the setting is done, run

ssh dcc

for future logins.


Jupyter kernel and remote development on DCC

In the following, I will show how to start a Jupyter backend kernel and remote development using VS Code. This could also be translated into default terminal or your favorite IDE.

If you'd like to follow the demo here with VS Code, make sure to install VS Code and several extensions (Jupyter and Remote Development) inside VS code before you start.


SSH Log in

Click the green area at the bottom left corner to connect to dcc, or cmd+shift+P and then search [SSH connect to host].

The hosts configured in ~/.ssh/config are listed here. Choose dcc.


Once logged in, you could open a project folder if you wish. It makes file editing and GitHub version control a lot easier. 

ctrl+` to open terminal or cmd+J to toggle the bottom panel that contains the terminal.


Run jobs on nodes and Create jupyter backend kernel

The login nodes are for basic file transfer and editing only. Computing tasks should be run on nodes with slurm. To start an interactive job on one of the cosmology node, run

srun --partition cosmology --pty bash -i

The bash terminal prompt indicates the node you're currently on. In this case, it's running on dcc-cosmology-5 node.

Before we initiate jupyter kernels, activate the conda environment you wish to run your jupyter.

jupyter kernelspec list

This command lists the available kernels available. New kernels using your customized environment can be installed by


python -m ipykernel install --user --name your-kernel-name

This adds your customized kernel in ~/.local/share/jupyter/kernels. You could modify kernel.json in the directory to change the name of the kernel or how the kernel starts up.


Next, start a jupyter backend with ip and port on the computing node

jupyter notebook --no-browser --port=7777 --ip=$(hostname -s)

ip has to be the node name (dcc-cosmology-05 in this case) and port could be any number you like. Usually you avoid default ports like 8888 to avoid conflict with other users.

Now, we can connect to this kernel from the login node with the ip, port and token (http://dcc-cosmology-05:7777/?token=5248f6043558ff53c69ef9e7d0d1a49e45e596d00e0b85e5).

Let's leave the terminal there and try to run a python notebook on that kernel.


Switch kernels by clicking the kernel name on the upper right corner. It lists all the python found in the local conda env.

Now, connect to the remote kernel we just created with the link.


You could also access this kernel from your PC browser. First, establish ssh link

ssh -N -f -L 1234:dcc-cosmology-05:7777 dcc

Now port 1234 of your PC listens to dcc-cosmology-05:7777. 

Use http://localhost:1234 on your PC to connect to the kernel.