Originally posted on the DAIS Lab blog. An updated and more sophiscated solution using Jupyter Lab with multi-user accounts can be found in this Biofoundry GitHub Repo.

Compute Canada Logo

Compute Canada provides the Canadian research community access to advanced computing resources. Jupyter Notebooks are a great way to create and share reproducible research. In this tutorial, we will show you how to set up a Jupyter Notebook server on the Compute Canada cloud system that you can access remotely.

Prerequisites

  1. You’ll need to apply for a Compute Canada Database (CCDB) account and get your PI to approve it.

  2. Using your CCDB username, you’ll need to apply for a separate Compute Canada Cloud account.

  3. Once your Cloud account has been approved, you should receive an email and Compute Canada will provide access to either the West or the East cloud (but not both).

I spent some time trying to login to the East and West cloud with my CCDB account without applying for a Cloud account, which obviously didn’t work. The need to register for a separate Cloud account wasn’t very clear to me because the Compute Canada website had the registration link and instructions hidden in the middle of their Cloud service page.

Step 1 — Setting up your Cloud instance

Login to either the West or the East cloud. Once you’re in, you should see a sidebar on the left with a section called Compute. We will set up 2 things there, a volume and an instance.

Instances are virtual machines and volumes are disk space that can be attached to instances. Data on instances are destroyed when you terminate an instance, but you can store persistent data on volumes.

Step 1.1 — Security

We first need to open up some ports to access the Jupyter Notebook server remotely. In the Access & Security tab, click on Manage Rules for the default security group. Click Add Rule and add SSH, HTTP, HTTPS and a Custom TCP Rule for Port 9999 with direction ingress.

Step 1.2 — Volume

In the Volumes tab, click on the Create Volume button and fill up the name and description. Use Image as the Volume Source and Ubuntu-16.04 as the image source. For the size, anything above 5 GiB should be sufficient. Use nova for the availability zone.

Step 1.3 — Instance

In the Instances tab, click on Launch Instance. Give it a name and use nova for the availability zone. In Instance Boot Source select Boot from volume, then choose the volume that you’ve just created.

You can select the size of your new virtual machine as pre-defined flavours. You have 2 main choices, a p (persistent) flavour and a c (compute) flavour. Briefly, c flavours are intended for running shorter, intensive computations and for development and testing purposes while p flavours are meant for running longer computations. You can find more information in the Compute Canada wiki.

DO NOT CHECK the Delete Volume on Instance Delete box or you will lose your data when you terminate the instance.

Step 1.4 — Creating an SSH key pair

Under Access & Security, select the default security group.

Click on the + button in the Key Pair form to create a new key pair. We will follow the instructions in the new popup window to import a new key pair. Give your key pair a name first, i.e. Dell Computer.

If you’re on Linux or Mac, open up your terminal (Ctrl+Alt+T for Ubuntu) the run the following commands:

Use ssh-keygen to create a key pair named canadacloud (you can name it anything you like). It will ask you for a passphrase, just leave it blank if you want.

ssh-keygen -t rsa -f canadacloud

Grab and copy the public key using this command

cat canadacloud.pub

Then paste the contents back in the popup window in the Public Key box then click the Import button.

Once your done, click launch and wait for your instance to boot up.

Step 1.5 — Associating a Floating IP

To connect to our new instance, we need to assign a public IP to it. In the Access & Security tab, select Floating IPs and click on Allocate IP to Project. Select the default pool and click on the Allocate IP button.

Click on the Associate button for the floating IP address and then select your instance in Port to be associated.

Step 2 — Installing Anaconda and Jupyter

In your terminal, login to the instance with the following command, replacing xxx.xxx.xxx.xxx with your floating IP address and computecanada with your key name.

ssh -i computecanada ubuntu@xxx.xxx.xxx.xxx

If you’re having trouble logging in, try restarting your instance using a soft (or hard) reboot from the dropdown menu in the Instances page.

First, update your system with the following commands:


sudo apt-get update
sudo apt-get upgrade

Once that’s done, download and install Anaconda:


wget https://repo.continuum.io/archive/Anaconda3-4.4.0-Linux-x86_64.sh
bash Anaconda3-4.4.0-Linux-x86_64.sh 

Step 3 — Configuring Jupyter Notebook

Once Anaconda is installed, you should be able to launch Jupyter with this command:

jupyter notebook

Press Ctrl+C to close the server.

We will follow the instructions here to set up Jupyter as a public server and secure it with HTTPS and a password.

Step 3.1 — Generate a Jupyter configuration file

First, create a configuration file

jupyter notebook --generate-config

and generate a hashed password

jupyter notebook password

Step 3.2 — Create an SSL certificate

openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mykey.key -out mycert.pem

Step 3.3 — Edit the configuration file

First copy your hashed password (the string starting with sha1) with the following command and save it somewhere safe:

cat ~/.jupyter/jupyter_notebook_config.json

Open the Jupyter config file in nano:

nano ~/.jupyter/jupyter_notebook_config.py

Then add the following lines to the top of the file. Replace YourPassword with the password that you copied.


c.NotebookApp.certfile = u'/home/ubuntu/mycert.pem'
c.NotebookApp.keyfile = u'/home/ubuntu/mykey.key'
c.NotebookApp.ip = '*'
c.NotebookApp.password = u'YourPassword'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 9999

To close and save the file, press Ctrl+X and select Yes.

Step 4 — Testing your new Notebook

Start the Jupyter notebook:

jupyter notebook

Point your browser to the following address, replace xxx.xxx.xxx.xxx with your floating IP:

https://xxx.xxx.xxx.xxx:9999/

If everything works, you should see a login screen and you will be able to enter your password and login to your Notebook!

Step 5 — Persistent Notebooks

Closing your SSH connection will close the notebook server.

To create a persistent notebook, we will use the screen terminal multiplexer. First, close the server with Ctrl+C then run:

screen

Then in the new window, run the Jupyter notebook:

jupyter notebook

Now if you close the terminal, the Notebook server will still be running.

Step 6 — Conclusion

Congratulations, you should now have a Jupyter Notebook running on the Compute Canada Cloud that you can access remotely. For a quick tour on using Jupyter notebooks, visit this link.

Notes

If you get an SSL security error in your browser, it’s because you’re using a self-signed certificate. To get rid of the error, you’ll need to use a custom domain and use a Let’s Encrypt certificate instead of a self-signed certificate:


sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install certbot
sudo certbot certonly --standalone -d your.domain.name

Then replace the certificate and key in your Jupyter configuration file with the Let’s Encrypt certificates


c.NotebookApp.certfile = u'/etc/letsencrypt/live/your.domain.name/fullchain.pem'
c.NotebookApp.keyfile = u'/etc/letsencrypt/live/your.domain.name/privkey.pem'