 HackMD
Setting up remote coding for Harvard HMS O2 and HTEM nodes (non-GUI)
We will be running VS Code locally and using SSH under the hood to connect to a VS Code instance running remotely. This gives us access to data and compute resources on the remote server, while giving us a fully featured coding environment.
1. Download, install, and setup the Harvard HMS Virtual Private Network.
You should connect to https://secure.med.harvard.edu/ for the VPN.
2. Connect to the VPN (unless you are on HMS WiFi) to make sure it works.
3. Install a supported SSH client as specified here.
4. Create an SSH Key pair and add the Public Key to your Remote Server
Open a local terminal and run ssh-keygen -t rsa
When prompted enter a blank passphrase (just hit Enter).
Then run the following, replacing user and hostname with your username on the remote server, and the path to the server (such as catmaid2.hms.harvard.edu or 10.117.28.249):
cat ~/.ssh/id_rsa.pub | ssh user@hostname 'cat >> .ssh/authorized_keys'
You should be prompted for your server password.
Try sshing in to verify it works, using the following command (replacing user and hostname):
ssh user@hostname
5. Download and install Microsoft Visual Studio Code (VS Code) on your local computer.
6. Install the Remote Development Extension Pack for VS Code.
7. Connect to a remote VS Code session
In VS Code with the Remote Development Extension Pack installed, you will see a new Status bar item at the bottom far left of the window (the >< looking button). Clicking on the item will bring up the Remote - SSH commands. Click Remote-SSH: Connect to Host...
Fill in user@hostname using your username and the server path, as in step #4, and hit Enter.
It may take a little bit to connect, especially the first time. But then you will have all the functionality of VS Code, such as terminal, AI-assisted coding environment, live python scripts, built in Jupyter notebook, virtual environment management, and Source Control (i.e. Git/GitHub), assuming you have the necessary extensions installed ;)
Once you've connected to a server once, you should be able to connect again by selecting it from the Remote Explorer tab:
Level 2: Control your environment
For those not well acquianted with managing virtual python environments, we'll do a quick demo of using Anaconda (or Miniconda actually) to make and maintain a coding environment for us. All of this should be done in a terminal session on the remote server.
1. Make sure Anaconda/Miniconda is installed.
See if conda is already installed by running conda in the terminal and seeing if it gives you an error or explains the command. If it explains the command, it is installed - continue to step #2.
If it is not already installed, then:
Change to the tmp directory then download the installer by running
Where <url> should be replaced with the link to the correct installer for your server found here. Most likely this will be the Linux 64-bit version, and you can find additional instructions here.
It is recommended that you verify the download before running the installer. For Linux this would consist of running sha256sum <filename> where <filename> is the path to the downloaded install script (example: Miniconda3-latest-Linux-x86_64.sh).
Then run bash <filename> and follow the prompts, choosing the default options (you can change them later).
To make the changes take effect, close and then re-open your terminal window.
Test your installation by running conda list in a new terminal window. If a list of installed packages appears it has been installed correctly.
2. Create your environment.
Pick an appropriate or otherwise useful name for your environment. We'll call ours next_level because fun is important for morale. Replacing next_level with your name of choice, run the following:
conda create -n next_level python=3.9
This creates a new environment called next_level using python 3.9. This tends to be a pretty safe choice in terms of compatability, but feel free to replace with older or newer versions as you need.
3. Activate your environment.
conda activate next_level
4. Control your environment.
Let's install some example packages with pip:
pip install numpy pandas networkx python-catmaid navis
Basically add any packages available on pypi.org that you would like to install. Additionally, you can add pip-installable GitHub repos with git+<git-url>.
For instance, to install the full-refactor branch of raygun, run:
pip install git+https://github.com/htem/raygun@full_refactor
If you need to install new dependencies that would conflict with your current environment, simply deactivate it (conda deactivate) and repeat from step #2.
Level 3: Remote Python Coding with VS Code Interactive Windows in Conda Environments
Now we'll put things together and do some simple coding. We'll save true 'development' including version control (via GitHub) for the next 'Next Level' ;)
1. Connect to a VSCode remote session.
2. Select and activate an environment.
By default, the Python extension looks for and uses the first Python interpreter it finds in the system path. To select a specific environment, use the Python: Select Interpreter command from the Command Palette (Ctrl+Shift+P).
The Python: Select Interpreter command displays a list of available global environments, conda environments, and virtual environments. Pick the one you made.
The selected interpreter version will show on the right side of the Status Bar.
The Status Bar also reflects when no interpreter is selected.
In either case, clicking this area of the Status Bar is a convenient shortcut for the Python: Select Interpreter command.
Why do we do this? The Python extension uses the selected environment for running Python code, providing language services (auto-complete, syntax checking, linting, formatting, etc.) when you have a .py file open in the editor, opening a terminal, and, if we're lucky, running Interactive Windows.
3. Code in an Python Interactive Window.
The Python Interactive window can be used as a standalone console with arbitrary code run using the interpreter and virtual environment of your choice. To use the window as a console, open it with the Jupyter: Create Interactive Window command from the Command Palette (Ctrl+Shift+P). You can then type in code, using Enter to go to a new line and Shift+Enter to run the code.
To change the environment you're coding in, simply click the Kernel selector in the upper right corner of the Interactive Window:
If you make something you're pleased with, make sure you save it, or are developing as we will discuss in the next section.
Level 4: Remote Development with all of the above + Version Control (GitHub)
This section will get you to a place where you should be able to comfortably develop python code remotely, saving different versions as you go to GitHub for safekeeping. We will use everything in the past sections + GitHub. Proper package development will be dealt with in the next 'Next Level' ;)
1. Setup your GitHub account.
If you haven't already, sign up for an account at github.com. I strongly recommend using a '.edu' email address if you have it, as it makes it easier to get certain perks targeted at students and the academic research community.
I also recommend configuring 'settings sync' for VSCode by logging in with your GitHub account.
2. Setup a new repository.
While we're at github.com, let's go ahead and setup a new repository. In the upper-right corner of any page, use the drop-down menu, and select New repository.
Type a short, memorable name for your repository. For example, "hello-world".
Optionally, add a description of your repository.
Choose a repository visibility.
Select Initialize this repository with a README.
Click Create repository.
3. Connect to a VS Code remote session.
4. Connect VS Code and GitHub.
First, install the GitHub Pull Requests and Issues extension. Once you've installed the GitHub Pull Requests and Issues extension, you'll need to sign in. Follow the prompts to authenticate with GitHub in the browser and return to VS Code.
If you are not redirected to VS Code, you can add your authorization token manually. In the browser window, you will receive your authorization token. Copy the token, and switch back to VS Code. Select Signing in to github.com… in the Status bar, paste the token, and hit Enter.
Generate an SSH key, add it to the ssh-agent and add it to your GitHub.
NOTE: Should you run into mysterious trouble pushing updates to the remote repo, make sure your local repo is setup to use SSH instead of HTTPS authentification
(Comment if this was confusing and I should expand the details here :)
5. Clone the repo you made in step 2.
Run the following in a terminal, replacing <username> with your GitHub username and <repository name> with the name of your repository. Do this in the directory where you wish the repo to reside.
git clone git@github.com:<username>/<repository name>.git
Open the folder containing the repo in the File Explorer (icon on the top left).
Alternatively:
You can search for and clone a repository from GitHub using the Git: Clone command in the Command Palette (Ctrl+Shift+P) or by using the Clone Repository button in the Source Control view (available when you have no folder open).
From the GitHub repository dropdown you can filter and pick the repository you want to clone locally.
You will likely be prompted to authenticate with GitHub.
Follow the steps to sign in or, if you run into trouble, try consulting the GitHub page about authentification.
6. Develop.
Start a new python file (e.g. hello.py) and add #%% as the first line. After this, you will be able to run code blocks (delineated by #%%'s) by hitting Ctrl+Enter. They will automatically be run in a new Interactive Python Window, as we used in the last level.
Between the interactive window, saving to your python file, and committing changes to GitHub, you now have a fully fledged remote development environment. Congratulations!
