---
# System prepended metadata

title: Setting up the strong lensing pipeline

---

# Setting up the strong lensing pipeline

## Getting accounts
* [DAS-5](https://www.cs.vu.nl/das5/accounts.shtml)
* [Lisa](https://userinfo.surfsara.nl/systems/lisa/account)

## Python setup
Optionally, first install [pyenv](https://github.com/pyenv/pyenv) and [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv). Then install python >=3.8 and create a virtual environment for lensing work:
```
$ pyenv install 3.8.6
$ pyenv virtualenv 3.8.6 lens-3.8.6
$ mkdir ~/lensing
$ cd ~/lensing
$ pyenv local lens-3.8.6
$ pip install wheel
$ pip install -U pip
```
Otherwise, make sure you have python >=3 installed.

## Core dependencies
You will need to make cuda/cuDNN, gcc and cmake available and install the correct version of pytorch.

### DAS-5
Put the following in your `.bashrc`:
```
module load slurm  # for submitting jobs
module load gcc cmake cuda10.1/toolkit/10.1.243 cuDNN/cuda10.1/7.6.4
```
Install pytorch with
```
pip install torch==1.7.0+cu101 torchvision==0.8.1+cu101 torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
```

### Lisa
Log into the GPU island: `ssh username@login-gpu.lisa.surfsara.nl`. Put the following in your `.bashrc`:
```
module load 2020 GCC/9.3.0 CMake/3.16.4-GCCcore-9.3.0 CUDA/11.0.2-GCC-9.3.0
```
Install pytorch with
```
pip install torch==1.7.0+cu110 torchvision==0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
```

## Other dependencies

### External
Clone these and install with `pip install`:
* [torchinterp1d](https://github.com/aliutkus/torchinterp1d)

### Our repositories
Clone these and install these with `pip install -e` so you don't have to reinstall them whenever you edit the code:
* [clipppy](https://github.com/kosiokarchev/clipppy/): wrapper around pyro. This lets you define a pyro model using a yaml file, load it, fit it and more.
* [pyrofit_core](https://github.com/cweniger/pyrofit_core): other utilities. We will hopefully merge this with clipppy and pyrofit_lensing soon.
* [pyrofit_lensing](https://github.com/cweniger/pyrofit_lensing): this contains all the lensing-specific code, data and notebooks. Use the `refactor` branch. The installation does not install everything required to run the notebooks and scripts in experiments. To install their dependencies, run `pip install -r requirements.txt`.

## Running the code
Now everything is set up! The jupyter notebook `experiments/03-neurips_nb/ac-01-mock_fit_example.ipynb` shows how to generate mock data and fit it with a GP source.

## Possible issues
* The most annoying dependency is keops. You can test whether your installation is working by following [these instructions](https://www.kernel-operations.io/keops/python/installation.html#testing-your-installation).
* The first time you run the lensing code, keops has to compile a bunch of cuda code. This takes a while, so if you're running on a Lisa login node your job might get killed before compilation finishes. In this case you may get weird errors when you rerun the code. These can usually be fixed by clearing the keops cache and running the code on a compute node.

## Tips
* On Lisa, you can find idle GPU nodes with `sinfo | grep gpu | grep idle`. Use the partition name (first column) to decide which partition to submit jobs to (indicated with the `-p` flag for `sbatch`).
* Run [this script](https://gist.github.com/adam-coogan/d3adec4c93c0b477f157cb14914182db) to start an interactive shell on a DAS-5 GPU node. [Here is an equivalent script for Lisa](https://gist.github.com/adam-coogan/bb59b342b6b3f279afe58015ae77d731).
* It is really useful to be able to run jupyter notebooks on compute nodes that have GPUs and connect to them through an SSH tunnel. You can do that with [this script](https://gist.github.com/adam-coogan/1d46315cd4c4999c407b48b72840058d).
* **To do this on Lisa, you will need to email the system administrators to request they enable port forwarding.** [Here is the job script](https://gist.github.com/adam-coogan/ee1d6cb030751719abd78a69cbcd0f59). You may need to use `sinfo` to find a different partition with idle nodes to submit to (line 3). You will also need to change `login3` in line 26 to the name of your login node. You can find this by looking at your prompt. For example, if my prompt was `amcoogan@login-gpu1:~$` I would change `login3` to `login-gpu1`.

## To do
- [ ] Lisa cuda setup
- [x] Lisa GPU island instructions
- [ ] swyft