# Server configuration for imaging-neuroscience.org

**The imaging neuroscience server was shut down on 26th September 2024.**

Both https://imaging-neuroscience.org and https://janeway.imaging-neuroscience.org run on a single Amazon EC2 instance, provisioned through the FSL AWS account:

- Name: Imaging Neuroscience Janeway Web Server
- Type: t3.xlarge
- Zone: eu-west-2 (London)
- Elastic IP: 13.41.96.240
- OS: Ubuntu 22.04
- Username: `ubuntu`

The instance is SSH-accessible with a private key available to both Taylor and Paul.

The imaging-neuroscience.org and janeway.imaging-neuroscience.org domains and associated SSL certificates are managed by Steve through an ionos.co.uk account. Both domains point to 13.41.96.240

Until 1st August 2023, outgoing email was provided by a https://postmarkapp.com/ account. Paul and Steve are both admins on this account. On the 1st of August, we switched over to using an IONOS email account for outgoing email.

## Pre-requisites

- SSH private key for login
- `imaging-neuroscience.org` SSL certificate and key file
- `ubuntu` UNIX account password
- `janeway` UNIX account password
- `janeway` PostgreSQL account password
- PostgreSQL database dump from a testing server, to initialise the new database
- IONOS SMTP settings for outgoing emails
- Django `SECRET_KEY` (used for password salting/hashing)

## System installation and configuration

# Logged in as ubuntu
cd ~

### Paul's personal config files

wget https://raw.githubusercontent.com/pauldmccarthy/dotfiles/master/.tmux.conf
wget https://raw.githubusercontent.com/pauldmccarthy/dotfiles/master/.gitconfig

### System dependencies

# Janeway v1.4.4 uses a version of Django
# which requires python 3.8, but Ubuntu
# 22.04 uses python 3.10 by default.
sudo add-apt-repository ppa:deadsnakes/ppa -y 
sudo apt update

# Apache, postgresql, python 3.8, compiler 
# toolchain and some dev libs for compiling
# mod_wsgi and other janeway dependencies.
sudo apt install -y apache2 apache2-dev emacs postgresql libxml2-dev \
                    libxslt1-dev python3-dev zlib1g-dev lib32z1-dev libffi-dev \
                    libssl-dev libjpeg-dev libmysqlclient-dev python3.8 \
                    python3.8-dev python3.8-venv python3-pip \
                    build-essential libpq-dev

### Firewall

Restrict access to SSH, HTTP and HTTPS. Also ensure that inbound traffic is allowed on these ports (22, 80, 443) in the AWS security group rules.

sudo ufw allow OpenSSH
sudo ufw allow "Apache Full"
sudo ufw enable

Install SSL certificate/key - Steve gets these files from ionos.co.uk.

# enable SSL in apache
sudo a2enmod ssl
sudo systemctl restart apache2

sudo mv <keyfile>  /etc/ssl/private/imaging-neuroscience.org.key
sudo mv <certfile> /etc/ssl/certs/imaging-neuroscience.org.cer

sudo chown root:ssl-cert /etc/ssl/private/imaging-neuroscience.org.key
sudo chown root:root     /etc/ssl/certs/imaging-neuroscience.org.cer

sudo chmod 640 /etc/ssl/private/imaging-neuroscience.org.key
sudo chmod 644 /etc/ssl/certs/imaging-neuroscience.org.cer

### Make a swap file.

The instance has 16GB RAM, but a little swap is good, right?

sudo dd if=/dev/zero of=/swapfile bs=1024 count=4194304
sudo mkswap /swapfile
sudo chmod 600 /swapfile
sudo swapon /swapfile
sudo emacs /etc/fstab
  -> /swapfile    none    swap    sw    0    0

### PostgreSQL

We create a PostgreSQL account and database for use by the Janeway system.

# Create a db user for Janeway
#   username:    janeway
#   permissions: createdb:y, everything else: n
#   password:    stored in Paul's password manager,
#                and also in the janeway settings.py
sudo -u postgres createuser --interactive --pwprompt

# use md5 pw auth, so the db account is distinct
# from the UNIX user account ("peer" tells 
# postgresql to use the UNIX account credentials)
# Add a line to this file:
#
#   local all janeway md5
sudo emacs /etc/postgresql/14/main/pg_hba.conf

sudo systemctl restart postgresql

# Create a db called janeway, owned by user janeway
sudo -u postgres createdb --username=janeway janeway

### User account for Janeway

We use a separate account which does not have `sudo` privileges. Not really necessary, as the site will be executed by `www-data`, but it seems a bit nicer.

# Password stored in Paul's password manager
sudo useradd -m janeway
sudo chsh janeway -s /bin/bash
sudo passwd janeway

# give apache access to /var/www/*imaging-neuroscience.org/
sudo adduser www-data janeway

## Janeway installation

We are using a [fork of Janeway](https://github.com/pauldmccarthy/janeway/tree/mnt/imgneuro) which is based on [Janeway v1.4.4](https://github.com/BirkbeckCTP/janeway/tree/v1.4.4), with a few small tweaks and customisations.

sudo su janeway

cd ~
python3.8 -m venv  janeway.env
. ./janeway.env/bin/activate

pip install --upgrade pip wheel
# https://github.com/pypa/setuptools/issues/3898
pip install "setuptools==57.*"

git clone https://github.com/pauldmccarthy/janeway/
cd janeway
git checkout mnt/imgneuro

# We install mod_wsgi via pip, because the 
# Ubuntu mod_wsgi package is compiled against 
# python 3.10, but we are using python 3.8. 
pip install mod_wsgi
pip install -r requirements.txt

### Janeway configuration

Create `src/core/settings.py` and populate it as follows:

# SECURITY WARNING: keep the secret key used in production secret!
#
# Generate your own django secret key by running:
# 
#   python3.8 -c 'import secrets; print(secrets.token_hex(100))'
SECRET_KEY = '<insert-key-kere>'

# This is the default redirect if no other sites are found.
DEFAULT_HOST = 'https://janeway.imaging-neuroscience.org'
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
LOGIN_REDIRECT_URL = '/user/profile/'

# Default Langague
LANGUAGE_CODE = 'en'

URL_CONFIG = 'domain'  # path or domain

EMAIL_HOST          = 'smtp.postmarkapp.com'
EMAIL_PORT          = 587
EMAIL_HOST_USER     = '<postmark-user-token>'
EMAIL_HOST_PASSWORD = '<postmark-user-token>'
EMAIL_USE_TLS       = True
EMAIL_USE_SSL       = False
EMAIL_HEADERS       = {'X-PM-Message-Stream' : 'outbound'}

DATABASES = {
    'default': {
        'ENGINE'   : 'django.db.backends.postgresql',
        'NAME'     : 'janeway',
        'USER'     : 'janeway',
        'PASSWORD' : '<janeway-db-password>',
        'HOST'     : 'localhost',
        'PORT'     : '5432',
    }
}

# CATCHA_TYPE should be either 'simple_math', 'recaptcha' or 'hcaptcha' to enable captcha
#fields, otherwise disabled
CAPTCHA_TYPE = 'simple_math'

# ORCID Settings
ENABLE_ORCID = False
ORCID_API_URL = 'http://pub.orcid.org/v1.2_rc7/'
ORCID_URL = 'https://orcid.org/oauth/authorize'
ORCID_TOKEN_URL = 'https://pub.orcid.org/oauth/token'
ORCID_CLIENT_SECRET = ''
ORCID_CLIENT_ID = ''

# OIDC Settings
ENABLE_OIDC = False
OIDC_SERVICE_NAME = 'OIDC Service Name'
OIDC_RP_CLIENT_ID = ''
OIDC_RP_CLIENT_SECRET = ''
OIDC_RP_SIGN_ALGO = 'RS256'
OIDC_OP_AUTHORIZATION_ENDPOINT = ""
OIDC_OP_TOKEN_ENDPOINT = ""
OIDC_OP_USER_ENDPOINT = ""
OIDC_OP_JWKS_ENDPOINT = ''

ENABLE_FULL_TEXT_SEARCH = False # Read the docs before enabling full text

# Model used for indexing full text files
CORE_FILETEXT_MODEL = "core.FileText"  # Use "core.PGFileText" for Postgres

### Janeway initialisation and DB migration

Run ```python src/manage.py install_janeway```, and enter the following information (their values will actually be overwritten later, so they don't really matter here):

- press name: `MIT Press`
- press domain: `imaging-neuroscience.org`
- journal code: `imaging-neuroscience`
- journal domain: `janeway.imaging-neuroscience.org`
- journal name: `Imaging Neuroscience`
- journal desc: `an open-access non-profit journal of brain imaging`

Install the `material` theme:

cp -r src/static/material src/collected-static

On the test server, take copies of the following folders, and copy them into the same location on the new server:

- `src/media` - Custom journal images/logos/etc
- `src/files/articles/` - actual article files, if you want to import submitted articles from the test server
- `src/files/journals/` - More custom journal images

Populate the site and database with a DB dump from a test instance (this includes all of the customisations that Steve has made). On the test/source server, run `pg_dump --no-owner -U <dbuser> <dbname> >> <db_dump_file>`. Then on the produciton/destination server, run:

sudo -u postgres psql
# drop database janeway;
# create database janeway with owner janeway;
# grant create on database janeway to janeway;
 
# this can be run as the janeway user
psql < <db_dump_file>

Update the DB domain entries (as the test server was probably set up with different domains):

psql
# update journal_journal set domain='janeway.imaging-neuroscience.org' where id=1;
# update press_press     set domain='imaging-neuroscience.org'         where id=1;

Re-run django migrations to ensure that the restored DB schema is up to date:

sudo su janeway
cd ~/janeway/
. ~/janeway.env/bin/activate
python src/manage.py migrate

## Static landing page: https://imaging-neuroscience.org

This website just comprises a few static files, obtained from Steve. Edit `etc/apache2/sites-available/imaging-neuroscience.org.conf`, and insert the following:

<VirtualHost *:443>
        ServerName imaging-neuroscience.org
        ServerAlias www.imaging-neuroscience.org
        DocumentRoot /var/www/imaging-neuroscience.org/

        SSLEngine on
        SSLCertificateFile /etc/ssl/certs/imaging-neuroscience.org.cer
        SSLCertificateKeyFile /etc/ssl/private/imaging-neuroscience.org.key
</VirtualHost>

<VirtualHost *:80>
        ServerName imaging-neuroscience.org
        ServerAlias www.imaging-neuroscience.org
        Redirect / https://imaging-neuroscience.org/
        # DocumentRoot /var/www/imaging-neuroscience.org/
</VirtualHost>

Then install the website files, and turn the site on:

sudo mkdir /var/www/imaging-neuroscience.org/
sudo mv Announcement.pdf index.html ImgNeuro.png /var/www/imaging-neuroscience.org/

# check that the config is valid
sudo apache2ctl configtest

# turn on the landing page
sudo a2ensite imaging-neuroscience.org
sudo a2dissite 000-default
sudo systemctl reload apache2 

## Janeway journal submission system: https://janeway.imaging-neuroscience.org

Edit `etc/apache2/sites-available/janeway.imaging-neuroscience.org.conf`, and insert the following. The `LodModule` line comes from running `mod_wsgi-express module-config`:

<VirtualHost *:443>
        ServerName janeway.imaging-neuroscience.org
        ServerAlias www.janeway.imaging-neuroscience.org

        Alias /static/ /var/www/janeway.imaging-neuroscience.org/janeway/src/collected-static/
        <Directory /var/www/janeway.imaging-neuroscience.org/janeway/src/collected-static>
        Require all granted
        </Directory>

        Alias /media/ /var/www/janeway.imaging-neuroscience.org/janeway/src/media/
        <Directory /var/www/janeway.imaging-neuroscience.org/janeway/src/media>
        Require all granted
        </Directory>

        <Directory /var/www/janeway.imaging-neuroscience.org/janeway/src/core/>
        <Files wsgi.py>
        Require all granted
        </Files>
        </Directory>

        LoadModule wsgi_module "/var/www/janeway.imaging-neuroscience.org/janeway.env/lib/python3.8/site-packages/mod_wsgi/server/mod_wsgi-py38.cpython-38-x86_64-linux-gnu.so"
        WSGIApplicationGroup %{GLOBAL}
        WSGIDaemonProcess janeway python-home=/var/www/janeway.imaging-neuroscience.org/janeway.env/ python-path=/var/www/janeway.imaging-neuroscience.org/janeway/src
        WSGIProcessGroup janeway
        WSGIScriptAlias / /var/www/janeway.imaging-neuroscience.org/janeway/src/core/wsgi.py

        SSLEngine on
        SSLCertificateFile /etc/ssl/certs/imaging-neuroscience.org.cer
        SSLCertificateKeyFile /etc/ssl/private/imaging-neuroscience.org.key
</VirtualHost>

<VirtualHost *:80>
        ServerName janeway.imaging-neuroscience.org
        ServerAlias www.janeway.imaging-neuroscience.org
        Redirect / https://janeway.imaging-neuroscience.org/
</VirtualHost>

Then set up the website and turn it on:

sudo mkdir                 /var/www/janeway.imaging-neuroscience.org
sudo chown janeway:janeway /var/www/janeway.imaging-neuroscience.org
sudo chmod -R 755          /var/www/janeway.imaging-neuroscience.org
ln -s ~/janeway            /var/www/janeway.imaging-neuroscience.org/janeway
ln -s ~/janeway.env        /var/www/janeway.imaging-neuroscience.org/janeway.env

sudo a2ensite janeway.imaging-neuroscience.org
sudo apache2ctl configtest
sudo systemctl reload apache2

## Backups

After the server has been configured as outlined above, we modify the live instance to allow backups of the Janeway database, and all files submitted through Janeway.

The Janeway files are moved to a separate EBS volume, and daily snapshots of the PostgreSQL database are also saved to that volume. The volume is called `imaging-neuroscience-janeway-data-store`, has ID `vol-0ba9f56a7b94cefae`, and is in availability zone `eu-west-2c`.

> This change was performed after the server went live due to time constraints. It would obviously make more sense to install Janeway onto a separate EBS volume from the outset.

### Move Janeway to a separate EBS volume

- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-attaching-volume.html
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
- https://www.microhost.com/docs/tutorial/how-to-move-a-postgresql-data-directory-to-a-new-location-on-ubuntu-22-04/

Create an EBS volume and attach it to the instance. Make sure that the volume is created in the same Availability Zone as the instance (e.g. `eu-west-2c`). Then mount the volume in the running instance:

# The volume device will be called /dev/xvdf or /dev/nvme1n1 
# or something - find out by running sudo lsblk -f
sudo mkfs -t ext4 /dev/xvdf
sudo mkdir /data/
sudo chmod 777 /data/
sudo mount /dev/xvdf /data/

Add the volume to `/etc/fstab` so it is mounted at boot:

UUID=<find by running sudo blkid>  /data  ext4  defaults,nofail  0  2

Stop Apache:

sudo systemctl stop apache2

Move the Janeway directory to the new volume:

sudo su janeway
cd ~
mv janeway /data/
ln -s /data/janeway ./janeway
sudo chmod -R 775 /data/janeway

Restart Apache:

sudo systemctl stop apache2

> I had originally also planned to relocate the PostgreSQL database files to the EBS volume as well, but due to time constraints the system was made public before I was ready, and I was too afraid to do it on the running instance. But it would be done with something like the following:
>
> ```bash
> # double check config file / data directory
> sudo -u postgres psql
> # -> show config_file;
> # -> sheow data_directory;
> sudo systemctl stop apache2
> sudo systemctl stop postgresql
> sudo mkdir /data/pgdata
> sudo chown postgres:postgres /data/pgdata
> sudo chmod 700 /data/pgdata
> sudo rsync -av /var/lib/postgresql/14/main/ /data/pgdata
> sudo emacs /etc/postgresql/14/main/postgresql.conf
> # -> data_directory = '/data/pgdata'
> sudo systemctl start apache2
> sudo systemctl start postgresql
> ```

### Set up daily snapshots for PostgreSQL

We do this in case the database is somehow corrupted or compromised.

Create a snapshot script at `/data/pgdata_snapshot` containing the following:

#!/usr/bin/env bash
# This script is added to the postgres user crontab to run daily
snapshotdir=/data/pgdata_snapshots
datestamp=$(date +"%Y%m%d-%H%M%S")
filename=${snapshotdir}/${datestamp}.db

mkdir -p ${snapshotdir}
pg_dump -U postgres janeway | gzip > ${filename}

Make the script executable, and add it to the `postgres` user crontab - run:

sudo crontab -u postgres -e

and add `0 0 * * * /data/pgdata_snapshot`

### Set up a backup schedule on the EBS volume

We do this in case the server goes down and is unrecoverable.

An EBS snapshot lifecycle policy (`policy-091e2a3e4e1dcf879`) has been set up according to the instructions at: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-ami-policy.html

A snapshot of the Janeway EBS volume is saved every 24 hours. Snapshots more than 7 days old are deleted.

### Create an Amazon Machine Image (AMI)

We do this so that we can quickly re-provision a new server instance in case the server is compromised.

An AMI template (id `ami-06dd467c5ef1bcb37`, name `Imaging Neuroscience Janeway Server Template`) has been created according to the setup instructions above.
The Janeway data files and PostgreSQL backups are stored in the `imaging-neuroscience-janeway-data-store` EBS volume (ID `vol-0ba9f56a7b94cefae`).

### Provisioning a new server instance from a backup

1. Terminate the previous server instance.
2. Restore the Janeway EBS volume from a snapshot if necessary (if it has been somehow corrupted).
3. Create a new `t3.xlarge` EC2 instance from the AMI template.
- Make sure that the instance is created in the same availability zone as the EBS volume.
- Make sure that inbound SSH/HTTP/HTTPS traffic is allowed from anywhere.
- Make sure that login is allowed via the `Imaging Neuroscience Janeway Web Server key` key pair.
4. Associate elastic IP `13.41.96.240` with the instance
5. Attach the Janeway EBS volume to the instance.
6. Log in to the instance and mount the EBS volume at `/data/`
7. Restore the PostgreSQL database from a snapshot in `/data/pgdata_snapshots/`:

   sudo systemctl stop apache2
   sudo -u postgres psql
   # -> drop database janeway;
   # -> create database janeway with owner janeway;
   # -> grant create on database janeway to janeway;
   psql --username=janeway < <snapshot-file>
   sudo systemctl start apache2

## Janeway updates

It may occasionally be necessary to update the Janeway installation if e.g. some changes need to be made to the code or database. The procedure for doing this is outlined below. It goes without saying that this should be performed on a test system before being attempted on the live system.

1. Update the janeway source code on the `mnt/imgneuro` branch at https://github.com/pauldmccarthy/janeway/.
2. Log into the web server
3. Shut down apache2 (as `ubuntu` user):

   sudo systemctl stop apache2

4. Update the janeway installation (as `janeway` user):

   sudo su janeway
   cd /data/janeway/
   git fetch --all
   git merge origin/mnt/imgneuro

5. Apply Janeway database migrations if necessary (still as `janeway` user, in `/data/janeway/`):

   source activate ~/janeway.env
   python src/manage.py migrate

6. Restart apache2 (as `ubuntu` user):

   sudo systemctl start apache2