# TRE Deployment Guide

This guide assumes that you have already [set up your SSH keys and GPG keys for gitlab.er.kcl.ac.uk](https://docs.gitlab.com/ee/user/ssh.html#add-an-ssh-key-to-your-gitlab-account).

!!! attention
TRE Projects are created on receipt of a completed OnBoarding form or a request that explicitly states the research_vm_count, server_vm_count, linux_vm_count and smb_data_volume_size.

##  Create project in e-Research Portal

1. Navigate to [Admin > Groups](https://portal.er.kcl.ac.uk/admin/groups/new) in the e-Research Portal

2. Create new group in the format `er_tre_<dept>_<project>_sysadmins`

!!! Warning
This project name must not exceed 10 characters

3. Leave the `Pretty` and 'descriptions' field blank.

4. Set AD membership: `CN=er_tre_<dept>_<project>_sysadmins,OU=tres,OU=groups,OU=e-research,DC=kclad,DC=ds,DC=kcl,DC=ac,DC=uk`

5. Ensure `Add/Remove members via Portal` & `Trusted Research Environment` are selected.

6. Set Resource quota as per project requirements, Other values may be given although the default is the 4cpu8ram flavours with 40GB of Local storage:

!!! note
* `total_vm_count = research_vm_count + linux_vm_count + 2` (the `+ 2` is for the smb and guac VMs) `+ 1` if database_vm is required
* `CPUs = 4 * (total_vm_count)`
* `RAM = 8 * (total_vm_count)`
* `Ceph-NVME = 50 * (total_vm_count)`
* `Ceph-HDD = smb_data_volume + guac_data_volume`
* `Security Groups 50 Minimum`
* `Security rules = 300+`

!!! note
* for projects that include a database_vm, please allocate an additional 4core8ram to the project (for 8core16ram total for the database_vm)

7. Ensure that you add yourself and Michal (K2256745) to the `er_tre_<dept>_<project>_sysadmins` group.

8. Add Openstack project as a resource to the TRE portal group.

9. Add Openstack VDI and select current project from the list

10. Once the project has been created the portal will automatically generate five Active Directory Security groups to manage access to the project within an hour.

* `er_tre_<dept>_<project>_sysadmins` - This is the Project administration group
* `er_tre_<dept>_<project>_users` - Membership of this group grants access to the project resources, every project member need to be added to this group
* `er_tre_<dept>_<project>_data_managers` - Members of this group will have full access to the project dataset directory
* `er_tre_<dept>_<project>_data_ingress` - Members of this group will be able to ingress data on using the Datamovers VM
* `er_tre_<dept>_<project>_data_egress` - Members of this group will be able to approve egress data requests via the egress portal

11. Once these groups are created, you will need to add the VDI resource (corresponding to the Cloud project) to `er_tre_<dept>_<project>_users`.

## Generate Openstack Application Credentials

1. Go to the [Openstack Horizon Dashboard](https://cloud.er.kcl.ac.uk)

2. Select your project from the drop down list

3. Go to Identity > Application Credentials > Create Application Credential

4. Set the name of your application credential to be `er_tre_<dept>_<project>_gitlab_runner`

5. Set the expiry date for your credentials to be 1 year

6. Create your application credential, copy the application ID and the secret. You can leave this open to make it easier to refer to later.

7. Find your application credential in the list and make note of the project ID

## Create Gitlab project

1. [Navigate to our tre test repo on our Gitlab.](https://gitlab.er.kcl.ac.uk/infra_terraform/openstack-projects-terraform/tf_er_tre_demo_dev)

2. Clone the tf_er_tre_demo_dev repository to your local machine.

    PROJECT_NAME="tf_er_tre_abc_xyz"
    git clone [e-mail]:infra_terraform/openstack-projects-terraform/tf_er_tre_demo_dev.git $PROJECT_NAME

!!! Note
TRE Project names in Gitlab should follow the format: tf_er_tre_*department*_*project*

3. Remove git folder

    cd tf_er_tre_abc_xyz
    rm -rf $PWD/.git

4. Update README.md with TRE information and then create the initial commit for your project.

    git init --initial-branch=production
    git add README.md .gitignore
    git commit -S -m "feat: Initial commit"
    git push --set-upstream [e-mail]:infra_terraform/openstack-projects-terraform/$PROJECT_NAME.git production

## Setup project CI/CD Configuration

1. Open [Gitlab](https://gitlab.er.kcl.ac.uk/infra_terraform/openstack-projects-terraform) and find your newly created project.

2. Go to *Settings > CI/CD > Variables*

3. Create the following variables, using the credentials generated earlier and the Openstack Project ID, ensure that the OS Application Credential ID and Secret are set to masked.

    OS_APPLICATION_CREDENTIAL_ID="OSApplicationID"
    OS_APPLICATION_CREDENTIAL_SECRET="OSApplicationSecret"
    TF_VAR_tenant_id="123456789123456789"

!!! Note
If you are creating a development environment you will need to set the VAULT_ADDR, VAULT_CAPATH variables so the project is using the development Vault server.

You may also wish to set TF_DESTROY if you want to be able to destroy the environment from gitlab.

Example values can be found [here](https://gitlab.er.kcl.ac.uk/infra_terraform/openstack-projects-terraform/tf_er_tre_demo_dev/-/settings/ci_cd)

4. Go to *Settings > CI/CD > Runners*

5. Ensure that 'Enable shared runners for this project' is switched off.

!!! Important
If shared runners are not disabled the Gitlab CI job will fail! TRE's have their own dedicated Gitlab runners with line of sight to the TRE Vault.

##   Add TRE project to Vault

1. Go to the [tf_er_tre_vault](https://gitlab.er.kcl.ac.uk/infra_terraform/openstack-projects-terraform/tf_er_tre_vault) repository

2. Clone the repository and create a new branch

3. Add your Gitlab project ID (`Settings > General`) to the end of the appropirate file in [environments](https://gitlab.er.kcl.ac.uk/infra_terraform/openstack-projects-terraform/tf_er_tre_vault/-/tree/production/environments?ref_type=heads)

    ## er_tre_abc_xyz
    ## https://gitlab.er.kcl.ac.uk/infra_terraform/openstack-projects-terraform/tf_er_tre_abc_xyz
    {
        id                     = 001 # Gitlab Project ID
        ref                    = "*"
        ref_type               = "*"
        ref_protected          = "*"
        token_explicit_max_ttl = 86400
        environment            = "production" # Default branch name for your project, this will usually be 'production'
    },

!!! warning
Some older TRE projects may have a different Environment name set to `default` or `main`, if this is not set correctly the deployment will fail!

4. Push your changes

5. Open a merge request selecting the appropriate branch you want to merge to and assign reviewer to Michal or Glyn. You must open your merge request to production as staging will go to a test TRE vault rather than production.

6. Once approved, merge your changes and ensure the CI job deployed the changes. Ensure the jobs are successful before merging.

## Add project to Windows Puppet

1. Clone the [Windows Puppet Repository](https://gitlab.er.kcl.ac.uk/puppet/puppet-windows)

2. Create a new branch for your project deployment

3. Add a new Hiera file to `data/openstack/er_tre_<dept>_<project>.yaml`, including software to be installed on the Research VMs and User Groups for VDI Access.

    # data/openstack/er_tre_abc_xyz.yaml
    role::vdi::base::groups:
    - kclad.ds.kcl.ac.uk\er_tre_abc_xyz_users
    base::packages::chocolatey::managed:
    - name: spss29
        version: 29.0.1
    - name: office2021enterprise
        version: 1.0.0
    - name: rproject
        version: 4.3.0
    - name: rstudio
        version: 4.9.4

4. Add a new Hiera file `data/nodes/er-tre-<project>-dm01.er.kcl.ac.uk.yaml` to allow members of egress and ingress AD groups access to their Data Mover VM.

    # data/nodes/er-tre-abc-dm01.er.kcl.ac.uk.yaml
    role::vdi::base::groups:
    - kclad.ds.kcl.ac.uk\er_tre_abc_xyz_data_egress
    - kclad.ds.kcl.ac.uk\er_tre_abc_xyz_data_ingress

5. Commit and push your changes

6. Open a Merge request targeting the staging branch

7. When the CI Lint job has finished merge your changes into staging

8. Create another merge request to production from staging and assign it to a Project Administrator, once they have reviewed and merged the changes you can continue to creating your TRE project resources.

!!! note
Puppet Windows certificates will need signing on `er-win-puppet1` by a member of the team with necessary permissions

## Add project to Puppet

1. Clone the [Puppet Repo](https://gitlab.er.kcl.ac.uk/puppet/puppet) and create a new branch. You can use the `<dept>_<project_name>` for this.
2. Add a new file at data/nodes/er-tre-<Gitlab project ID>-lm01.er.kcl.ac.uk.yaml with the software required. The contents of the file should be similar to this:

    ---
    role::tre_desktop::base_packages:
      - 'nfs-common'
      - 'xubuntu-icon-theme'
      - 'cifs-utils'
      - 'python3-virtualenv'
      - 'xfce4-screensaver'
      - 'python-is-python3'

3. Commit the changes to your test branch before signing them and creating a merge request to the staging branch. Assign a member of the team to review your changes and they will push the merge when appropriate. Ensure you are following the moggified repo template or you will fail the pipeline!

## Create TRE Resources

1. Go to your local TRE Project repository that you cloned earlier

2. Create a new branch

    git checkout -b initial_deployment

3. Amend the variables.tf file to your project requirements, such as number of VMs, their Openstack Flavours and the Openstack Project name in Puppet facts.

4. Update `vault_approle_path` and `AUTH_JWT_ROLE` with the ID from the gitlab project (`Settings > General`)

5. Check `research_vm_count`, `server_vm_count`, `linux_vm_count` and `smb_data_volume_size` match the requirements from the OnBoarding form

6. Edit `research_vm_name`, `server_vm_name` and `linux_vm_name` as appropriate, using the Gitlab Project ID (3 digits) instead of <dept>_<project> for brevity
Storage VM: `^er-tre-[0-9]{3}-sto1$`
Guacamole VM: `^er-tre-[0-9]{3}-guac?1$`
Linux desktops: `^er-tre-[0-9]{3}-l[gm][0-9]{2}$`

7. Stage and push your changes

    git add .
    git commit -S -m "feat: deploy new TRE project"
    git push --set-upstream origin initial_deployment

8. Go to your project in Gitlab and create a merge request to the production branch.

9. Wait for the Terraform Plan CI Job to finish

10. Validate the results of the CI Job, ensuring that the correct resources have been deployed and correcting your terraform manifests if not.

11. Once you've reviewed the Terraform plan, accept the merge request. When merged the Gitlab CI runner will re-run the plan and then deploy the project.

## Common Errors

### Error waiting for instance to become ready unexpected state ERROR

**Example:**

│ Error: Error waiting for instance (6bfe72c9-dcc9-4b08-8f27-9f60dd9627a0) to become ready: unexpected state 'ERROR', wanted target 'ACTIVE'. last error: %!s

**Troubleshooting:**

* Check Project storage quota

### Saved Plan is stale

**Example:**

│ Error: Saved plan is stale
│
│ The given plan file can no longer be applied because the state was changed
│ by another operation after the plan was created.
╵

**Troubleshooting:**

* Re-run the Build job in your pipeline, then retry the deploy job.

### Error validating claims: claim "environment" does not match any associated bound claim values

**Example:**

╷
│ Error: Error making API request.
│
│ URL: PUT https://er-tre-dev-vault1.er.kcl.ac.uk:8200/v1/auth/jwt/login
│ Code: 400. Errors:
│
│ * error validating claims: claim "environment" does not match any associated bound claim values
│
│   with provider["registry.terraform.io/hashicorp/vault"],
│   on provider.tf line 22, in provider "vault":
│   22: provider "vault" {
│
╵

**Troubleshooting:**

* Ensure the Environment name in the [Vault repository](https://gitlab.er.kcl.ac.uk/infra_terraform/openstack-projects-terraform/tf_er_tre_vault/-/tree/production/environments?ref_type=heads) matches the Gitlab Environment name (This is usually the default branch name)