---
# System prepended metadata

title: ARC4 remediation project
tags: [research-computing]

---

---
tags: research-computing
---
# ARC4 remediation project

## Prerequisites

* Appoint new infrastructure team:
    * 1 x Research Infrastructures lead
    * 3 x HPC systems adminstrators

* Expand and enhance the RSE team - as per proposed structure with immediate appointments of:
    * 2 x Scientific and parallel RSEs
    * 1 x Data Scientist

... to enhance the current team in project 3 scientific discovery work, benchmarking and tender preparation.  Subsequently for the significant user support effort in migrating researchers ARC3 -> ARC4 -> ARC5 and optimising code and pipelines towards a more efficient use of HPC and research code in general.

## Three parallel projects

| Project 1                                   | Project 2: ARC4+                                        | Project 3                                      |
|---------------------------------------------|---------------------------------------------------------|------------------------------------------------|
|ARC3 retirement                              |ARC4 remediation (D2)                                    |ARC 5 procurement                               | 
|(ramp down capacity over project)            |Move Astbury nodes from ARC3 (1)                         |Technology exploration                          |
|Extra maintenance on infrastructure nodes?   |Move public GPU nodes from ARC3                          |Access to new CPUs/GPUs/Cloud: benchmarking     |
|                                             |Stand up new infrastructure for Slurm                    |Decision on new modules system                  |
|                                             |Rebuild and refactor Lustre filesystem (2)               |Decision on OpenOnDemand                        |
|                                             |Slurm: pre-emptive scheduler for private nodes (3) (D1)  |Determine size and structure of cluster(s)      |
|                                             |Better management of projects and core-hours allocation  ||
|                                             |Evaluate Cloud (burst? FPGA/Graphcore? single core?) (4) ||



## Notes

(1) Need to discuss timescales etc. with Astbury
(2) This will probably involve a period of system downtime and a Lustre wipe
(3) Pre-emptive scheduling will need discussion and agreement from owners of private nodes
(4) 78% of single-core jobs potentially suited to Azure as PAYG. £68000 PAYG or £15000 on reserved instances -> cheap instances for eg. MC simulations

## Dependencies
(D1) After purge/ clearout use RDS to store the remaining ~0.5PB data -> don't underestimate time taken for this.
(D2) Researcher-led consultation group underwrite and agree with proposals particularly wrt. outage planning

## Timescales
(T1) TDA presentation on 2021-12-08
(T2) ARC4+ consultation group < 2021-11-30

## Actions
(A1) TDA high-level presentation (All)
(A1a)Architecture help from Venetia/ JohnW
(A2) Paper (Liz)
(A3) Consultation group for researcher buy-in? (Liz/ Mark)




