# Maintenance break 2022-05

## Proposed timeline of actions

Timeline proposal before update:

- [ ] Do a preliminary sync of home2 -> ssd3
- [ ] Do a preliminary sync of ssd2 -> clusterstor

Timeline proposal during update:

- [ ] Start taking nodes down, unmount lustre everywhere
- [ ] Start home2 -> ssd3 sync (infiniband needed)
- [ ] Sync ssd2 -> clusterstor (clusterstor needed)
- [ ] Update clusterstor (clusterstor down)
- [ ] Update slurm packages & pmix packages, slurm accounts, slurm config, job_submit, ohpc images
- [ ] Update Mellanox cards / IB switches that do not affect the home2 -> ssd3 transfer
- [ ] Update rest of Mellanox switches
- [ ] Enable slurm jobstats on slurm & clusterstor
- [ ] Restart nodes, bring queue back up

## Overview of actions

| Affected service | Action | Requirements on other services | Effects on other services | Who does |
| ------ | ------ | ------ | ------ | ------ |
| Slurm | New account structure with weights | None | `job_submit.lua` needs to be updated, new_user.py | Simppa |
| Slurm | gpushort partition removal | None | `job_submit.lua` needs to be updated | ? |
| Slurm | GPU plugin activation | None | None (documentation update needed) | Simo |
| Slurm | Slurm pmix update | Slurm will go down and needs to be updated everywhere | None (sw build against new pmix need to be tested before) | Simo & Mikko |
| Slurm | Jobstats enabling on Slurm & ClusterStor | Slurm & ClusterStor | slurm prolog scripts need to be updated, feature needs to be enabled on ClusterStor | Simo & Mikko |
| Home | home2 to ssd3 transfer | Infiniband network is needed for transfer | None | Mikko |
| Apps | ssd2 reinstall | ClusterStor is needed for backup | None | Simo can handle the backup, but someone else should perhaps do the reinstall |
| Infiniband | EDR switch update | None | Nodes that use the switches are affected | Ivan |
| Infiniband | Mellanox cards' HCA's firmware check and upgrade | None | ? | Ivan |
| ClusterStor | Firmware update | None | Everything that needs ClusterStor is affected | Simppa |