# Downtime 14.9-16.9 mailout

## Text on the email

Hi,

This mail is about an upcoming service break of the Triton cluster taking place Tue 14.9.- Thu 16.9. due to switching of filesystems.

TL;DR

- We're migrating to a newer and faster filesystem during a service break between Tue 14.9.- Thu 16.9.
- Triton and data (aka "scratch") will be inaccessible during the service break.
- All computing nodes will be down.
- All users who are logged in will be logged out.
- All pending jobs will stay pending until the system is fully up again.
- All current data (scratch group folders + users work directories) will be copied automatically by us.
- All system paths will stay the same (/m, /scratch, $WORKDIR, etc.)

Important: Remember that "/scratch" is NOT backed up. This is a good time to check that your code is under version control and important data is backed up.

For more information, read issue: https://version.aalto.fi/gitlab/AaltoScienceIT/triton/-/issues/1125

If you're a project manager or responsible for data management for your group, read the full text on the issue linked above.
If you have questions, please comment on the issue linked above.

Regards,
Simo Tuomisto
Aalto Scientific Computing

## Text on issue

**--INCLUDE HERE THE TL;DR FROM THE EMAIL--**

Triton's current filesystem is reaching its end of life and between Tue 14.9. - Thu 16.9. we're going to migrate to a new filesystem procured from HPE Cray. The new filesystem is Lustre-based and compared to our current filesystem it has much better performance and larger 5.2PB capacity.

Throughout the summer we have done exhaustive testing on the system and have already started to transfer currently stored data. These data copying operations might cause occasional slowdowns in the old system and thus we try to do them during nights.

During the coming service break we will do a last incremental sync of the old filesystem to the new filesystem and will update all of our services to use the new filesystem. During this service break there won't be access to data or Triton.

During the migration process we will :

1. On Tuesday 14.9 at 9am kill all running jobs and close any open session. Pending jobs will stay in the queue.
2. Copy all active group shared folders. This means /scratch/{department_name}/{project_name} a.k.a. /m/{department_name}/scratch/{project_name} directories.
3. Copy work directories for all users that have an active Aalto account. This means /scratch/work/{user_name} a.k.a. /m/{department_name}/work/{user_name} directories.

This migration does not require any action from you, the users. However, now is a good time to check that all your important files are backed-up, do some clean up, and check on your current data handling practices.

It is important to remember that Triton's scratch directories are meant for large and temporary data you create from your analysis and it is not backed-up. It is not designed for long term storage of your code or your data.

We recommend that you:

1. Terminate your running jobs and close your open sessions to avoid any potential loss of data
2. Check for unnecessary data that you do not need any more and remove it. Do be careful though when removing data. Remember that the filesystem does not have backups or checkpoints.
3. Try to organize your data so that you know what data you have in your project/work folder. See e.g. our page on data organization strategies (https://scicomp.aalto.fi/data/organization/#organize-files-within-directories) for more information.
4. Check if you have some data in your work directory that should be in a project directory and migrate this data. If you know that somebody in your project might have data in their work directory that should be in a project directory, work with them to migrate the data to the project directory.
5. Check if your code files are under version control. See e.g. our page on Aalto Gitlab (https://scicomp.aalto.fi/aalto/git/) for more information on version control.
6. Check if you have important data that is hard to recreate (such as measurements from experiments etc.) with no backups. Ask us to help you create a backup of such datasets. See e.g. our page on data storage services (https://scicomp.aalto.fi/data/services/) for alternative storage locations for such data.
7. Check if you have completed projects that could be archived. Ask us to help you archive such projects. See our page on data storage services (https://scicomp.aalto.fi/data/services/) for possible archival locations.

If you have any questions, do not hesitate to contact us through the usual channels:
- garage https://scicomp.aalto.fi/help/garage/
- Aalto SciComp's Zulip chat https://scicomp.aalto.fi/help/#chat
- Triton issue tracker (this issue)
- mail us to [e-mail].

## Text on MODT

███╗   ██╗ ██████╗ ████████╗███████╗██╗
████╗  ██║██╔═══██╗╚══██╔══╝██╔════╝██║
██╔██╗ ██║██║   ██║   ██║   █████╗  ██║
██║╚██╗██║██║   ██║   ██║   ██╔══╝  ╚═╝
██║ ╚████║╚██████╔╝   ██║   ███████╗██╗
╚═╝  ╚═══╝ ╚═════╝    ╚═╝   ╚══════╝╚═╝

- >>  UPCOMING MAINTENANCE, READ THIS :

- Downtime on : Tue 14.9/Thu 16.9.2021
- We are getting a new "scratch" storage
- Old data will be synced to the new scratch
- No access to scratch files, no access to computing nodes
- Running jobs will be killed, pending jobs will stay pending
- scratch is *NOT* backed-up
- Back up important files before by Mon 13.9 end of day

`´`