| Column 1 | Column 2 | Column 3 |
| -------- | -------- | -------- |
| Text | Text | Text |
# Table for `clim-recal` data on Azure shared drive

**Please Note: Any files has been changed/deleted from the SMB shared drive is UNRECOVERABLE, so please always make a backup of all important files to the blob storage.**

| Name | Folder path | Data type | Size | Suggetion Process (as of 02/04/2024) | Status update | Note |
| ------------------------------ | ----------------------------------------- | ---------------- | --------------------- | ------------------------------------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| UKCP2.2 | `ClimeData/Raw/UKCP2.2` | Raw Data | 8.92TB | Archive | 12APR transferd&achived, 25APR deleted from the vmshare (excepting `tasmin`, `tasmax`, `pr`). 22MAY transferd&achived the `tasmin`, `tasmax`, `pr`, excepting the folder in notes | For detailed folders and their processing please see the notes 1 |
| Projected UKCP2.2 | `ClimeData/Reprojected_infill/UKCP2.2` | Almost Raw Data? | 9.25TB | Hot+Archive storage | 17APR transferd&achived, 25APR deleted from the vmshare (excepting `tasmin`, `tasmax`, `pr`) 22MAY transferd&achived the `tasmin`, `tasmax`, `pr`, excepting the folder in notes | For detailed folders and their processing please see the notes 2 |
| HadsUKgrid | `ClimeData/Raw/HadsUKgrid` | Raw Data | 561.31 GB | Hot+Archive storage | 17APR transferd&achived, 25APR deleted from the vmshare (excepting `tasmin`, `tasmax`, `rainfall`) | For detailed folders and their processing please see notes 3 |
| CHESS-SCAPE | `ClimeData/Raw/CHESS-SCAPE` | Raw? | 39.43GB | Archive | 12APR transferd&achived, 25APR deleted from the vmshare | Future climate projections data for whole UK for 1980-2018 by [UK-SCAPE poejct](https://https://uk-scape.ceh.ac.uk/our-science/projects/SPEED/future-climate-projections) |
| QuantileMapping | `ClimeData/Debaised/R` | Interim Data | 439.41 GB | Try compressing first | | |
| tasmax | `ClimeData/Debaised/tasmax` | Interim Data | 18.45GB | Keep Hot | | |
| debased "three.cities.cropped" | `ClimeData/Debaised/three.cities.cropped` | Interim Data | 50.3MB | Keep Hot | | |
| intrim CPM | `ClimeData/Interim/CPM` | Interim Data | 532GB | Archive | 07APR transferd&achived, 25APR deleted from the vmshare | CPM data as df? |
| interm HadsUK | `ClimeData/Interim/HadsUK` | Interim Data | 600GB | Archive | 12APR transferd&achived, 25APR deleted from the vmshare | |
| workshop | `ClimeData/workshop` | Interim Data | 31MB | Keep Hot | | |
| Processed HadsUK | `ClimeData/Processed/HadsUK` | Interim Data | 277GB | Keep Hot | | |
| Processed CHESS-SCAPE | `ClimeData/Processed/CHESS-SCAPE` | Interim Data | 7.5GB | Archive | 12APR transferd&achived, 25APR deleted from the vmshare | |
| Total | | | 19.1 TB (On 12th Mar) | | Reduced to 4.3TB on 25th Apr | Archived size:15.60TB |

## Notes

1. Raw UKCP2.2
`Raw/UKCP2.2/tasmin/05` -> we now need (numbers indicate urgency)
- `Raw/UKCP2.2/tasmax/`
- 0: `Raw/UKCP2.2/tasmax/01`
- 1: `Raw/UKCP2.2/tasmax/05`
- `Raw/UKCP2.2/tasmax/06`
- `Raw/UKCP2.2/tasmax/07`
- `Raw/UKCP2.2/tasmax/08`
- `Raw/UKCP2.2/tasmin`
- `Raw/UKCP2.2/tasmin/01`
- `Raw/UKCP2.2/tasmin/05`
- `Raw/UKCP2.2/tasmin/06`
- `Raw/UKCP2.2/tasmin/07`
- `Raw/UKCP2.2/tasmin/08`
- `Raw/UKCP2.2/pr`
- 2: `Raw/UKCP2.2/pr/01`
- `Raw/UKCP2.2/pr/05`
- `Raw/UKCP2.2/pr/06`
- `Raw/UKCP2.2/pr/07`
- `Raw/UKCP2.2/pr/08`

3. Projected UKCP2.2 hot storage folders:

`Reprojected_infill/UKCP2.2/tasmin/05`
`Reprojected_infill/UKCP2.2/tasmin/06`
`Reprojected_infill/UKCP2.2/tasmin/07`
`Reprojected_infill/UKCP2.2/tasmin/08`
`Reprojected_infill/UKCP2.2/tasmax/05`
`Reprojected_infill/UKCP2.2/tasmax/06`
`Reprojected_infill/UKCP2.2/tasmax/07`
`Reprojected_infill/UKCP2.2/tasmax/08`
`Reprojected_infill/UKCP2.2/pr/05`
`Reprojected_infill/UKCP2.2/pr/06`
`Reprojected_infill/UKCP2.2/pr/07`
`Reprojected_infill/UKCP2.2/pr/08`

2. HadsUKgrid hot storage folders:

`Raw/HadsUKgrid/tasmin`
`Raw/HadsUKgrid/tasmax`
`Raw/HadsUKgrid/rainfall`

## Comments:

*RB*: I'd be inclined to keep tasmax, tasmin and precip in all folders, but store the other vars, rather than store whole folders.
CHESS-SCAPE can be stored as we aren't using at the moment
Also we could store the runs we aren't focusing depending what we decide about applying the 'chosen' method to the whole dataset
Also I think `ClimateData/Debaised/tasmax` probably could be put into storage but I will double check

## Azure teirs

- Hot tier - An online tier optimized for storing data that is accessed or modified frequently. The hot tier has the highest storage costs, but the lowest access costs.
- Cool tier - An online tier optimized for storing data that is infrequently accessed or modified. Data in the cool tier should be stored for a minimum of 30 days. The cool tier has lower storage costs and higher access costs compared to the hot tier.
- Cold tier - An online tier optimized for storing data that is rarely accessed or modified, but still requires fast retrieval. Data in the cold tier should be stored for a minimum of 90 days. The cold tier has lower storage costs and higher access costs compared to the cool tier.
- Archive tier - An offline tier optimized for storing data that is rarely accessed, and that has flexible latency requirements, on the order of hours. Data in the archive tier should be stored for a minimum of 180 days.

Costs : Hot: $0.018 per GB Cool: $0.01 per GB Archive: $0.00099

## Update 09th APR by BZ

The 'tier change' function has been tested and seems to work as expected. I have copied the `ClimeData/Interim/CPM` dataset to the new `archieve` container

The offical guideline is [here](https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-online-manage?tabs=azure-portal#bulk-tiering), the website portal can only operate single files, not applicable for bulks. It advised to use either `Azure CLI` or PowerShell to manage bulks or folders.

The easiest way I found is to use Microsoft Azure Storage Explorer to select "all files" and switch their access tier to "archive."

## Update 21th May by BZ

The current daily basis cost is around £9.82 for storage only, along with some cost of IO.

# Update on 10 Oct by Griff

Below is a table for files left in the shared drive for processing within the `clim-recal-test` environment

## Main folders

*As of 11/20/2024*

**Note**: The Type column has 4 states
- **Test**: used for testing
- **Raw**: raw data
- **Prev**: *previous* results in December 2023
- **Final**: Final results for public release

Folder path | Type | Size | Process | Todo | Note |
------------------------- | ---- | ----- | --------- | ------ |----- |
`CPM-365` | Test | 1.8TB | Delete | Done | |
`Raw` | Raw | 1.1TB | Delete | All of this has been previousled archieved. We have deleted all files from fileshare **except** 1990 (to allow for confirmation of changes to code) | Known issues `tasmax`: 1981, 1990? |
`processed_2024_09_26` | Final | 848GB | Delete | Done | |
`processed_2024_june` | Test | 824GB | Delete | Done | |
`Interim` | Prev | 630GB | Delete | Done | |
`Reprojected_infill` | Prev | 554GB | Delete | Done | |
`group_runs` | Final | 475GB | Delete | Done | |
`Processed` | Prev | 259GB | Delete | Done | |
`Debaised` | Prev | 238GB | Keep for now | | Believed this is used in the vunerable population explorer |
`Reprojected_infill_HADS` | Test | 93 GB | Delete | Done | |
`Scripts` | Prev | 3.4GB | Delete | Done | |
`Cropped` | Prev | 3.1GB | Delete | Done | Initial results |
`Sharefiles` | Prev | 68 MB | Delete | Done | |
`workshop` | Prev | 31 MB | Delete | Done | |
`processed_2024_08_30` | Test | 0 B | Delete | Done | |
**Total** | | **6.2TB** | | | |

## Sub folders

*As of 11/20/2024*

Folder path | Raw | Size | Process | Status update | Note |
-------------------------------| --- | ----- | -------- | ------------- |----- |
`CPM-365` | N | 1.8 TB | Delete | | Held initial results |
`Raw/UKCP2.2` | Y | 599 GB | Archive | Run Checksum | | Known issues `tasmax`: 1981, 1990? |
`Reprojected_infill/UKCP2.2` | N | 554 GB | Delete | | |
`Raw/python-refactor` | N | 41 KB | Delete | | |
`Raw/HadsUKgrid` | Y | 478 GB | Run Checksum 10 Oct | | Known issue: `tasmin` Nov 1982 |
`Debaised/R` | N | 410 GB | Ask Ruth | | |
`Debaised/tasmax` | N | 18 GB | Ask Ruth | | |
`Debaised/three.cities.cropped`| N | 624 MB | Ask Ruth | | |
`Interim/CPM` | N | 630 MB | Delete | | |
`workshop` | N | 31 MB | Delete? | | |
`Processed/HadsUK` | N | 259 GB | Delete? | | |
**Total** | | **6.2TB** | | | |