---

tags: meetings
title: Material - CSC-GIS user support and networking event

---
[TOC]

# CSC-GIS user support and networking event

:::info

**This page:** https://hackmd.io/@GeospatialCSC/support_event

:::

We have tried to identify people from research institutes and universities that do **some kind Geoinformatics and/or CSC support** for their colleagues and/or are more experienced users of CSCs Geoinformatics related services and hope that with you, we reached the right person.

:::danger
Thanks for supporting us supporting researchers! :heart:
:::
[TOC]

---

## Introduction round

:::info
* Name
* Affiliation
* what do you do?
* Why are you here?
* How do you use CSC resources in your work?
:::

> Notes by Kylli during introduction, feel free to edit (use "pen symbol" up top) or remove :)

* Samantha Wittke; CSC/FGI(NLS)/Aalto/CodeRefinery; CSC geoinformatics support/ PhD student (EO time series); meeting YOU, getting to know your issues and ideas
* Kylli Ek, CSC, GIS-coordinator, Paituli, HPC GIS support, courses-workshops, universities ArcGIS consortium
* Katri Tegel, CSC, manager, GIS-background
* Tatu Leppämäki, HY, Digital geography lab, PhD student, social media data analysis, contact person for my group. cPouta, data storage.
* Eelis Halme, VTT/Aalto, PhD student, remote sensing team, have not used CSC resoruces yet, planning to do.
* Matti Mõttus, VTT, forest remote sensing, biodiversity, used CSC HPC.
* Arttu Kivimäki, FGI/NLS, Department of remote sensing and photogrammetry, working with optical and radar satellite data time series, use almost daily CSC HPC and Allas
* Vuokko Heikinheimo, SYKE (past HY), urban land use research, I have used CSC services for many years, but not that much lately. We could use more.
* Eetu Jutila, SYKE/Aalto, GIS-expert, change detection from Sentinel1, help internally other CSC researchers. Mammutti project: data storage and computing. Daily basis using Puhti and Allas.
* Arto Viinikka, SYKE, remote sensing, I have used HPC a couple of times. Also in the Mammutti project.
* Janne Mäyrä, SYKE, biodiversity center, computational background, I have used CSC resources all my working time at SYKE, computer vision projects, so high GPU needs.
* Antti-Jussi Kieloaho, Luke (Natural Resources Institute Finland), Information system specialist; data science/data engineering, scientific computing coordinator; I do and help on GIS analysis and spatial modelling that is reason to be here; I occasionally use also CSC resources. Puhti, Mahti, cPouta, Paituli, Allas.
* Matthieu Molinier, VTT, remote sensing team, change detection, optical data, deep learning, would like to use CSC more regularly
---

## [Services available at CSC](https://research.csc.fi/en/service-catalog)

**Compute & Analyze**

* cPouta / ePouta
* Puhti / Mahti / LUMI
* CSC Notebooks
* Rahti

**Store, Share & Publish Data**

* Allas
* (EUDAT)
* Fairdata IDA
* Paituli

**Sensitive Data (SD) services**

---

## What's new?

**Puhti**
* puhti.csc.fi
* Files, shells, RStudio/Jupyter/Desktop Applications
* RHEL-8 update is coming, 4-5.10 ?:
* Test system opened 15.9.
* Old versions will disappear.
* New versions available.
* Python2, Sofi3D, solaris, SPLITS and mapnik discontinued
* Own software might need to be recompiled
* New names for login-nodes
* [New documentation for accessing](https://csc-guide-preview.rahtiapp.fi/origin/puhti-rhel8/support/tutorials/puhti-rhel8-test/)
* [New documentation about apps](https://csc-guide-preview.rahtiapp.fi/origin/puhti-rhel8/apps/by_system/#puhti-rhel8)
* [Tykky](https://docs.csc.fi/computing/containers/tykky/) for Apptainer container creation, based on:
* Conda .yml file/ pip requirements.txt
* Existing Docker image

**Self-learning/examples/materials**
* [Geocomputing examples](https://github.com/csc-training/geocomputing):
* [STAC + xarray + dask](https://github.com/csc-training/geocomputing/tree/master/python/STAC): [static STAC with FMI Tuulihaukka catalogue](https://github.com/csc-training/geocomputing/blob/master/python/STAC/static_stac.ipynb) and [STAC API](https://github.com/csc-training/geocomputing/blob/master/python/STAC/stac_xarray_dask_example.ipynb)
* docs.csc.fi
* [JupyterLab and Dask](https://docs.csc.fi/support/tutorials/dask-python/#dask-with-jupyter)
* EO & Pointcloud workshop materials (and much more): https://research.csc.fi/gis-learning-materials

**CSC Notebooks**
* notebooks.csc.fi
-> self-learning, collaboration, courses

[**Paituli**](https://paituli.csc.fi/):
* OGC-API support

---

## Support

* 'Z is not working as expected'
* 'my code gives error Y '
* 'can A be installed to Puhti?'
* 'any advice how to do X?'
* 'which service suits my needs?'
* training/example wishes

-> [e-mail]

[Speed up your request](https://docs.csc.fi/support/support-howto/)

CSC can also be project partner/subcontractor

piloting currently: **weekly user support sessions** , every Wednesday at 14 in Zoom : https://ssl.eventilla.com/event/PP4WB

---

## Acknowledging CSC in publications and reports

If you used any of our resources for your research, please acknowledge CSC and Geoportti in your publications, it is important for project continuation and funding reports. As an example, you can write:

"The authors wish to thank CSC - IT Center for Science, Finland (urn:nbn:fi:research-infras-2016072531) and the Open Geospatial Information Infrastructure for Research (Geoportti, urn:nbn:fi:research-infras-2016072513) for computational resources and support".

---

## Course and event advertisements
(see also https://www.csc.fi/en/training#training-calendar)

**CodeRefinery**
CodeRefinery invites everyone interested in improving their software practice skills to join the CodeRefinery workshop on September 20.-22. and 27.-29. See https://coderefinery.github.io/2022-09-20-workshop/ for more details.

**Using CSC environment efficiently - self learning**
Highly suggested for anyone starting out (one may also choose only specific topics): https://csc-training.github.io/csc-env-eff/

**Machine Learning for spatial data**
7.-9. November: https://ssl.eventilla.com/event/VDK2b

**GIS lecturer meeting**
-> discussion and information on how to integrate CSC services in teaching and how CSC services may support your teaching
coming soon, let us know via [e-mail] if you or your colleagues are interested in joining

**Weekly user support sessions**
Everyone welcome to ask questions from our experts!
Every Wednesday at 14 in Zoom : https://ssl.eventilla.com/event/PP4WB (currently piloting)

---

## Want to get latest news?

* Mailing list: email reminders about events and news around GIS at CSC, sign up to: https://postit.csc.fi/sympa/info/gis-hpc
* Related Twitter accounts to follow: @CSCfi , @CSChpc , @geoportti , @kylli_ek
* service related e-mail lists (automatic subscription when using some services)

---

## Discussion

:::info
* Questions? Requests? Ideas?
* Wishes for future (for you/your colleagues): courses, materials, examples, events, ... ?
* What can we do to make your life (wrt helping colleagues) easier?
* How can we make servicedesk and support more approachable?
* What kind of problems do your colleagues often need help with?
:::

> Notes from meeting plus few additions; feel free to edit, **Thank you for raising all these points**, we will pick them up internally and let you know if there are any interesting outcomes

### Continuity

- CSC project and its continuity is kind of unclear concept: what about continuation over projects, even when the project leader maybe changes? What to do with project and data when its lifetime ends, and how to integrate or move project to outside CSC?
- Publish project results in Fairdata or Paituli; also research data can be stored in Fairdata; see also: https://docs.csc.fi/data/datasets/hosting-datasets-at-CSC/#when-the-research-project-ends
- Have software/scripts under version control outside of CSC, e.g. github/-lab

### Puhti scratch vs Allas

- When is Puhti scratch directory the right place for data, and when is it Allas?
- From [docs](https://docs.csc.fi/data/Allas/using_allas/common_use_cases/): "The CSC supercomputers provide disk environments [e.g. scratch] for working with large datasets. These storage areas are however not intended for storing data that is not actively used. One of the main use cases of Allas is to store data while it is not actively used in the CSC supercomputers. When you start working, you stage in the data from Allas. And when the data is no longer actively used, it can be staged out to Allas. "

### Portability

- To create something for specific cloud environment is work; concerns about portability
- One option is to use containers for your software environment
- As much as possible use independent scripts for setting up your workflow and only run those from within sbatch-script
- Linux subsystem for Windows is a great way to make use of Linux flexibility also on Windows computers

### Research -> Operational

- When does research turn operational from CSC perspective; what changes?
- We trust the users to inform us about that. In many cases, using CSC resources can be continued, but need to be paid. See https://research.csc.fi/free-of-charge-use-cases

### Overwhelming amount of services

- Too many options for end user to choose from, it is difficult to know what to do as a single researcher starting to use CSC services
- You can always send a message to [e-mail]; also already when planning a project; explain your project and your needs as much as possible
- Another option is to visit our weekly research user support session, currently every Wednesday at 14: https://ssl.eventilla.com/event/edit/102432
- Computing: most of our Geoinformatics cases fit best to Puhti supercomputer ("need for more memory/CPU/GPU or parallelization")
- Data storage: most usecases make use of Allas ("where can I share my data with project members, so that everyone can use data for computations in different environments")

### Fairdata services
- How to deal with data versions in Fairdata? Is there something implemented at CSC for versioning?
- Fairdata has [upload new version option](https://www.fairdata.fi/en/qvain-user-guide/#QvainVersion) to add new version as new file. No Github-kind of way to see fast what has been changed in the files.
- DOI rules require that a new DOI is given to the new version.
- URN is technically more flexible.

### Puhti webinterface

- Can same view of Jupyter/Desktop be shared among multiple users in web interface?
- Not Jupyter or RStudio, but Desktop view can be shared, after your session is ready, you can find "view only(shareable link)" button in bottom right of your session window: https://docs.csc.fi/computing/webinterface/desktop/#connect

- Why would I need desktop in browser?
- Desktop in browser or especially via VNC-client gives usually faster response time than using `ssh -X`.
- It is offered in Browser, to give everyone anywhere the possibility to use it. Other ways of providing a desktop environment would require most users to install separate tools (e.g. nomachine).
- Webinterface is mainly to give users, that prefer graphical tools over command line, the possibility to also use them on Puhti. Also it can be very handy to have e.g. QGIS available for checking some results of processing.

- Why "outdated" design/Desktop interface? I would like to have more desktop tools available.
- Focus on function, not looks.
- Focus on scientific tools, but ask if you need something specific general tool.
- Most users never see the Desktop, but use single applications instead.

### Puhti - containers

- How to build your own containers for Puhti if apptainer ( new name of singularity) not available on own computer or own computing setup so different from Puhti that containers are not fully portable?
- You can set up a cPouta instance and install apptainer there to build your image.
- Use remote builder: https://cloud.sylabs.io/builder

### Training session

- Training session at SYKE & LUKE wished
- CSC to Viikki to have presentation about Puhti web interface (targeted training)
- SYKE and LUKE to scout the interest and level of researchers

### Linux get started

- [Linux basics Tutorial for CSC](https://docs.csc.fi/support/tutorials/env-guide/overview/)
- [Material for CSC Computing Environment -course](https://csc-training.github.io/csc-env-eff/)
- [Tutorial - Login to Puhti with SSH (essential)](https://csc-training.github.io/csc-env-eff/hands-on/connecting/ssh-puhti.html)
- [Tutorial - Basic linux commands](https://csc-training.github.io/csc-env-eff/hands-on/linux_prerequisites/basic-linux-commands.html)
- [Tutorial - Basic file editing](https://csc-training.github.io/csc-env-eff/hands-on/linux_prerequisites/basic-file-editing.html)
- [LinuxSurvival](https://linuxsurvival.com/) - two first modules should be enough for starting with Puhti.
- CSC has also ready materials for keeping longer instructor-lead Linux courses, but no plans to keep an open CSC Linux 101 course now.
- [(Aaltos) One-page Linux crash course](https://scicomp.aalto.fi/scicomp/shell/)

### CSC service usage

- Possible reasons for not using CSC resources:
- Too difficult,
- Too small process/job,
- Don’t know where to start how to use it,
- No linux skills
-> Webinterface may help to make Puhti use "less scary"
-> See Linux links above

### Data transfer bottleneck

- Lots of data at SYKE "verkkolevy" and on researchers computers: upload to CSC and downloading to own computers takes too much time; faster tools?
- You can find all tools and tipps around data transfer here: https://docs.csc.fi/data/moving/
- Do not use WinSCP for Allas S3 uploads, the tool itself simply is slow.
- We will investigate on fast-lines

- Aalto and VTT moved to Microsoft cloud service; can data be directly transferred between Microsoft cloud and CSC?
- We will investigate :)

---

# CSC services cheat-sheet and links
[CSC service catalogue](https://research.csc.fi/en/service-catalog)

## Quicklinks

* Check out our [Geocomputing page](https://research.csc.fi/geocomputing)
* [Geoportti project](https://www.geoportti.fi)
* [Step by step instructions for getting started with CSC services](https://research.csc.fi/en/accounts-and-projects)
* [Find your account and project information on my.csc.fi](my.csc.fi)
* [Documentation](https://docs.csc.fi)
* [Tutorials](https://docs.csc.fi/support/tutorials/) ,
* [Geocomputing examples](https://github.com/csc-training/geocomputing)
* Geoinformatics and related course and presentation materials: https://research.csc.fi/gis-learning-materials

---

**Cloud services**

[General information page](https://research.csc.fi/cloud-computing)
[Documentation](https://docs.csc.fi/cloud/concepts/)

<img src="https://i.imgur.com/JbunQzA.png" alt="Infrastructure vs Platform as a service" width="400"/>

**Pouta (Virtual Machines (VM))**

* available on demand
* under own administration
* ideal for webserver / databases

**Rahti (Container)**

* for eg web applications, "(docker) image hub", ...

---
**Supercomputers**

-> High Performance Computing (HPC)
[General information page](https://research.csc.fi/csc-s-servers)
[Documentation](https://docs.csc.fi/computing/overview/)

* memory and CPU(/GPU) availability (software needs to make use of this!)
* mainly non-interactive
* resource knowledge
* pre-installed and maintained software

**Puhti**

[General information page](https://research.csc.fi/-/puhti)
[Documentation](https://docs.csc.fi/computing/systems-puhti/)

* Geoinformatics related software and Python and R packages pre-installed: https://docs.csc.fi/apps/#geosciences
* many geographic datasets available
* https://docs.csc.fi/data/datasets/spatial-data-in-csc-computing-env/

**Mahti**

[General information page](https://research.csc.fi/-/mahti)
[Documentation](https://docs.csc.fi/computing/systems-mahti/)

* more CPU than Puhti, also provides GPU, but much smaller software stack than Puhti

**LUMI**

[General information page](https://research.csc.fi/-/lumi)
[Documentation](https://docs.lumi-supercomputer.eu/generic/overview/)

* when Puhti/Mahti is not enough OR very large GPU projects OR companies OR large international projects
* Each project should install its own software stack: EasyBuild, Spack or Singularity/Apptainer container.

---

**CSC Notebooks**

[General information page](https://research.csc.fi/en/-/notebooks)
[Documentation](https://docs.csc.fi/cloud/csc_notebooks/)

www.notebooks.csc.fi
-> self-learning, collaboration, courses

---

**Object storage: Allas**

[General information page](https://research.csc.fi/-/allas)
[Documentation](https://docs.csc.fi/data/Allas/)

* data storage during project lifetime
* CSC account and project required
* access from other services and own computer
* some tools can read directly from Allas
* data is immutable
* Maximum size for free: 200TB

---

**Sharing research data: Fairdata (general)**

[General information page](https://www.fairdata.fi/en/)


---

**Sharing research data: EUDAT (general)**

[General information page](https://research.csc.fi/en/-/eudat-services)

* open for all
* 20 GB limit for dataset
* 10 GB per file
* currently only guaranteed for 2 years
* customizable (paid premium access)

---

**Sharing research data: Paituli (geospatial)**

[General information page](https://paituli.csc.fi/)

* spatial data download service
* open to anyone, unrestricted access
* includes historical versions of datasets
* publish own geospatial datasets -> URN
* webbased data preview
* not limited to Finland
* annual usage reports

---

**Digital preservation (long-term data storage)**

[General information page](https://research.csc.fi/-/digital-preservation-service

(Pitkäaikaissäilytys (PAS))

"Preservation of digital information for several decades or centuries, even though hardware, software and file formats become outdated"

-> Geoportti project: Definition of spatial data format

---

**Sensitive Data (SD) services**

[General information page](https://research.csc.fi/sensitive-data-services-for-research)

"Secure workspace for all phases of research"

- webinterface - on-demand - data-controller - always encrypted

* SD Connect: store and share
* SD Desktop: isolated, secure private cloud environment
* SD Submit: publish under controlled access (pilot)
* SD Apply: re-use (pilot)