Try   HackMD

CSC-GIS user support and networking event

This page: https://hackmd.io/@GeospatialCSC/support_event

We have tried to identify people from research institutes and universities that do some kind Geoinformatics and/or CSC support for their colleagues and/or are more experienced users of CSCs Geoinformatics related services and hope that with you, we reached the right person.

Thanks for supporting us supporting researchers!

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Introduction round

  • Name
  • Affiliation
  • what do you do?
  • Why are you here?
  • How do you use CSC resources in your work?

Notes by Kylli during introduction, feel free to edit (use "pen symbol" up top) or remove :)

  • Samantha Wittke; CSC/FGI(NLS)/Aalto/CodeRefinery; CSC geoinformatics support/ PhD student (EO time series); meeting YOU, getting to know your issues and ideas
  • Kylli Ek, CSC, GIS-coordinator, Paituli, HPC GIS support, courses-workshops, universities ArcGIS consortium
  • Katri Tegel, CSC, manager, GIS-background
  • Tatu Leppämäki, HY, Digital geography lab, PhD student, social media data analysis, contact person for my group. cPouta, data storage.
  • Eelis Halme, VTT/Aalto, PhD student, remote sensing team, have not used CSC resoruces yet, planning to do.
  • Matti Mõttus, VTT, forest remote sensing, biodiversity, used CSC HPC.
  • Arttu Kivimäki, FGI/NLS, Department of remote sensing and photogrammetry, working with optical and radar satellite data time series, use almost daily CSC HPC and Allas
  • Vuokko Heikinheimo, SYKE (past HY), urban land use research, I have used CSC services for many years, but not that much lately. We could use more.
  • Eetu Jutila, SYKE/Aalto, GIS-expert, change detection from Sentinel1, help internally other CSC researchers. Mammutti project: data storage and computing. Daily basis using Puhti and Allas.
  • Arto Viinikka, SYKE, remote sensing, I have used HPC a couple of times. Also in the Mammutti project.
  • Janne Mäyrä, SYKE, biodiversity center, computational background, I have used CSC resources all my working time at SYKE, computer vision projects, so high GPU needs.
  • Antti-Jussi Kieloaho, Luke (Natural Resources Institute Finland), Information system specialist; data science/data engineering, scientific computing coordinator; I do and help on GIS analysis and spatial modelling that is reason to be here; I occasionally use also CSC resources. Puhti, Mahti, cPouta, Paituli, Allas.
  • Matthieu Molinier, VTT, remote sensing team, change detection, optical data, deep learning, would like to use CSC more regularly

Services available at CSC

Compute & Analyze

  • cPouta / ePouta
  • Puhti / Mahti / LUMI
  • CSC Notebooks
  • Rahti

Store, Share & Publish Data

  • Allas
  • (EUDAT)
  • Fairdata IDA
  • Paituli

Sensitive Data (SD) services

What's new?

Puhti

  • puhti.csc.fi
    • Files, shells, RStudio/Jupyter/Desktop Applications
  • RHEL-8 update is coming, 4-5.10 ?:
  • Tykky for Apptainer container creation, based on:
    • Conda .yml file/ pip requirements.txt
    • Existing Docker image

Self-learning/examples/materials

CSC Notebooks

Paituli:

  • OGC-API support

Support

  • 'Z is not working as expected'
  • 'my code gives error Y '
  • 'can A be installed to Puhti?'
  • 'any advice how to do X?'
  • 'which service suits my needs?'
  • training/example wishes

-> servicedesk@csc.fi

Speed up your request

CSC can also be project partner/subcontractor

piloting currently: weekly user support sessions , every Wednesday at 14 in Zoom : https://ssl.eventilla.com/event/PP4WB

Acknowledging CSC in publications and reports

If you used any of our resources for your research, please acknowledge CSC and Geoportti in your publications, it is important for project continuation and funding reports. As an example, you can write:

"The authors wish to thank CSC - IT Center for Science, Finland (urn:nbn:fi:research-infras-2016072531) and the Open Geospatial Information Infrastructure for Research (Geoportti, urn:nbn:fi:research-infras-2016072513) for computational resources and support".

Course and event advertisements

(see also https://www.csc.fi/en/training#training-calendar)

CodeRefinery
CodeRefinery invites everyone interested in improving their software practice skills to join the CodeRefinery workshop on September 20.-22. and 27.-29. See https://coderefinery.github.io/2022-09-20-workshop/ for more details.

Using CSC environment efficiently - self learning
Highly suggested for anyone starting out (one may also choose only specific topics): https://csc-training.github.io/csc-env-eff/

Machine Learning for spatial data
7.-9. November: https://ssl.eventilla.com/event/VDK2b

GIS lecturer meeting
-> discussion and information on how to integrate CSC services in teaching and how CSC services may support your teaching
coming soon, let us know via giscoord@csc.fi if you or your colleagues are interested in joining

Weekly user support sessions
Everyone welcome to ask questions from our experts!
Every Wednesday at 14 in Zoom : https://ssl.eventilla.com/event/PP4WB (currently piloting)

Want to get latest news?

  • Mailing list: email reminders about events and news around GIS at CSC, sign up to: https://postit.csc.fi/sympa/info/gis-hpc
  • Related Twitter accounts to follow: @CSCfi , @CSChpc , @geoportti , @kylli_ek
  • service related e-mail lists (automatic subscription when using some services)

Discussion

  • Questions? Requests? Ideas?
  • Wishes for future (for you/your colleagues): courses, materials, examples, events, … ?
  • What can we do to make your life (wrt helping colleagues) easier?
  • How can we make servicedesk and support more approachable?
  • What kind of problems do your colleagues often need help with?

Notes from meeting plus few additions; feel free to edit, Thank you for raising all these points, we will pick them up internally and let you know if there are any interesting outcomes

Continuity

  • CSC project and its continuity is kind of unclear concept: what about continuation over projects, even when the project leader maybe changes? What to do with project and data when its lifetime ends, and how to integrate or move project to outside CSC?

Puhti scratch vs Allas

  • When is Puhti scratch directory the right place for data, and when is it Allas?
    • From docs: "The CSC supercomputers provide disk environments [e.g. scratch] for working with large datasets. These storage areas are however not intended for storing data that is not actively used. One of the main use cases of Allas is to store data while it is not actively used in the CSC supercomputers. When you start working, you stage in the data from Allas. And when the data is no longer actively used, it can be staged out to Allas. "

Portability

  • To create something for specific cloud environment is work; concerns about portability
    • One option is to use containers for your software environment
    • As much as possible use independent scripts for setting up your workflow and only run those from within sbatch-script
    • Linux subsystem for Windows is a great way to make use of Linux flexibility also on Windows computers

Research -> Operational

  • When does research turn operational from CSC perspective; what changes?

Overwhelming amount of services

  • Too many options for end user to choose from, it is difficult to know what to do as a single researcher starting to use CSC services
    • You can always send a message to servicedesk@csc.fi; also already when planning a project; explain your project and your needs as much as possible
    • Another option is to visit our weekly research user support session, currently every Wednesday at 14: https://ssl.eventilla.com/event/edit/102432
    • Computing: most of our Geoinformatics cases fit best to Puhti supercomputer ("need for more memory/CPU/GPU or parallelization")
    • Data storage: most usecases make use of Allas ("where can I share my data with project members, so that everyone can use data for computations in different environments")

Fairdata services

  • How to deal with data versions in Fairdata? Is there something implemented at CSC for versioning?
    • Fairdata has upload new version option to add new version as new file. No Github-kind of way to see fast what has been changed in the files.
    • DOI rules require that a new DOI is given to the new version.
    • URN is technically more flexible.

Puhti webinterface

  • Can same view of Jupyter/Desktop be shared among multiple users in web interface?

  • Why would I need desktop in browser?

    • Desktop in browser or especially via VNC-client gives usually faster response time than using ssh -X.
    • It is offered in Browser, to give everyone anywhere the possibility to use it. Other ways of providing a desktop environment would require most users to install separate tools (e.g. nomachine).
    • Webinterface is mainly to give users, that prefer graphical tools over command line, the possibility to also use them on Puhti. Also it can be very handy to have e.g. QGIS available for checking some results of processing.

  • Why "outdated" design/Desktop interface? I would like to have more desktop tools available.

    • Focus on function, not looks.
    • Focus on scientific tools, but ask if you need something specific general tool.
    • Most users never see the Desktop, but use single applications instead.

Puhti - containers

  • How to build your own containers for Puhti if apptainer ( new name of singularity) not available on own computer or own computing setup so different from Puhti that containers are not fully portable?

Training session

  • Training session at SYKE & LUKE wished
    • CSC to Viikki to have presentation about Puhti web interface (targeted training)
    • SYKE and LUKE to scout the interest and level of researchers

Linux get started

CSC service usage

  • Possible reasons for not using CSC resources:
    • Too difficult,
    • Too small process/job,
    • Don’t know where to start how to use it,
    • No linux skills
      -> Webinterface may help to make Puhti use "less scary"
      -> See Linux links above

Data transfer bottleneck

  • Lots of data at SYKE "verkkolevy" and on researchers computers: upload to CSC and downloading to own computers takes too much time; faster tools?

    • You can find all tools and tipps around data transfer here: https://docs.csc.fi/data/moving/
    • Do not use WinSCP for Allas S3 uploads, the tool itself simply is slow.
    • We will investigate on fast-lines

  • Aalto and VTT moved to Microsoft cloud service; can data be directly transferred between Microsoft cloud and CSC?

    • We will investigate :)

CSC services cheat-sheet and links

CSC service catalogue

Cloud services

General information page
Documentation

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Pouta (Virtual Machines (VM))

  • available on demand
  • under own administration
  • ideal for webserver / databases

Rahti (Container)

  • for eg web applications, "(docker) image hub", …

Supercomputers

-> High Performance Computing (HPC)
General information page
Documentation

  • memory and CPU(/GPU) availability (software needs to make use of this!)
  • mainly non-interactive
  • resource knowledge
  • pre-installed and maintained software

Puhti

General information page
Documentation

Mahti

General information page
Documentation

  • more CPU than Puhti, also provides GPU, but much smaller software stack than Puhti

LUMI

General information page
Documentation

  • when Puhti/Mahti is not enough OR very large GPU projects OR companies OR large international projects
  • Each project should install its own software stack: EasyBuild, Spack or Singularity/Apptainer container.

CSC Notebooks

General information page
Documentation

www.notebooks.csc.fi
-> self-learning, collaboration, courses

Object storage: Allas

General information page
Documentation

  • data storage during project lifetime
  • CSC account and project required
  • access from other services and own computer
  • some tools can read directly from Allas
  • data is immutable
  • Maximum size for free: 200TB

Sharing research data: Fairdata (general)

General information page

Sharing research data: EUDAT (general)

General information page

  • open for all
  • 20 GB limit for dataset
  • 10 GB per file
  • currently only guaranteed for 2 years
  • customizable (paid premium access)

Sharing research data: Paituli (geospatial)

General information page

  • spatial data download service
  • open to anyone, unrestricted access
  • includes historical versions of datasets
  • publish own geospatial datasets -> URN
  • webbased data preview
  • not limited to Finland
  • annual usage reports

Digital preservation (long-term data storage)

[General information page](https://research.csc.fi/-/digital-preservation-service

(Pitkäaikaissäilytys (PAS))

"Preservation of digital information for several decades or centuries, even though hardware, software and file formats become outdated"

-> Geoportti project: Definition of spatial data format

Sensitive Data (SD) services

General information page

"Secure workspace for all phases of research"

  • webinterface - on-demand - data-controller - always encrypted
  • SD Connect: store and share
  • SD Desktop: isolated, secure private cloud environment
  • SD Submit: publish under controlled access (pilot)
  • SD Apply: re-use (pilot)
Last changed by 
 
GeospatialCSC
Geospatial team of CSC
0
811

Read more

Spatial data analysis with R

Link to this page: siili.rahtiapp.fi/Rspatial_23

May 9, 2023
EO tutorial comments

Comments and additions to EO tutorial draft EO tutorial draft What about Forestry TEP, Sen2agri etc? Earth Observation guide This guide aims to help researchers to work with Earth Observation (EO) data using CSC's computing resources. The purpose of this guide is to give an overview of available options, so it would be easier to decide if CSC has suitable services for your your EO reseach. It also helps you find the right data and tools for raster data based EO tasks. If you are interested in the fundamentals of EO, please check the resources and further reading section. What are the benefits of using EO data?

Jan 17, 2023
EO tutorial draft

Table of contents TODO: author names (?) STAC paragraph edit data storage paragraph edit Allas usage, more info Learning material links * Geoportti mention

Jan 10, 2023
GeoML

Practical machine learning with spatial data :::info This document: https://hackmd.io/@GeospatialCSC/GeoML Slides Exercise material Computing environment: www.puhti.csc.fi ::: Schedule :::spoiler Monday 7.11.

Nov 1, 2022
Read more from GeospatialCSC
Published on HackMD