--- tags: meetings title: Echolab shared notes --- # Echolab notes and further reading **Attending:** Echolab: Janne, Ian, Ilja CSC: Katri, Kylli, Samantha ## Notes from discussion * Hyperspectral data processing requires commercially licensed software -> possibilities at CSC: Pouta cloud * you set up own virtual machine (resources available depend on chosen "flavor" -> https://docs.csc.fi/cloud/pouta/vm-flavors-and-billing/#cpouta-flavors) which you can connect to via terminal or desktop , see https://docs.csc.fi/cloud/pouta/ for all pouta related info * you install your software (linux!) there with all the license stuff etc; it is handled like a separate computer * you can share access to that virtual machine with colleagues from same (CSC)project, they can then also use your installed software * Kylli told about the instructions for setting up a "similar software", those can be found here: https://github.com/csc-training/geocomputing/tree/master/pouta/metashape_with_VNC * CSC Notebooks for remote sensing course * https://notebooks.csc.fi * possibility for Jupyter and RStudio * access for students via Haka (no extra account needed) * students are invited to working environment via join code * possibilities for custom environment * help from GIS team and notebooks team available via servicedesk * Step by step instructions and guidelines: https://docs.csc.fi/cloud/csc_notebooks/ * installing own Python packages on Puhti * make sure to check if the packages are available somewhere before: https://docs.csc.fi/apps/python/#available * if you want to add to an existing module: https://docs.csc.fi/apps/python/#installing-python-packages-to-existing-modules * if you want to create completely own environment: https://docs.csc.fi/apps/python/#creating-your-own-python-environments * general own/licenced software in CSC services * **Linux only**, with very few exceptions * containers are a great way to do a lot of this in different environments: https://docs.csc.fi/computing/containers/overview/ * possibility/ feasibility on case-by-case basis; ask from servicedesk@csc.fi --- ## Quicklinks * Check out our [Geocomputing page](https://research.csc.fi/geocomputing) * [Geoportti project](https://www.geoportti.fi) * [Step by step instructions for getting started with CSC services](https://research.csc.fi/en/accounts-and-projects) * [Find your account and project information on my.csc.fi](my.csc.fi) * [Documentation](https://docs.csc.fi) * [Tutorials](https://docs.csc.fi/support/tutorials/) , * [Geocomputing examples](https://github.com/csc-training/geocomputing) * Geoinformatics and related course and presentation materials: https://research.csc.fi/gis-learning-materials --- ## Course and event advertisements (see also https://www.csc.fi/en/training#training-calendar) ### CodeRefinery CodeRefinery invites everyone interested in improving their software practice skills to join the CodeRefinery workshop on September 20.-22. and 27.-29. See https://coderefinery.github.io/2022-09-20-workshop/ for more details. ### Using CSC environment efficiently - self learning Highly suggested for anyone starting out (one may also choose only specific topics): https://csc-training.github.io/csc-env-eff/ ### Machine Learning for spatial data 7.-9. November : https://ssl.eventilla.com/event/VDK2b ### GIS lecturer meeting -> discussion and information on how to integrate CSC services in teaching and how CSC services may support your teaching coming soon, let us know via giscoord@csc.fi if you or your colleagues are interested in joining ### Weekly user support sessions Everyone welcome to ask questions from our experts! Every Wednesday at 14 in Zoom : https://ssl.eventilla.com/event/PP4WB (currently piloting) --- ## Support * 'Z is not working as expected' * 'my code gives error Y ' * 'can A be installed to Puhti?' * 'any advice how to do X?' * 'which service suits my needs?' * training/example wishes -> servicedesk@csc.fi [Speed up your request](https://docs.csc.fi/support/support-howto/) CSC can also be project partner/subcontractor piloting currently: **weekly user support sessions** , every Wednesday at 14 in Zoom : https://ssl.eventilla.com/event/PP4WB --- ## Acknowledging CSC in publications and reports If you used any of our resources for your research, please acknowledge CSC and Geoportti in your publications, it is important for project continuation and funding reports. As an example, you can write: "The authors wish to thank CSC - IT Center for Science, Finland (urn:nbn:fi:research-infras-2016072531) and the Open Geospatial Information Infrastructure for Research (Geoportti, urn:nbn:fi:research-infras-2016072513) for computational resources and support". --- ## CSC service cheat sheet ### Cloud services [General information page](https://research.csc.fi/cloud-computing) [Documentation](https://docs.csc.fi/cloud/concepts/) <img src="https://i.imgur.com/JbunQzA.png" alt="Infrastructure vs Platform as a service" width="400"/> #### Pouta (Virtual Machines (VM)) * available on demand * under own administration * ideal for webserver / databases #### Rahti (Container) * for eg web applications, "(docker) image hub", ... --- ### Supercomputers -> High Performance Computing (HPC) [General information page](https://research.csc.fi/csc-s-servers) [Documentation](https://docs.csc.fi/computing/overview/) * memory and CPU(/GPU) availability (software needs to make use of this!) * mainly non-interactive * resource knowledge * pre-installed and maintained software #### Puhti [General information page](https://research.csc.fi/-/puhti) [Documentation](https://docs.csc.fi/computing/systems-puhti/) [Web interface](https://puhti.csc.fi) * Geoinformatics related software and Python and R packages pre-installed: https://docs.csc.fi/apps/#geosciences * many geographic datasets available * https://docs.csc.fi/data/datasets/spatial-data-in-csc-computing-env/ #### Mahti [General information page](https://research.csc.fi/-/mahti) [Documentation](https://docs.csc.fi/computing/systems-mahti/) * more CPU than Puhti, also provides GPU, but much smaller software stack than Puhti #### LUMI [General information page](https://research.csc.fi/-/lumi) [Documentation](https://docs.lumi-supercomputer.eu/generic/overview/) * when Puhti/Mahti is not enough OR very large GPU projects OR companies OR large international projects --- ### CSC Notebooks [General information page](https://research.csc.fi/en/-/notebooks) [Documentation](https://docs.csc.fi/cloud/csc_notebooks/) www.notebooks.csc.fi -> self-learning, collaboration, courses --- ### Object storage: Allas [General information page](https://research.csc.fi/-/allas) [Documentation](https://docs.csc.fi/data/Allas/) * data storage during project lifetime * CSC account and project required * access from other services and own computer * some tools can read directly from Allas * data is immutable * Maximum size for free: 200TB --- ### Sharing research data: Fairdata (general) [General information page](https://www.fairdata.fi/en/) ![](https://i.imgur.com/4tJlU5n.png) --- ### Sharing research data: EUDAT (general) [General information page](https://research.csc.fi/en/-/eudat-services) * open for all * 20 GB limit for dataset * 10 GB per file * currently only guaranteed for 2 years * customizable (paid premium access) --- ### Sharing research data: Paituli (geospatial) [General information page](https://paituli.csc.fi/) * spatial data download service * open to anyone, unrestricted access * includes historical versions of datasets * publish own geospatial datasets -> URN * webbased data preview * not limited to Finland * annual usage reports --- ### Digital preservation (long-term data storage) [General information page](https://research.csc.fi/-/digital-preservation-service (Pitkäaikaissäilytys (PAS)) "Preservation of digital information for several decades or centuries, even though hardware, software and file formats become outdated" -> Geoportti project: Definition of spatial data format --- ### Sensitive Data (SD) services [General information page](https://research.csc.fi/sensitive-data-services-for-research) "Secure workspace for all phases of research" - webinterface - on-demand - data-controller - always encrypted * SD Connect: store and share * SD Desktop: isolated, secure private cloud environment * SD Submit: publish under controlled access (pilot) * SD Apply: re-use (pilot)