---
# System prepended metadata

title: Python in Puhti
tags: [PythonGIS]

---

---

tags: PythonGIS
title: Python in Puhti

---

# Lesson 7: Python in Puhti
 
* [Slides about Python on Puhti supercomputer](https://a3s.fi/gis-courses/pythongis_2022/PythonGIS_in_Puhti.pdf)
* [Exercise materials](https://github.com/csc-training/geocomputing/tree/master/python/puhti)

## Links
* [Puhti documentation](https://docs.csc.fi/computing/running/getting-started/)
* [Puhti web interface](https://puhti.csc.fi/)
* [cPouta documentation](https://docs.csc.fi/cloud/pouta/pouta-what-is/)
* [CSC Notebooks for courses](https://notebooks-beta.rahtiapp.fi) 
* [CSC Computing Environment course](https://ssl.eventilla.com/enveff2022)
* [Geoscience related tools on Puhti](https://docs.csc.fi/apps/#geosciences)
* Most geopsatial python packages are available in [geoconda module](https://docs.csc.fi/apps/geoconda/)
* Questions? Missing software? Ask servicedesk@csc.fi


## Editing files in Puhti
If you need to edit a file in Puhti there are four main options:

1. Open a graphical tool in Puhti. For example
    * [Visual Studio code](https://docs.csc.fi/computing/webinterface/vscode/) 
    * [Jupyter](https://docs.csc.fi/computing/webinterface/jupyter/)
    * [Spyder](https://docs.csc.fi/apps/geoconda) with for example geoconda module
3. Edit files in Puhti with commandline tools: nano, vim etc.
4. Edit Puhti files files from your local PC with a tool that supports that, see [Developing scripts remotely](https://docs.csc.fi/support/tutorials/remote-dev/)
5. Edit files on your local PC and copy them to Puhti.


## Questions

* Do we have LiDAR packages in python? for example 3d point clouds to convert elvation model or canopy height model?
    * what tool would you use on your own computer for this?
        * currently only arcMap and ArcGIS pro 3D analysis. Also "lidR" R package.
        * lidR is available in Puhti
        * From Python we have at the moment: laspy, laxpy, pdal, likely more in future. Which you would need?
        * We had point cloud workshop some weeks ago, some new packages were suggested there: https://hackmd.io/vAsn2q-JSTiBszEdXbfTeg
    * Some tools that can work with LiDAR data that we have on Puhti
        * [PDAL](https://pdal.io/) /Python-pdal
        * [laspy](https://pythonhosted.org/laspy/tut_part_1.html)
        * [QGIS](https://docs.csc.fi/apps/qgis/) 
        * [PCL](https://docs.csc.fi/apps/pcl/)
        * [Whiteboxtools](https://docs.csc.fi/apps/whiteboxtools/) 
        * [lastools](https://docs.csc.fi/apps/lastools/)
        * [PCL/PCLpy](https://docs.csc.fi/apps/pcl/)
        * we are also looking into getting CloudCompare to work on Puhti, would that be of interest to anyone?
    * Something missing? -> servicedesk@csc.fi :)
    * Thank you 

* Do we have any packages in python for digital aerial photogrammetry to make 3D point clouds. Soemthing similar that pix4D does?
    * OpenDroneMap does at last partly similar processes as pix4D, but it is not Python.
    
* Is OpenDroneMap is open access?
    * OpenDroneMap is open source, so no license needed. https://docs.csc.fi/apps/opendronemap/
    * GeoPortti site has also instructions for installing OpenDroneMap web (has graphical interface) to cPouta: https://www.geoportti.fi/tools/instruments/ (at the end of page)
    * UEF has plans in GeoPortti project to improve usability of OpenDroneMap in CSC computing environment, so if interested in this (for example as test-user) please send us e-mail.


* Is it better, timewise, to use data directly from Allas, or transfer it to Puhti? 
    * Accessing data from Allas directly is a bit slower than accessing data on Puhti, but in general processing time it should not matter much.

* So, what is the difference in running the code using the .sh file and the arrow? Is the arrow running it in the login node? 
    * Arrow in VScode: runs the script within the interactive session that we started from the web interface
    * running the sh file from terminal with `sbatch filename.sh` submits a job to the computing node from the login node (-> login node shell) 

> make sure to run the `sbatch ...` command from the login shell and not within VSCode terminal

> multiprocessing can be really powerful, highly suggest to check and try it out :)



----