owned this note   changed 3 years ago  
Published Linked with GitHub

Point cloud research in CSC computing environment, 08.02.2022

Used tools

Which software would help you get your work done on Puhti supercomputer? Zoom poll answers marked to each row.

  • QGIS 10x
  • R:
  • Python:
    • PCLpy 3x ,
    • geoconda
      • laspy 9x, python-pdal 4x
      • lidar 4x, pyntcloud 1x (added after workshop)
    • open3d 2x (can not be added at the moment to Puhti, may-be later)
    • pointcloudset 0x, whitebox-python 1x, pye57 1x (not currently available in Puhti, ask if you need)
  • lastools
    • Open source 9x
    • Closed source 5x
  • CloudCompare (added after workshop) 9x
  • Machine learning frameworks:
  • PDAL 4x
  • Matlab 4x
  • PCL 3x
  • Whiteboxtools 1x
  • Julia 1x
  • Not currently available in Puhti, in principle should be possible, ask if you need:
    • AMAPVox (voxelization) 0x
    • displaz 1x
  • Not currently available in Puhti, would need further feasibility study:
    • ROS (Robot operating system) environment with PCL, rviz 1x
  • Not possible in Puhti (Windows only):
    • ArcGIS Pro
    • Terrasolid tools
    • Fugroviewer

Open Finnish datasets

Which datasets are you using? Please vote on the Zoom poll and add here the datasets not included in the poll.

Use cases

Please add a x behind every use case that relates to your research. Please add also new use cases as needed.

  • Pointclouds to DEM/DTM/DSM 5x
  • Viewing point cloud data interactively 4x
  • Machine learning with lidar data 4x
  • Single tree detection from aerial lidar data 4x
  • Single tree volume estimation from terrestial lidar data 2x
  • Forest stand volume estimation from aerial lidar data 2x
  • Single tree species detection from aerial lidar data 2x
  • Tree delineation from multi-scan TLS data 2x
  • Pointclouds to voxels x
  • Leaf-wood separation for TLS data x
  • Data hosting/sharing/publication of large point clouds, tools like zenodo limited to small data and slow x
  • Creating hydrologically correct DEM's from point clouds x
  • Creating true orthophotos with the assisting of Lidar data for more accurate building edges x

Learning material

Questions and answers

Have you looked at lib Open3D, seems easy to use in Python for smallish clouds (as an alternative to PCLPY), or any other libs?

  • Open3D is very easy to use, if you can program in python.

Could you maybe also integrate PCL to Matlab or R if you want to integrate preprocessing or point classification or so and you don’t have lastools commercial licence?

  • PCL has Matlab package, for interfacing with PCL

Is MetaShape faster or just better than OpenDroneMap, for example? Which of them gives better results for the same set of images? Have you compared the obtained point cloud and orthophoto to e.g. Pix4Dmapper Pro?

  • MetaShape has nice graphical interface, easy to see data and mark ground control points.
  • OpenDroneMap has web interface, but not in Puhti.

General questions

I want to use software X in Puhti, what should I do?

  • Option 1: Ask CSC to install it to everybody. servicedesk@csc.fi
  • Option 2: Install yourself for your own project usage. Follow installation guidelines for RHEL/Centos. Tools with several dependecies might be difficult, consult CSC. See Tykky if you would like to use conda.

What is required from lidar software that it can be run on Puhti?

  • Available for Linux
  • Open source
  • OR licensed software might be possible with very simple licensing schemes (LasTools) or with floating licenses. Node-locked licenses are possible only in cPouta.

Can I run preprocessing of lidar with current tools?

  • Sensor related preprocessing is usually done with tools provided by sensor manufactor. These are not available in Puhti for general use. Depending on the licensing it might be possible to install these with own license for own project usage.

Many times when you process point clouds, you end up creating more data, classifications, subsets, indices. How do you manage these in CSC environment? Do you need to transfer the result data back to your own computer or is the quota you can have in CSC enough to keep it there?

  • In Puhti locally a project has by default 1 Tb of space. This can be increased if needed, 20-30 Tb should be possible with good reasons.
  • Really big data should be kept in Allas.

How do I move data to CSC?

  • From linux/Mac command line tools: scp, sftp, rsync etc.
  • Graphical FTP tools: FileZilla, WinSCP
  • Puhti web interface for smaller transfers
  • Allas: Cyberduck, rclone

I used easybuild (https://easybuild.io/) on my university's cluster for compiling dependencies from source. Examples of my dependencies is a specific python version, Tensorflow and Horovod. Is easybuild available on Puhti?

  • EasyBuild is not available in Puhti, but Tensorflow and Horovod are.
  • Puhti has Spack for similar purposes.

Can CSC help me share large point clouds (with other researchers) or other large datasets? I used funet before, is that best way?

  • Funet FileSender can be used for sending any type of file. The link is valid for ~2 weeks. Up to 300 Gb.
  • Allas can be used for longer term data sharing or bigger amounts of data.
  • IDA and Paituli are options for sharing final results as open data.
Select a repo