parallel h5py install

# Build h5py with MPI support reference:[How to build h5py with MPI support (against Parallel HDF5) on Linux, 2023](https://stackoverflow.com/questions/77207467/how-to-build-h5py-with-mpi-support-against-parallel-hdf5-on-linux-2023) 、 [Parallel HDF5](https://docs.h5py.org/en/stable/mpi.html)、[Parallel HDF5 on T3](https://hackmd.io/CVw_QLqlR3aEBJqdZOPV1A) #### Solve`h5py was built without MPI support, can't use mpio driver` --- ubuntu22.04 [安裝 Ubuntu Desktop](https://www.youtube.com/watch?v=YR6qRSE56tE&list=PLd5k7R9j29NpoCEmyIqzXRAmrpo1Pv_Yy&index=15&t=271s) do in anaconda(base) [How to Install Anaconda in Ubuntu 22.04](https://linuxhint.com/install-anaconda-ubuntu-22-04/) [Linux卸載Anaconda](https://blog.csdn.net/hang916/article/details/79530108) ## 1. Install openmpi [check](https://blog.csdn.net/qq_26822029/article/details/107930758) : gcc、g++ ``` gcc --version ``` ``` g++ --version ``` if no ``` sudo apt-get install gcc ``` ``` sudo apt-get install g++ ``` ### Go to openmpi official website(end with".tar.gz") [official website](https://www.open-mpi.org/software/ompi/v5.0/) ### Open terminal in download destination and extract, or directly extract: ``` tar xvzf openmpi-4.1.1.tar.gz ``` #### get "openmpi-4.1.1" folder，Open terminal ### Configure: error:[Could not run a simple Fortran program when trying to install the OpenMPI.](https://www.reddit.com/r/fortran/comments/sv53cr/could_not_run_a_simple_fortran_program_when/?rdt=60964) ``` CC=gcc FC=gfortran ./configure --prefix=/home/brian/lib/openmpi ``` ### Install: ``` sudo make all install ``` ### Open .bashrc: ``` sudo vim ~/.bashrc ``` #### add `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/brian/lib/openmpi/lib` to bottom of .bashrc [vim](https://linux.vbird.org/linux_basic/centos7/0310vi.php) ### Activate changes: ``` source ~/.bashrc ``` ![Screenshot from 2024-01-18 17-32-34](https://hackmd.io/_uploads/HkgPeO8Fa.png) ### Go to the example folder to test，compile(Or look online for examples): ``` make ``` or ``` mpicc -o hello_c hello_c.c ``` ### hello_c: ``` mpirun -np 4 hello_c ``` ![Screenshot from 2024-01-18 17-34-14](https://hackmd.io/_uploads/H1Q3gd8YT.png) #### HYDU_create_process:execvp error on file hello_c(No such file or directory)` and `error while loading shared libraries: libevent-2.1.so.7: cannot open shared object file:No such file or directory` ### libevent-2.1.so.7: ``` sudo apt-get install libevent-dev ``` --- ## 2. Install HDF5 ### Go to HDF5 official website(end with".tar.gz") [official website](https://www.hdfgroup.org/downloads/hdf5/source-code) ### Open terminal in download destination and extract, or directly extract: ``` tar xvzf hdf5-1.12.3.tar.gz ``` ### Install tree additional packages: ``` conda install gcc_linux-64 gxx_linux-64 gfortran_linux-64 ``` #### get "hdf5-1.14.2" folder，open terminal ### Set MPI-C as compiler: ``` export CC=mpicc ``` ### Turn on HDF5_MPI: ``` export HDF5_MPI="ON" ``` ### Change build config: ``` ./configure --enable-shared --enable-parallel --prefix=/home/brian/lib/hdf5 ``` ### Set default HDF5_dir: ``` export HDF5_DIR="/home/brian/lib/hdf5" ``` ### Compile and install: ``` make ``` ``` make install ``` ``` make check-install ``` ![Screenshot from 2024-01-18 17-52-53](https://hackmd.io/_uploads/SJ0Gr_8t6.png) --- ## 3. Install h5py with MPI and HDF5 build Based on HDF5 1.12.3 / OpenMPI 4.1.1 ``` sudo apt install openmpi-bin ``` ### Create and activate a new env: ``` conda create -n env1 python=3.9.5 conda activate env1 ``` ### Install openmpi in the env: ``` conda install openmpi ``` ### Set MPI-C as compiler: ``` export CC=mpicc ``` ### Turn on HDF5_MPI: ``` export HDF5_MPI="ON" ``` ### Set default HDF5_dir: ``` export HDF5_DIR="/home/brian/lib/hdf5" ``` ### Install h5py: (this also install numpy and mpi4py) ``` pip install --no-binary=h5py h5py ``` --- ## 4.Install other package ### install numpy、sympy、mpi4py: ``` pip install numpy mpi4py sympy ``` ``` # Name Version Build Channel _libgcc_mutex 0.1 main _openmp_mutex 5.1 1_gnu ca-certificates 2023.12.12 h06a4308_0 contourpy 1.2.0 pypi_0 pypi cycler 0.12.1 pypi_0 pypi fonttools 4.47.2 pypi_0 pypi h5py 3.10.0 pypi_0 pypi importlib-resources 6.1.1 pypi_0 pypi kiwisolver 1.4.5 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1 libffi 3.3 he6710b0_2 libgcc-ng 11.2.0 h1234567_1 libgomp 11.2.0 h1234567_1 libstdcxx-ng 11.2.0 h1234567_1 matplotlib 3.8.2 pypi_0 pypi mpi4py 3.1.5 pypi_0 pypi mpmath 1.3.0 pypi_0 pypi ncurses 6.4 h6a678d5_0 numpy 1.26.3 pypi_0 pypi openssl 1.1.1w h7f8727e_0 packaging 23.2 pypi_0 pypi pillow 10.2.0 pypi_0 pypi pip 23.3.1 py39h06a4308_0 pyparsing 3.1.1 pypi_0 pypi python 3.9.5 h12debd9_4 python-dateutil 2.8.2 pypi_0 pypi readline 8.2 h5eee18b_0 setuptools 68.2.2 py39h06a4308_0 six 1.16.0 pypi_0 pypi sqlite 3.41.2 h5eee18b_0 sympy 1.12 pypi_0 pypi tk 8.6.12 h1ccaba5_0 tzdata 2023d h04d1e81_0 wheel 0.41.2 py39h06a4308_0 xz 5.4.5 h5eee18b_0 zipp 3.17.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0 ``` --- ## 5. test parallel h5py ### test code (use `mpirun -n 4 python test.py`) ``` from mpi4py import MPI import h5py import numpy as np comm = MPI.COMM_WORLD rank = comm.Get_rank() # Create a file collectively with h5py.File('parallel_test.hdf5', 'w', driver='mpio', comm=comm) as f: # Create a dataset collectively dset = f.create_dataset('test', (4,), dtype='i') # Each process writes its rank to the dataset dset[rank] = rank # All processes need to wait for others to finish writing comm.Barrier() # Read the dataset collectively with h5py.File('parallel_test.hdf5', 'r', driver='mpio', comm=comm) as f: data = np.array(f['test']) ``` ![image](https://hackmd.io/_uploads/rkoJ6j9FT.png) ### Install HDFView ``` sudo apt-get install hdfview ``` ### hdf5 file ``` h5dump parallel_test.hdf5 ``` ![image](https://hackmd.io/_uploads/S1svTj9Yp.png) ## 6.Other reference ### module 'h5py' has no attribute 'File' #### pip、conda混用出錯 ``` pip uninstall h5py conda install h5py ``` [install Python 3.11](https://www.youtube.com/watch?v=w7v4CZt1po8&list=PLd5k7R9j29NpoCEmyIqzXRAmrpo1Pv_Yy&index=16) [The following packages are not available from current channels](https://bobbyhadz.com/blog/packages-not-found-error-the-following-packages-are-not-available) [Failed building wheel for mpi4py](https://blog.51cto.com/u_9453611/5569621) [module 'h5py' has no attribute 'File' 解決辦法](https://blog.csdn.net/wangdong2017/article/details/82883044) [name 'mpi4py' is not defined](https://stackoverflow.com/questions/56633981/h5py-file-function-throws-nameerror-for-mpi4py) [Configuring H5py with mpio using Anaconda on Windows](https://stackoverflow.com/questions/58345261/configuring-h5py-with-mpio-using-anaconda-on-windows) [hdf5文件視覺化](https://juejin.cn/s/hdf5%E6%96%87%E4%BB%B6%E5%8F%AF%E8%A7%86%E5%8C%96) [Install hdfview on Ubuntu 22.04](https://installati.one/install-hdfview-ubuntu-22-04/?expand_article=1) [Ubuntu下phg的安裝](https://blog.csdn.net/lusongno1/article/details/84393821) [Ubuntu安裝openmpi](https://blog.csdn.net/songbaiyao/article/details/72858184) [AttributeError: module 'numpy' has no attribute 'typeDict'](https://stackoverflow.com/questions/74852225/attributeerror-module-numpy-has-no-attribute-typedict) [RuntimeWarning: mpi4py.MPI.File size changed, may indicate binary incompatibility.](https://fenicsproject.discourse.group/t/runtimewarning-mpi4py-mpi-file-size-changed-may-indicate-binary-incompatibility-expected-32-from-c-header-got-40-from-pyobject-def-compile-class-cpp-data-mpi-comm-mpi-comm-world/6496)