OpenMX ========== ## I. GCC/Gfortran with OpenMPI with CUDA GCC 11 OpenMPI 5.0.3 Intel MKL 2024.2.0 ### Env setting ```bash= ml tbb compiler-rt compiler openmpi/5.0.3 cd openmx3.9/source3.962 make install ``` ### Makefile line 9-12 ```makefile= MKLROOT = /home/sysadmin/intel/mkl CC = mpicc -O3 -ffast-math -fopenmp -Wno-error=implicit-function-declaration -Wl,--allow-multiple-definition -I${MKLROOT}/include/fftw FC = mpif90 -O3 -ffast-math -fopenmp -fallow-argument-mismatch -Wl,--allow-multiple-definition LIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -liomp5 -lpthread -lm -ldl ``` line 247-248 ```makefile= openmx: $(OBJS) $(FC) $(OBJS) $(LIB) -lm -Wno-amin -o openmx ``` 需要`-Wno-error=implicit-function-declaration`才可以讓他不要一直抓C99以後禁止隱性宣告的問題 另外，還會出現 ```bash= ld: -liomp5: Not found ``` 原因：libiomp5.so其實長在`.../intel/lib/intel64/` 裡面，但我們link的MKL庫位置在`.../intel/mkl/lib/intel64` 解決辦法：創建soft link到 .../intel/mkl/lib/intel64/libiomp5.so ```bash= ln -s /home/sysadmin/intel/lib/intel64/libiomp5.so /home/sysadmin/intel/mkl/lib/intel64/libiomp5.so ``` 最後還需要`-Wl,--allow-multiple-definition` 才不會在linking階段報錯 ### Complete compilation ![image](https://hackmd.io/_uploads/SkyU1bi5kl.png) ### Test ![image](https://hackmd.io/_uploads/Byicy-i9yl.png) ### Result ![image](https://hackmd.io/_uploads/SkGnyZo51g.png) ## II. ICX/Ifort with IntelMPI ### Env setting ```bash= ml tbb compiler-rt compiler mpi export I_MPI_CC=icx export I_MPI_FC=ifort export I_MPI_F90=ifort cd openmx3.9/source3.962 make install ``` fortran的部分會取用mpi module,這個時候會抓該mpi需要和fortran用的是同一個compiler 因為這裡官方示範用的是ifort 所以我們要嘛改成用ICX自編一份OpenMPI with CUDA 不然就是先用intel mpi擋著用一下 還有單改I_MPI_FC不夠 實際上在用的是mpif90 ### Makefile Line: 9-12 ```makefile= MKLROOT = /home/sysadmin/intel/mkl CC = mpicc -O3 -xHOST -qopenmp -Wno-error=implicit-function-declaration -I${MKLROOT}/include/fftw -diag-disable=10441 FC = mpif90 -O3 -xHOST -qopenmp -Wl,--allow-multiple-definition -diag-disable=10441 LIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lifcore -liomp5 -lpthread -lm -ldl ``` Line: 248-249 ```makefile= openmx: $(OBJS) $(FC) $(OBJS) $(LIB) -lm -nofor-main -o openmx ``` ### Complete compilation ![image](https://hackmd.io/_uploads/SJQ8ibo91x.png) ### Test ![image](https://hackmd.io/_uploads/rJzdsWjc1e.png) 然後會報錯 ``` *** The MPI_Comm_c2f() function was called before MPI_INIT was invoked. *** This is disallowed by the MPI standard. *** Your MPI job will now abort. [art2:103528] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! ``` ![image](https://hackmd.io/_uploads/r1QKsbicye.png) 爬stackoverflow之後，感覺是編譯過程中用到的MPI版本可能有點歧義 回去再維持好環境後重編看看