---
# System prepended metadata

title: OpenMX

---

OpenMX
==========
## I. GCC/Gfortran with OpenMPI with CUDA

GCC 11
OpenMPI 5.0.3
Intel MKL 2024.2.0

### Env setting
```bash=
ml tbb compiler-rt compiler openmpi/5.0.3
cd openmx3.9/source3.962
make install
```

### Makefile 
line 9-12
```makefile=
MKLROOT = /home/sysadmin/intel/mkl
CC = mpicc -O3 -ffast-math -fopenmp -Wno-error=implicit-function-declaration -Wl,--allow-multiple-definition -I${MKLROOT}/include/fftw
FC = mpif90 -O3 -ffast-math -fopenmp -fallow-argument-mismatch -Wl,--allow-multiple-definition
LIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -liomp5 -lpthread -lm -ldl
```
line 247-248
```makefile=
openmx:	$(OBJS)
	$(FC) $(OBJS) $(LIB) -lm  -Wno-amin -o openmx
```


需要`-Wno-error=implicit-function-declaration`才可以讓他不要一直抓C99以後禁止隱性宣告的問題

另外，還會出現
```bash=
ld: -liomp5: Not found
```
原因：libiomp5.so其實長在`.../intel/lib/intel64/` 裡面，但我們link的MKL庫位置在`.../intel/mkl/lib/intel64`
解決辦法：創建soft link到 .../intel/mkl/lib/intel64/libiomp5.so
```bash=
ln -s /home/sysadmin/intel/lib/intel64/libiomp5.so /home/sysadmin/intel/mkl/lib/intel64/libiomp5.so
```

最後還需要`-Wl,--allow-multiple-definition` 才不會在linking階段報錯


### Complete compilation
![image](https://hackmd.io/_uploads/SkyU1bi5kl.png)

### Test
![image](https://hackmd.io/_uploads/Byicy-i9yl.png)

### Result
![image](https://hackmd.io/_uploads/SkGnyZo51g.png)




## II. ICX/Ifort with IntelMPI
### Env setting
```bash=
ml tbb compiler-rt compiler mpi
export I_MPI_CC=icx
export I_MPI_FC=ifort
export I_MPI_F90=ifort

cd openmx3.9/source3.962
make install
```

fortran的部分會取用mpi module,這個時候會抓該mpi需要和fortran用的是同一個compiler 
因為這裡官方示範用的是ifort 所以我們要嘛改成用ICX自編一份OpenMPI with CUDA 
不然就是先用intel mpi擋著用一下

還有單改I_MPI_FC不夠 實際上在用的是mpif90


### Makefile 
Line: 9-12
```makefile=
MKLROOT = /home/sysadmin/intel/mkl
CC = mpicc -O3 -xHOST -qopenmp -Wno-error=implicit-function-declaration -I${MKLROOT}/include/fftw -diag-disable=10441
FC = mpif90 -O3 -xHOST -qopenmp -Wl,--allow-multiple-definition -diag-disable=10441
LIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_openmpi_lp64 -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lifcore -liomp5 -lpthread -lm -ldl
```

Line: 248-249
```makefile=
openmx:	$(OBJS)
	$(FC) $(OBJS) $(LIB) -lm  -nofor-main -o openmx
```



### Complete compilation

![image](https://hackmd.io/_uploads/SJQ8ibo91x.png)

### Test
![image](https://hackmd.io/_uploads/rJzdsWjc1e.png)

然後會報錯
```
*** The MPI_Comm_c2f() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[art2:103528] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
```
![image](https://hackmd.io/_uploads/r1QKsbicye.png)

爬stackoverflow之後，感覺是編譯過程中用到的MPI版本可能有點歧義 回去再維持好環境後重編看看
