---
# System prepended metadata

title: LAMMPS
tags: [SCC]

---

# LAMMPS
[toc]

## Tips
1. **最重要的：所有人都要用一樣的compiler！！**
2. 每新安裝一個dependency 記得 export path
3. 如果發現是路徑有問題直接開新的terminal重裝
4. 有用cmake開一個folder用來build，失敗就刪掉重來
5. configure 和 install 的 folder 分開
   - build: for configuration
   - opt: for installation

## Setup for build

```bash
mkdir lammps
cd lammps
mkdir build opt
```
   
### Paths
```bash
ROOT=$HOME/lammps
BUILD=$HOME/lammps/build
OPT=$HOME/lammps/opt

HDF5=hdf5-1.8.18
PNETCDF=pnetcdf-1.12.3
NETCDF=netcdf-c-4.8.1
NETCDF_FORTRAN=netcdf-fortran-4.6.1
PYTHON=Python-3.12.0
ADIOS=ADIOS-2.8.0
VORO=voro++-0.4.6
LAMMPS=lammps-2Aug2023
```
### <.bashrc>
> Compiler: GCC 11.4.0
```bash
export PATH=/home/scteam06/openmpi/bin:$PATH
export LD_LIBRARY_PATH=/home/scteam06/openmpi/lib:$LD_LIBRARY_PATH

export HDF5_DIR=/home/scteam06/lammps/opt/hdf5-1.8.18
export PNETCDF_DIR=/home/scteam06/lammps/opt/pnetcdf-1.12.3
export NETCDF_DIR=/home/scteam06/lammps/opt/netcdf-c-4.8.1
export NETCDF_FORTRAN_DIR=/home/scteam06/lammps/opt/netcdf-fortran-4.6.1
export ADIOS_DIR=/home/scteam06/lammps/build/ADIOS-2.8.0
export VORO_DIR=/home/scteam06/lammps/opt/voro++-0.4.6
export ZFP_DIR=/home/scteam06/lammps/opt/zfp

export PATH=$HDF5_DIR/bin:$PNETCDF_DIR/bin:$NETCDF_DIR/bin:$ADIOS_DIR/bin:$VORO_DIR/bin:$ZFP_DIR/bin:$PATH
export LD_LIBRARY_PATH=$HDF5_DIR/lib:$PNETCDF_DIR/lib:$NETCDF_DIR/lib:$ADIOS_DIR/lib:$VORO_DIR/lib:$ZFP_DIR/lib:$LD_LIBRARY_PATH
export C_INCLUDE_PATH=$HDF5_DIR/include:$PNETCDF_DIR/include:$NETCDF_DIR/include:$ADIOS_DIR/include:$VORO_DIR/include:$ZFP_DIR/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=$HDF5_DIR/include:$PNETCDF_DIR/include:$NETCDF_DIR/include:$ADIOS_DIR/include:$VORO_DIR/include:$ZFP_DIR/include:$CPLUS_INCLUDE_PATH
export LIBRARY_PATH=$HDF5_DIR/lib:$PNETCDF_DIR/lib:$NETCDF_DIR/lib:$ADIOS_DIR/lib:$VORO_DIR/lib:$ZFP_DIR/lib:$LIBRARY_PATH

export PATH=/home/scteam06/lammps/build/lammps-2Aug2023/bin:$PATH
export LD_LIBRARY_PATH=/home/scteam06/lammps/build/lammps-2Aug2023/lib:$LD_LIBRARY_PATH
```
```bash
source ~/.bashrc
```

## Dependencies
### Zlib 1.2.11
```bash
cd $BUILD
wget https://nchc.dl.sourceforge.net/project/libpng/zlib/1.2.11/zlib-1.2.11.tar.gz
tar -xvf zlib-1.2.11.tar.gz
cd zlib-1.2.11

CC=mpicc CXX=mpicxx  \
./configure \
--prefix=$HOME/opt/zlib-1.2.11  #zlib路徑不影響

make -j
sudo make install
```

### HDF5 1.8.18
```bash
cd $BUILD
wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8/hdf5-1.8.18/src/hdf5-1.8.18.tar.gz
tar xf $HDF5.tar.gz
cd $HDF5

CC=mpicc CXX=mpicxx FC=mpifort \
./configure \
--prefix=$OPT/$HDF5 \
--enable-shared \
--enable-parallel 

make -j
make install
```

![image](https://hackmd.io/_uploads/HJiogLc70.png =600x)


### PNETCDF 1.12.3
```bash
cd $BUILD
wget https://parallel-netcdf.github.io/Release/pnetcdf-1.12.3.tar.gz
tar xf $PNETCDF.tar.gz 
cd $PNETCDF

CC=mpicc CXX=mpicxx FC=mpifort \
./configure \
--enable-shared \
--enable-subfiling \
--prefix=$OPT/$PNETCDF

make -j
make install
```
![image](https://hackmd.io/_uploads/Hk2VML9XA.png =600x)


### NETCDF-C 4.8.1
```bash
cd $BUILD
wget https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.8.1.tar.gz
tar xf v4.8.1.tar.gz
cd $NETCDF

CC=mpicc CXX=mpicxx FC=mpifort \
./configure \
--enable-pnetcdf \
--enable-hdf5 \
--enable-shared \
--prefix=$OPT/$NETCDF

make -j
make install
```
![image](https://hackmd.io/_uploads/rkumQ8cQA.png =600x)


### NETCDF-fortran 4.6.1
```bash
cd $BUILD
wget https://downloads.unidata.ucar.edu/netcdf-fortran/4.6.1/$NETCDF_FORTRAN.tar.gz
tar xf $NETCDF_FORTRAN.tar.gz
cd $NETCDF_FORTRAN

CC=mpicc CXX=mpicxx FC=mpifort \
./configure \
--prefix=$OPT/$NETCDF_FORTRAN \
--enable-shared \

make -j
make install
```
![image](https://hackmd.io/_uploads/HJYxVI9QA.png =500x)

### ADIOS 2.8.0
```bash
cd $BUILD
git clone -b v2.8.0 https://github.com/ornladios/ADIOS2.git $ADIOS
cd $ADIOS
mkdir build
cd build 

CCC=mpicc CXX=mpicxx FC=mpifort \
cmake .. \
-DMPI_Fortran_COMPILER=mpifort \
-DCMAKE_Fortran_COMPILER=mpifort \
-DCMAKE_C_COMPILER=mpicc \
-DCMAKE_CXX_COMPILER=mpicxx \
-DCMAKE_CXX_FLAGS="-O3 -fpic" \
-DADIOS2_USE_MPI=ON \
-DADIOS2_USE_Endian_Reverse=ON \
-DADIOS2_USE_Fortran=ON \
-DADIOS2_RUN_INSTALL_TEST=OFF \
-DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY \
-DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY \
-DCMAKE_INSTALL_PREFIX=$BUILD/$ADIOS \
-DADIOS2_INSTALL_GENERATE_CONFIG=OFF \
-DHDF5_DIR=/home/scteam06/lammps/opt/hdf5-1.8.18 
-DADIOS2_BUILD_HDF5=ON  #不要用 make -j

make install
```

### VORONOI
```bash
cd $ROOT
wget https://math.lbl.gov/voro++/download/dir/$VORO.tar.gz
tar xf $VORO.tar.gz
cd $VORO

# you have to modify config.mk first
# modify：CXX, CFLAG, INSTALL prefix
# CXX=mpicxx
# CFLAG=-ansi -O3 -march=cascadelake
# $BUILD=`your built loaction`
# $VORO=voro++-0.4.6
# PREFIX=${BUILD}/${VORO}
# Build shared library by yourself

make -j
make install
```

### Visualization tools
- JPEG
  ```bash
  sudo apt-get install libjpeg-dev
  ```
- FFMPEG
  ```bash
  sudo apt-get install ffmpeg
  ```

### Others
- ZFP
  ```bash
  cd $BUILD
  wget https://github.com/LLNL/zfp/archive/refs/tags/0.5.5.tar.gz
  tar -xzf 0.5.5.tar.gz
  cd zfp-0.5.5
  mkdir build
  cd build
  cmake .. -DCMAKE_INSTALL_PREFIX=$OPT/zfp
  make -j
  make install
  ```
- LAPACK
  ```bash
  sudo apt-get install liblapack-dev
  ```

## Build LAMMPS
> Version: 2Aug2023
```bash
OPTFLAGS="-march=cascadelake -O2"
CC=mpicc
CXX=mpicxx
FC=mpifort

CFLAGS="-fPIC -march=cascadelake -fopenmp -Wrestrict -DLMP_INTEL_USELRT -DLMP_USE_MKL_RNG -Df2cFortran $OPTFLAGS"
CXXFLAGS="-fPIC -march=cascadelake -fopenmp -Wrestrict -DLMP_INTEL_USELRT -DLMP_USE_MKL_RNG -Df2cFortran -std=c++17 $OPTFLAGS"
FCFLAGS="-fPIC -march=cascadelake -Wrestrict -DLMP_INTEL_USELRT -DLMP_USE_MKL_RNG $OPTFLAGS"
LDFLAGS="$OPTFLAGS -L$MKLROOT/lib/intel64 -ltbbmalloc -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -L$ZFP_DIR/lib -lzfp"

cmake ../cmake \
-DCMAKE_INSTALL_PREFIX=$BUILD/$LAMMPS \
-DBUILD_SHARED_LIBS=yes \
-DINTEL_ARCH=cpu \
-DFFT=MKL \
-DFFT_MKL_THREADS=on \
-DWITH_GZIP=yes \
-DPKG_ADIOS=yes \
-DADIOS2_DIR=$ADIOS_DIR \
-DPKG_NETCDF=yes \
-DBUILD_MPI=yes \
-DBUILD_OMP=yes \
-DLAMMPS_MACHINE=mpi \
-DPKG_OPENMP=yes \
-DPKG_OPT=yes \
-DPYTHON_EXECUTABLE=$(which python3) \
-DPKG_ML-QUIP=yes \
-DPKG_ML-HDNNP=yes \
-DPKG_VORONOI=yes \
-DWITH_JPEG=yes \
-DLAMMPS_FFMPEG=yes

make -j
make install
```


- Get `result.txt` (`lmp_mpi`就在 /lammps-2Aug2023/build)
  ```bash
  lmp_mpi -help > result.txt
  ```
  ![image](https://hackmd.io/_uploads/rkxOKgTXC.png =500x)


## Run LAMMPS
### Lennard Jones Simulation
- `lammps/bench/in.lj`: set *run = 10000*
- nproc = 8
```bash
mpirun -np 8 lmp_mpi -in in.lj
```
![image](https://hackmd.io/_uploads/H1lH7clHA.png =600x)
![image](https://hackmd.io/_uploads/SJDD7cgHA.png =600x)


### LAMMPS.out
Output with `time` command
```bash
/usr/bin/time -v mpirun -np 4 lmp_mpi -in in.lj > LAMMPS.out 2>&1
```
![image](https://hackmd.io/_uploads/ByDcgWCXR.png =600x)

## Visualization
### JPG
In <in.lj>, add:
```
dump            1 all image 100 image.*.jpg type type &
                zoom 1.6 adiam 1.0
dump_modify     1 pad 5
```
Since the simulation runs for 10000 timesteps, and an image will be created every 100 timesteps, we should get 100 images in total (one for each of the following timesteps: 0, 100, 200, ..., 9900).

- **image.00000**

  ![image.00000](https://hackmd.io/_uploads/BkyREOy4C.jpg =300x)

- **image.05000**

  ![image.05000](https://hackmd.io/_uploads/HJzR4O1EC.jpg =300x)

- **image.10000**

  ![image.10000](https://hackmd.io/_uploads/B1BRN_yV0.jpg =300x)

### FFMPEG
In <in.lj>, add:
```
dump            1 all movie 100 movie.mp4 type type &
                zoom 1.0 adiam 1.0
dump_modify     1 pad 5
```
This script creates [movie.mp4](https://drive.google.com/uc?export=download&id=1HHHyYj2UqOjjoehxxrv7bymTyAomf4jC), capturing frames every 100 timesteps of the simulation.


## Performance
因為LAMMPS跑出來就會提供滿詳盡的performance數據，且較為直覺，所以雖然我中間也有用vtune測試過，後來還是以它本來的結果為主。

![image](https://hackmd.io/_uploads/HJ1XcclrA.png =400x)

Lennard-Jones(LJ) potential這個數學模型是用來描述兩個電中性的分子或原子間交互作用位能，因此我們主要關心的結果就是圖中的的幾個物理量：

![image](https://hackmd.io/_uploads/SJiU55eSR.png =600x)

這是最開始的<in.lj>得到的模擬結果(10000 steps)，可以發現：
1. 溫度(Temp)明顯降低，表示系統趨於穩定(平衡)。
2. 由位能(E_pair)變大可知原子之間的距離變遠了。
3. 分子能量維持0，因為過程中沒有分子交互作用。
4. 總能量稍微降低(幾乎不變)，可能是平衡過程難免有能量散失。
5. 壓力由負的轉正，表示系統壓縮的狀態放鬆了。

因此接下來的優化過程中，要注意：
- 不能調整LJ相關的參數 `pair_style` 和 `pair_coeff`。
  ```
  pair_style lj/cut 2.5
  pair_coeff 1 1 1.0 1.0 2.5
  ```
  $$
  V(r) = 4\epsilon \left[ \left( \frac{\sigma}{r} \right)^{12} - \left( \frac{\sigma}{r} \right)^6 \right]
  $$
  - $\epsilon = 1.0$ (depth of the potential well)
  - $\sigma = 1.0$ (finite distance where potential is zero)
  - `2.5`: cutoff distance for the potential
- 結果得到的數值應保持一致，以確保在提升效能的同時模擬是精確的。
   
Performance主要看的是"tau/day", "timesteps/s", "Matom-step/s"CPU使用率。而由圖可看出該模擬最大的瓶頸就是在"pairing"這部份，"communication"也有進步空間。了解這些情況後就可以開始優化了。

![image](https://hackmd.io/_uploads/Sy7ZfjlHC.png =600x)

### Number of MPI ranks (np)
![Figure_1](https://hackmd.io/_uploads/ByFKVjlS0.png =600x)
> np = 8 時整體效率最好

### Number of OpenMP threads per MPI rank
![Figure_2](https://hackmd.io/_uploads/ryHALsxr0.png =600x)
> n = 1, np = 8 時整體效率最好

### Modify <in.lj> script
- Atom sorting on/off
   | Sort Interval    | Performance (tau/day) | Timesteps/s | Matom-step/s | CPU Usage (%) |
   |------------------|-----------------------|-------------|--------------|---------------|
   | 0 1.0            | 157222.225            | 363.940     | 11.646       | 96.7          |
   | 0 2.0            | 161417.419            | 373.651     | 11.957       | 95.9          |
   | 50 2.0           | 170291.700            | 394.194     | 12.614       | 96.4          |
   | 100 2.0          | 168046.347            | 388.996     | 12.448       | 96.2          |
   - **Performance (tau/day)** 隨著 `sort interval` 的增加而提高，特別是在 `50 2.0` 時達到最高值。
   - **Timesteps/s** 和 **Matom-step/s** 也在 `50 2.0` 時達到最高值，這表明此配置在這些條件下性能最佳。
   - **CPU Usage** 在各種配置下變化不大，保持在 95-97% 之間。
- Newton flag on/off (預設是on)：發現off之後效能退步了。
- `processors * * *`: LAMMPS可以自動偵測最佳的prossessor分配方式，但實驗發現影響不大。
- **Neighbor list**: 
  ```
  neighbor	0.05 bin
  neigh_modify	delay 0 every 100 check yes
  ```
  ![Figure_3](https://hackmd.io/_uploads/r1Ld9sgrA.png  =600x)
  - `neigh_modify	delay 0 every 100 check yes` : 提高更新頻率可以減少neighbor的重建次數，從而減少計算開銷，提升性能。然而，更新頻率過低可能導致鄰居列表過時，增加不必要的計算。從結果看來，every 100 的性能優於 every 20，說明在這個模擬中，減少鄰居列表的更新次數更有利於性能提升。
  - `neighbor	0.05 bin`: bin 尺寸影響鄰居查找的效率。較小的 bin 尺寸可以減少每個 bin 中的原子數量，從而加快鄰居查找速度。從數據看來，將 bin 尺寸從 0.2 減少到 0.05，性能顯著提升。這說明較小的 bin 尺寸在這個模擬中更加有效，能夠顯著減少計算時間。
  - 綜合考慮，最佳的配置是 delay 0 every 100 check yes 和 neighbor 0.05 bin，性能指標（tau/day、timesteps/s、Matom-step/s）均達到最高，==**整體效能提升了 42.16%**==。
    ![image](https://hackmd.io/_uploads/r1-9iolSR.png =600x)
