# HPC-I FAMIL build Answer
- [name=Jonathan Hsu]
[toc]
## Dependencies
### Variables
Modify with your own directory.
<!--
```bash=
export WORK_DIR=/work1/jonathan0hsu
export FAMIL_HOME=$WORK_DIR/ASC/FAMIL_v1
export SRC=$FAMIL_HOME/src
export PREFIX=$FAMIL_HOME/lib
export COMPILE_NP=4
```
-->
```bash=
export WORK_DIR=/home/jonathanhsu
export FAMIL_HOME=$WORK_DIR/FAMIL_v1
export SRC=$FAMIL_HOME/src
export PREFIX=$FAMIL_HOME/lib
export COMPILE_NP=4
```
### Intel compiler suit
- Intel oneAPI toolkit current version: 2025.1
- Fortran compiler: IFX
- MPI: Intel MPI
```bash=
# Download from intel website
sudo sh ./intel-oneapi-hpc-toolkit-2025.1.0.666_offline.sh
source /opt/intel/oneapi/setvars.sh
```
### Some tools
```bash=
sudo apt update
sudo apt install build-essential gcc g++ gfortran cmake curl libcurl4-openssl-dev flex csh
```
### Libraries
#### zlib
```bash=
cd $SRC
mkdir zlib-intel
wget https://zlib.net/zlib-1.3.1.tar.gz
tar xzf zlib-1.3.1.tar.gz --strip-components=1 -C zlib-intel
cd zlib-intel
./configure --prefix=$PREFIX/zlib-intel
make -j $COMPILE_NP && make install
export PATH=$PREFIX/bin/zlib-intel:$PATH
export LD_LIBRARY_PATH=$PREFIX/lib/zlib-intel:$LD_LIBRARY_PATH
```
#### hdf5 1.14.6
```bash=
cd $SRC
mkdir hdf5-intel
wget https://support.hdfgroup.org/releases/hdf5/v1_14/v1_14_5/downloads/hdf5-1.14.5.tar.gz
tar xzf hdf5-1.14.5.tar.gz --strip-components=1 -C hdf5-intel
cd hdf5-intel/hdf5-1.14.5
export CC=icx
export CXX=icpx
export FC=ifx
export F77=ifx
./configure --prefix=$PREFIX/hdf5-intel \
--with-zlib=$PREFIX/zlib-intel \
--enable-hl \
--enable-parallel \
--enable-fortran \
--enable-shared \
--disable-static \
CC=mpiicx CXX=mpiicpx FC=mpiifx F77=mpiifx
make -j $COMPILE_NP && make install
$PREFIX/hdf5-intel/bin/h5pcc -showconfig
```
#### netcdf
```bash=
cd $SRC
wget https://downloads.unidata.ucar.edu/netcdf-c/4.9.2/netcdf-c-4.9.2.tar.gz
mkdir netcdf-intel
tar xzf netcdf-c-4.9.2.tar.gz --strip-components=1 -C netcdf-intel
mkdir -p netcdf-intel/build && cd netcdf-intel/build
export I_MPI_CC=icx
export I_MPI_CXX=icpx
export I_MPI_F77=ifx
export I_MPI_FC=ifx
cmake .. \
-DHDF5_DIR=$PREFIX/hdf5-intel \
-DHDF5_INCLUDE_DIR=$PREFIX/hdf5-intel/include \
-DCMAKE_PREFIX_PATH=$PREFIX/hdf5-intel \
-DHDF5_ROOT=$PREFIX/hdf5-intel \
-DHDF5_LIBRARY=$PREFIX/hdf5-intel/lib/libhdf5.so \
-DHDF5_HL_LIBRARY=$PREFIX/hdf5-intel/lib/libhdf5_hl.so \
-DHDF5_VERSION_REQUIRED="1.14.0" \
-DCMAKE_C_COMPILER=$(which mpiicx) \
-DCMAKE_CXX_COMPILER=$(which mpiicpx) \
-DZLIB_ROOT=$PREFIX/zlib-intel \
-DCMAKE_INSTALL_PREFIX=$PREFIX/netcdf-intel \
-DENABLE_DAP=OFF \
-DENABLE_FORTRAN=ON
make -j $COMPILE_NP && make install
export NETCDFPATH=$PREFIX/netcdf-intel
export LD_LIBRARY_PATH=$PREFIX/netcdf-intel/lib:$LD_LIBRARY_PATH
# export LD_LIBRARY_PATH=$PREFIX/netcdf-intel/lib:$LD_LIBRARY_PATH
export PATH=$PREFIX/netcdf-intel/bin:$PATH
# nc-config --all
```
#### netcdf-f
```bash=
cd $SRC
wget https://downloads.unidata.ucar.edu/netcdf-fortran/4.6.1/netcdf-fortran-4.6.1.tar.gz
mkdir netcdf-fortran-intel
tar xzf netcdf-fortran-4.6.1.tar.gz --strip-components=1 -C netcdf-fortran-intel
mkdir -p $SRC/netcdf-fortran-intel/build && cd $SRC/netcdf-fortran-intel/build
export I_MPI_CC=icx
export I_MPI_CXX=icpx
export I_MPI_F77=ifx
export I_MPI_FC=ifx
cmake .. \
-DNETCDF_C_INCLUDE_DIR=$PREFIX/netcdf-intel/include \
-DNETCDF_C_LIBRARY=$PREFIX/netcdf-intel/lib/libnetcdf.so \
-DCMAKE_C_COMPILER=$(which mpiicx) \
-DCMAKE_Fortran_COMPILER=$(which mpiifx) \
-DMPI_Fortran_COMPILER=$(which mpiifx) \
-DCMAKE_INSTALL_PREFIX=$PREFIX/netcdf-fortran-intel
time make install -j $COMPILE_NP
mkdir -p $PREFIX/netcdf-intel/lib
# cp -r $PREFIX/netcdf-fortran-intel/lib/* $PREFIX/netcdf-intel/lib
export NETCDF_FORTRAN_ROOT=$PREFIX/netcdf-fortran-intel
export CPPFLAGS="-I$NETCDF_FORTRAN_ROOT/include $CPPFLAGS"
export LDFLAGS="-L$NETCDF_FORTRAN_ROOT/lib $LDFLAGS"
export LD_LIBRARY_PATH=$NETCDF_FORTRAN_ROOT/lib:$LD_LIBRARY_PATH
export PATH=$NETCDF_FORTRAN_ROOT/bin:$PATH
```
#### nco
```bash=
cd $SRC
mkdir -p nco-intel
wget https://github.com/nco/nco/archive/refs/tags/5.1.2.tar.gz
tar xzf 5.1.2.tar.gz --strip-components=1 -C nco-intel
cd nco-intel
export NETCDFPATH=$PREFIX/netcdf-intel
./configure --prefix=$PREFIX/nco-intel --enable-netcdf4 \
CPPFLAGS="-I$NETCDFPATH/include" \
LDFLAGS="-L$NETCDFPATH/lib"
# ./configure --prefix=$PREFIX/nco-intel --enable-netcdf4 CPPFLAGS="-I$NETCDFPATH/include" LDFLAGS="-L$NETCDFPATH/lib"
make -j $COMPILE_NP && make install
$PREFIX/nco-intel/bin/ncks --version
export PATH=$PREFIX/nco-intel/bin:$PATH
export LD_LIBRARY_PATH=$HOME/local/nco-intel/lib:$LD_LIBRARY_PATH
```
## Famli build & run
### Need modify
#### Add run permission
```bash=
chmod +x f_create_earth
chmod +x tools/c*
```
#### `mkmf.template.ifc`
```bash=
FFLAGS = -O0 -assume byterecl -i4 -r8 -zero
# CPPFLAGS = -I${NETCDFPATH}/include -I${HDF5PATH}/include -I${ZLIBPATH}/include -I${MPI_INCLUDE}
CPPFLAGS = -I${NETCDFPATH}/include -I${NETCDF_FORTRAN_ROOT}/include -I${HDF5PATH}/include -I${ZLIBPATH}/include -I${MPI_INCLUDE}
FC = ${MPI_BIN}/mpiifx
LD = ${MPI_BIN}/mpiifx
# LDFLAGS = -L${NETCDFPATH}/lib -lnetcdf -lnetcdff -L${MPI_LIB} -L${HDF5PATH}/lib -lhdf5 -lhdf5_hl -L${ZLIBPATH}/lib -lz
LDFLAGS = -Wl,-rpath,${NETCDFPATH}/lib64 -Wl,-rpath,${NETCDFPATH}/lib -Wl,-rpath,${NETCDF_FORTRAN_ROOT}/lib64 -Wl,-rpath,${NETCDF_FORTRAN_ROOT}/lib -Wl,-rpath,${HDF5PATH}/lib \
-L${NETCDF_FORTRAN_ROOT}/lib64 -L${NETCDF_FORTRAN_ROOT}/lib -L${NETCDFPATH}/lib64 -L${NETCDFPATH}/lib -L${HDF5PATH}/lib -L${ZLIBPATH}/lib \
-lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz -L${MPI_LIB} -lmpi
CFLAGS = -D__IFC
```
#### `utils/exp/namelists_earth`
All line about /path_to_famil/input/*,
change to yout own path
#### `f_create_earth`
```bash=
set WORK_DIR = /home/jonathanhsu
set FAMIL_HOME = $WORK_DIR/FAMIL_v1
set FAMIL_OPT = $FAMIL_HOME/lib
set NETCDFPATH = $FAMIL_OPT/netcdf-intel
set NETCDF_FORTRAN_ROOT = $FAMIL_OPT/netcdf-fortran-intel
set HDF5PATH = $FAMIL_OPT/hdf5-intel
set ZLIBPATH = $FAMIL_OPT/zlib-intel
set NCOPATH = $FAMIL_OPT/nco-intel
set netcdfpath = $NETCDFPATH
set netcdf_fortran_root = $NETCDF_FORTRAN_ROOT
set hdf5path = $HDF5PATH
set zlibpath = $ZLIBPATH
set ncopath = $NCOPATH
set mpipath = /opt/intel/oneapi/mpi/2021.15
set forpath = /opt/intel/oneapi/compiler/2025.1
set ccpath = /opt/intel/oneapi/compiler/2025.1
set gsl_lib = /usr/bin/gsl-config
set curl_lib = /usr/bin/curl
set udunits_lib = /usr/bin/udunits2
set mpi_include = $mpipath/include
set mpi_lib = $mpipath/lib
set mpi_bin = $mpipath/bin
set for_include = $forpath/include
set for_lib = $forpath/lib
set for_bin = $forpath/bin
set cc_include = $ccpath/include
set cc_lib = $ccpath/lib
set cc_bin = $ccpath/bin
```
```bash=
s#^ *setenv *NETCDF_FORTRAN_ROOT .*#setenv NETCDF_FORTRAN_ROOT $netcdf_fortran_root#g \
s#^ *setenv *LD_LIBRARY_PATH .*#setenv LD_LIBRARY_PATH /vol-th/lib\:$zlibpath/lib\:$hdf5path/lib\:$netcdfpath/lib\:$netcdf_fortran_root/lib\:$ncopath/lib\:$mpi_lib\:$cc_lib\:$gsl_lib\:$curl_lib\:$udunits_lib\:$LD_LIBRARY_PATH#g" \
```
#### `Make_fregrid_parallel.ifc`
```bash=
CFLAGS = -O2 -xHost -I${POSTP} -I${NETCDFPATH}/include -I${MPI_INCLUDE} -I${NETCDF_FORTRAN_ROOT}/include
# 修正 NetCDF 庫路徑
# LDFLAGS = -L${NETCDFPATH}/lib -lnetcdf -L${NETCDF_FORTRAN_ROOT}/lib -lnetcdff -L${HDF5PATH}/lib -lhdf5 -lhdf5_hl -L${ZLIBPATH}/lib -lz -L${MPI_LIB} -L/usr/lib64 -lm
LDFLAGS = -Wl,-rpath,${NETCDFPATH}/lib64 -Wl,-rpath,${NETCDFPATH}/lib -Wl,-rpath,${NETCDF_FORTRAN_ROOT}/lib64 -Wl,-rpath,${NETCDF_FORTRAN_ROOT}/lib -Wl,-rpath,${HDF5PATH}/lib \
-L${NETCDF_FORTRAN_ROOT}/lib64 -L${NETCDF_FORTRAN_ROOT}/lib -L${NETCDFPATH}/lib64 -L${NETCDFPATH}/lib -L${HDF5PATH}/lib -L${ZLIBPATH}/lib \
-lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz -L${MPI_LIB} -L/usr/lib64 -lm
DEFFLAG = -Duse_netCDF -Duse_libMPI
# 移除了 -v 旗標以避免過多輸出
LNFLAGS =
```
#### modify `famil/utils/exp/f_compile`
```bash=
set netcdf_f_include = $NETCDF_FORTRAN_ROOT/include
gcc -O -o $mppnccombine -I$NETCDFPATH/include -L$NETCDFPATH/lib -L$NETCDFPATH/lib64 \
-I$NETCDF_FORTRAN_ROOT/include -L$NETCDF_FORTRAN_ROOT/lib -L$NETCDF_FORTRAN_ROOT/lib64 \
$POSTP/mppnccombine.c -L$NETCDFPATH/lib -lnetcdf -L$NETCDF_FORTRAN_ROOT/lib -lnetcdff \
-L$HDF5PATH/lib -lhdf5 -lhdf5_hl -L$ZLIBPATH/lib -lz
$mkmf -p $EXE1DIR/famil.x -t $template \
-c "-Duse_libMPI -Duse_netCDF -DSPMD" \
# -c "-Duse_libMPI -Duse_netCDF -DSPMD -DSW_DYNAMICS" \
$pathnames $mpp_include $famil_include $netcdf_include $netcdf_f_include \
$hdf5_include $zlib_include $MPI_INCLUDE
```
#### modify `famil/utils/exp/f_env`
```bash=
# Add a line
setenv NETCDF_FORTRAN_ROOT unset
```
#### Code
- `$FAMIL_HOME/utils/postp/fregrid.c`
add `#include "mosaic_util.h"`
- `mpp_domain.c`
Line 16: `extern int pe, npes, root_pe;`
### Compile & run
#### `f_compile`
This steps will take some time.
Have a drink ~~and pray for no error occur~~.
```bash=
cd $FAMIL_HOME/utils
export CASENAME=test1
rm -r $FAMIL_HOME/run/$CASENAME
rm -r $FAMIL_HOME/exp/$CASENAME
./f_create_earth -case $CASENAME -plat ifc -res C48 -np 24
cd ../exp/$CASENAME
./f_compile > fcompile_result.txt 2>&1
```
#### `f_run`
```bash=
# It will take some time
# Recommand use tmux
./f_run > $FAMIL_HOME/run/$CASENAME/work/f_run.log 2>&1
```
If you got the following error messeages
```
lin_cld_micrphys diagnostics initialized.
NOTE from PE 0: topography_mod: Reading NetCDF formatted input data file: /home/sysadmin/ASC/FAMIL_v1/input/atm/navy_topography.nc
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libc.so.6 00007FF732445330 Unknown Unknown Unknown
famil.x 0000000000780AA7 Unknown Unknown Unknown
famil.x 000000000077DE56 Unknown Unknown Unknown
famil.x 00000000006EAABF Unknown Unknown Unknown
famil.x 00000000008763B7 Unknown Unknown Unknown
famil.x 00000000007EA7CB Unknown Unknown Unknown
famil.x 0000000000990369 Unknown Unknown Unknown
famil.x 000000000098EA5F Unknown Unknown Unknown
famil.x 0000000000410AFD Unknown Unknown Unknown
libc.so.6 00007FF73242A1CA Unknown Unknown Unknown
libc.so.6 00007FF73242A28B __libc_start_main Unknown Unknown
famil.x 0000000000410A15 Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
```
Check `ulimit -s`, you may need more stack space.
Modify `f_run` line 97 with following command ensure all nodes have sufficient stack space.
`mpirun --host comp1:12,comp2:12 -np $NP bash -lc "ulimit -s unlimited && exec $WORKDIR/famil.x"`
## With Spack
Here's another method for building FAMIL's dependencies: using a tool we introduced in class — Spack.
You can refer to the following script:
If you're at the installation step, replace `spack install` with `spack location --first -i` to retrieve the installation path.
[f_create_earth_spack.csh](https://365nthu-my.sharepoint.com/:u:/g/personal/112062334_office365_nthu_edu_tw/EaTuyA0eS6lEpnggyo5_5RMBV2TgXi-3YdPZWP6eFKC3Fg?e=DkbW2M)
This method should be easier, but you'll need to experiment on your own.
Sign in with Wallet
Connect another wallet