# SNBModules for SNB file transfer service ## Introduction Relevant talks can be found [here](https://indico.fnal.gov/event/60110/contributions/268713/attachments/167816/224207/Bookkeeper_data_transfer_plugins.pdf) and [here](https://indico.fnal.gov/event/59587/contributions/265469/attachments/166418/221652/SNB%20-%20Bookkeeper%20data%20transfer%20component.pdf) GitHub repo [`DUNE-DAQ/snbmodules`](https://github.com/DUNE-DAQ/snbmodules) ## Installation ### Install System packages Installed `libnsl` as its required by `aclocal`. ### Install and run `rclone` #### Installing GO ``` sudo dnf install golang ``` #### Clone RClone lib ``` git clone https://github.com/rclone/rclone.git ``` #### Build rclone ``` go build --buildmode=c-shared -o librclone.so github.com/rclone/rclone/librclone ``` #### Export var (add to the env.sh file) ``` export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/lib/folder export RCLIB_INC=/path/to/lib/folder export RCLIB_LIB=/path/to/lib/folder # I have the following on isc01.fnal.gov #export LD_LIBRARY_PATH=/home/dingpf/rclone:$LD_LIBRARY_PATH #export RCLIB_LIB=/home/dingpf/rclone #export RCLIB_INC=/home/dingpf/rclone ``` #### Run local server ``` cd rclone/ curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip unzip rclone-current-linux-amd64.zip rm rclone-current-linux-amd64.zip cd rclone-v1.63.1-linux-amd64 # as of August 1st 2023, this is the latest version in the zip file ./rclone serve http / --addr IP:PORT(8080) --buffer-size '0' --no-modtime --transfers 200 -v --multi-thread-cutoff=50G --multi-thread-streams=16 ``` ### Install libtorrent #### Clone lib ``` cd $HOME # or other places git clone --recurse-submodules https://github.com/arvidn/libtorrent.git git checkout v2.0.9 # as of August 1st, 2023, this is the latest versioned tag ``` #### Build rclone ``` mkdir build;cd build cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 -G Ninja .. ninja ``` #### Export var (add to the env.sh file) ``` export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/lib/folder/build export LIBTORRENT_INC=/path/to/lib/folder/include export LIBTORRENT_LIB=/path/to/lib/folder/build # I have the following on isc01.fnal.gov #export LD_LIBRARY_PATH=/home/dingpf/libtorrent/build:$LD_LIBRARY_PATH #export LIBTORRENT_INC=/home/dingpf/libtorrent/include #export LIBTORRENT_LIB=/home/dingpf/libtorrent/build ``` ### create a DUNE DAQ workarea ```bash= source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh setup_dbt fddaq-v4.1.0 dbt-create -s fddaq-v4.1.0 dev-snb-v4.1.0 ``` ### load the DUNE DAQ development environment ```bash= cd dev-snb-v4.1.0 # add the export statements from above into the env.sh file in the workarea cat <<EOF >>env.sh export LD_LIBRARY_PATH=/home/dingpf/rclone:$LD_LIBRARY_PATH export RCLIB_LIB=/home/dingpf/rclone export RCLIB_INC=/home/dingpf/rclone export LD_LIBRARY_PATH=/home/dingpf/libtorrent/build:$LD_LIBRARY_PATH export LIBTORRENT_INC=/home/dingpf/libtorrent/include export LIBTORRENT_LIB=/home/dingpf/libtorrent/build EOF echo "snbmodules" >> sourcecode/dbt-build-order.cmake source env.sh ``` ### build `snbmodules` ```bash= cd sourcecode git clone https://github.com/DUNE-DAQ/snbmodules # add snbmodules into the list in dbt-build-order.cmake, then run dbt-build # run the following to load the locally built snbmoudles dbt-workarea-env ``` ### Configure `snbmoudles` ### Run the application