# Provision miniDAQ

## Reconfigure logical volume

* Reduce `/home` LVM to be 100GB;
* Create `/scratch` with all the remaining space.

umount /home
lvreduce -L 100G /dev/centos/home
mkfs.xfs -f /dev/centos/home
mount /home
lvcreate -l 100%FREE -n scratch centos
mkfs.xfs /dev/centos/scratch
mkdir /scratch
echo "/dev/mapper/centos-scratch /scratch  xfs defaults  0 0" >> /etc/fstab
mount -a
chmod go+w /scratch

# recreate users' home directories if necessary.

## Install Ansible

* `sudo yum install epel-release`
* `sudo yum install ansible`

## Software installation via Ansible

* Git clone `np04online` repo from CERN GitLab;
* Add `localhost` to `hosts` file;
* Modify `install_software.yml`, add `hosts: localhost` and `connection: local`;
* Remove proxy setting with pip in `roles/np04software/tasks/main.yml`;
* Run `ansible-playbook install_software.yml --limit=localhost`.

## `cvmfs`

### Install `cvmfs`

yum-config-manager --add-repo http://cvmrepo.web.cern.ch/cvmrepo/yum/cernvm.repo
wget http://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM -O /etc/pki/rpm-gpg/RPM-GPG-KEY-CernVM
yum install cvmfs cvmfs-config-default

### Configure `cvmfs`

wget http://home.fnal.gov/~dingpf/cvmfs.tar.gz 
rm -rf /etc/cvmfs/*
tar zxvf cvmfs.tar.gz; sudo mv cvmfs/* /etc/cvmfs/
mkdir -p /cvmfs/dune.opensciencegrid.org
mkdir -p /cvmfs/fermilab.opensciencegrid.org
mkdir -p /cvmfs/larsoft.opensciencegrid.org
cvmfs_config reload
mount -t cvmfs dune.opensciencegrid.org /cvmfs/dune.opensciencegrid.org
mount -t cvmfs larsoft.opensciencegrid.org /cvmfs/larsoft.opensciencegrid.org
mount -t cvmfs fermilab.opensciencegrid.org /cvmfs/fermilab.opensciencegrid.org

## Run `artdaq` Demo

### Install artdaq related pacakges (most of dependencies are availabel in `cvmfs`)

cd /scratch/
mkdir dingpf
cd dingpf/
wget https://cdcvs.fnal.gov/redmine/projects/artdaq-demo/repository/revisions/develop/raw/tools/quick-mrb-start.sh
chmod +x quick-mrb-start.sh
./quick-mrb-start.sh --tag v3_06_01

### Setting up environment to run the demo (in VNC)

source setupARTDAQDEMO
setup artdaq_daqinterface v3_06_01
export DAQINTERFACE_USER_SOURCEFILE=$PWD/DAQInterface/user_sourcefile_example
source $ARTDAQ_DAQINTERFACE_DIR/source_me

### Now run the demo (in VNC)

ipcrm -a;./run_demo.sh --config demo --comps component{01..02} --no_om --runduration 20

## FELIX

### Hardware installation
Check compatibility of server specifications. Only motherboards with Wupper firmware will work.

Connect FELIX-712 to PCIe x16 slot
Connect FELIX-712 PCIe power connector to the motherboard (check connection of the motherboard; in our case, a CPU-to-PCIe GPU 8pin power cable was needed) (in our case it was SM CBL-PWEX-0665 Pcle 8pinfe)
Check if device is found:

 lspci | grep Xil

Connect JTAG
Install JTAG driver from Xilinx installation path

sudo source /scratch/Xilinx/Vivado/201*/data/xicom/cable_drivers/lin64/digilent/install_digilent.sh

### Driver and FELIX software installation

* FELIX driver needs `dkms` and `kernel-devel` installed;

 wget https://atlas-project-felix.web.cern.ch/atlas-project-felix/user/dist/software/driver/tdaq_sw_for_Flx-4.5.0-2dkms.noarch.rpm
 sudo yum install tdaq_sw_for_Flx-4.5.0-2dkms.noarch.rpm
 sudo /etc/init.d/drivers_flx start

In case you want to stop the driver:

/etc/init.d/drivers_flx stop

* Pay attention to the output of `yum install` command above, and verify that new kernel modules are built; if not, you will get errors when running the `drivers_flx start` saying modules `io_rcc`, `cmem_rcc` and `flx` are not found;
* In case of module building step was skipped when installing the driver rpm package, it is very likely dkms did not found the kernel header due to broken symlinks. [This page](https://support.jazznetworks.com/hc/en-us/articles/360007333134-Centos-RH-7-Getting-kernel-headers-for-this-kernel-does-not-seem-to-be-installed-) has more details about how to fix the issue.

* Get FELIX software, go to software directory, install the software and start it.

### Configuring and running FELIX software

* Used this tarball containing all FELIX releated software (felix-04-01-00-97-x86_64-centos7-gcc8-opt.tar.gz);
* some of the binaries requries gcc 8 and Qt 5.12 (both of these packages are available as UPS products in cvmfs);
* e.g. to run `elinkconfig`:
* `ssh` to `cleopatra` with X11 forwarding turned on;

cd /scratch/FELIX/
tar -xvzf felix-04-01-00-97-x86_64-centos7-gcc8-opt.tar.gz 
cd /scratch/FELIX/felix-04-01-00-97/x86_64-centos7-gcc8-opt/
source setup.sh

To set up:

cd /scratch/FELIX/felix-04-01-00-97/x86_64-centos7-gcc8-opt/
source setup.sh
source /software/cvmfs/dune.opensciencegrid.org/dunedaq/DUNE/products_dev/setup
setup qt v5_12_3a -q e19

To test, run

To download firmware:
* open Vivado
* open Hardware Manager
* Auto connect to FELIX
* Download bitstream `FLX712_FULLMODE_12CH_CLKSELECT_GIT_JBSC_refresh_rm-4.7_125_190821_11_11.bit`

To configure, open

elinkconfig

In window:
* Select `Full mode`
* Generate/Upload -> `8/Incr/8` -> Upload
To test, run

fdaq -e -t 2

### Install daqling

1. Swap zeromq3-devel with zeromq-devel (default is zeromq4) by "yum swap zeromq3-devel zeromq-devel”
2. Run the Ansible playbook to set up the host;
3. Add "export FELIX_LIB=${FELIX_ROOT}/lib” and "export FELIX_INC=${FELIX_ROOT}/include” to "/scratch/FELIX/felix-04-01-00-97/x86_64-centos7-gcc8-opt/setup.sh"
4. source /scratch/FELIX/felix-04-01-00-97/x86_64-centos7-gcc8-opt/setup.sh
5. Comment out line 164 and 165 in "daqling-master/src/Modules/FelixReadout/CardReader.cpp”
6. yum install libpmem-devel
7. ln -s /usr/lib64/libpmemblk.so.1 /usr/lib/libpmemblk.so
8. Add "set(FELIX_LIB $ENV{FELIX_LIB})”, "set(FELIX_INC $ENV{FELIX_INC})”, "include_directories(/software/software/flxcard)” and "/software/software/drivers_rcc” in “daqling/src/Modules/FelixReadout/CMakeLists.txt”
9. Line 139 in “daqling/CMakeLists.txt” changed to set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pedantic -Wall -Wextra -Wcast-align -Wcast-qual -Wzero-as-null-pointer-constant -Wdisabled-optimization -Wformat=2 -Winit-self -Wlogical-op -Wmissing-declarations -Wmissing-include-dirs -Wredundant-decls -Wsign-conversion -Wstrict-null-sentinel -Wstrict-overflow=5 -Wundef -Wformat-security -fdiagnostics-color=auto -Wno-overloaded-virtual”)
10. source daqling/setup.sh and then run `cmake3` and build.

(The modified package can be installed to /home/USERNAME/daqling-master, and the built directory can be set to /home/USERNAME/daqling-master/build)

cd /software
git clone https://gitlab.cern.ch/rsipos/daqling.git
cd daqling
source cmake/setup.sh
cd ansible
ansible-playbook set-up-host.yml --ask-become
ansible-playbook install-webdeps.yml --ask-become
ansible-playbook install-redis.yml --ask-become
ansible-playbook install-boost-1_70.yml --ask-become
cd ..
setup gcc v8_2_0
setup cmake v3_17_3
source cmake/setup.sh
mkdir build
cd build
cmake3 ../ -D CMAKE_C_COMPILER=/bin/gcc -D CMAKE_CXX_COMPILER=/bin/g++ -DENABLE_BOOST=1
make

Running the DAQ demo

source cmake/setup.sh
daqpy configs/demo.json
start [run_num]
stop
down

## Build and Run `dune DAQ release`

==To be finished==