# UPS products for protoDUNE DAQ

###### tags: `DAQ` `protoDUNE` `artdaq` `DUNE`

[TOC]

* [Jira Issue 95](https://its.cern.ch/jira/browse/NP04DAQ-95)

## Dependency graphs
* [dune_artdaq v2019_07_30_develop](https://home.fnal.gov/~dingpf/SetupDUNEARTDAQ_forRunning_dep_graph.svg) 
* [artdaq v3_06_01 s83](https://home.fnal.gov/~dingpf/e17_s83_v3_06_01.svg) 
* [artdaq v3_06_01 s85](https://home.fnal.gov/~dingpf/e17_s85_v3_06_01.svg) 

## List of UPS products

- [x] 1. dim (`v2-r26 e15|e17`)
- [x] 2. uhal (`v2_6_4 e17:prof:s85|s83`)
- [x] 3. pugixml (`v1_10 e15|e17`)
- [x] 4. netio (rebuilt `netio-Oct v0_8_0` into `netio v0_8_0a e17:prof:s83|s85` )
- [x] 5. rogue (`v3_7_0 e17:prof|debug:s85:s83`)
- [ ] 6. QAT - [QATzip](https://github.com/intel/QATzip)
- [x] 7. zmq (`v4_3_2 e17`)
- [x] 8. cppzmq (`v4_4_1 NULL`)
- [x] 9. pyzmq (`v18_1_0 p2715a`)
- [x] 10. czmq (`v4_2_0 e17`)
- [x] 11. protobuf (deployed `v3_5_2a e15|e17` from scisoft, also built newer version `v3_10_0 e17`)
- [x] 12. ftd2xx (`v1_4_8 NULL`, should be falvored build `-f Linux64bit+3.10-2.17`)
- [x] 13. ptmp (built `v0_6_2 e17`)
- [x] 14. ptmp_tcs (built `v0_1_1a e17`)
- [x] 15. protodune_wibsoft (`v351a`, SVN revision 351, `e17:s85|s83:prof|debug`)
- [x] 16. jsoncpp (`v1_8_4 e17:prof|debug`)
- [x] 17. dunepdsprce (deployed `v1_1_0 avx2|gen:e17:prof` from scisoft)
- [x] 18. protodune_timing
- [x] 19. libxml2 (compiled latest version `v2_9_9 prof|debug`)
- [ ] 20. artdaq_dim_plugin
- [ ] 21. dune-artdaq-InhibitMaster

### Building dim

* [build instruction](https://dim.web.cern.ch/dim_unix.html)
* `dim` depends on gcc only, making a e15 and e17 build.
* source package and install instructions can be found [here](https://dim.web.cern.ch/dim_unix.html).
* Here is the [source zip file.](https://dim.web.cern.ch/dim_v20r26.zip)
* `dim` requires `motif-devel`
* Tested local build on `mu2edaq13.fnal.gov:/home/dingpf/build_dim/dim_v20r26/linux`
* Installed LIB, BIN and INC directories in the UPS product.
* Did not make libjdim.so (shareable library to be used from java).
* Accidentally cleaned `UPS_OVERRIDE` in my docker container used for building the UPS product, which in turn, make the `Linux64bit+4.15-2.17-e15` and `Linux64bit+4.15-2.17-e17` builds.

### Building uhal

* [build instruction](https://ipbus.web.cern.ch/ipbus/doc/user/html/software/install/compile.html)
* [git repo](https://github.com/ipbus/ipbus-software)
* [source tarball](https://github.com/ipbus/ipbus-software/archive/v2.6.4.tar.gz)
* e and s qualified builds (depends on art);
* debug and prof build types;
* using makefile, not cmake;
* `uhal` requires `pugixml-devel`, previously pugixml shared lib and headers are put in uhal UPS package on protoDUNE DAQ;
* python-devel is needed (pyconfig.h) for build;
* ~~escript is needed for build; install `erlang-rebar`;~~ (This is for `controlhub`)
* `make all` does not work for `uhal/gui`; should use `make` instead;
~~* `make install` requires dbus, docker container needs to be started with `--privileged` and needs `systemd` in it; otherwise gets `Failed to get D-Bus connection: Operation not permitted` [more info here.](https://serverfault.com/questions/824975/failed-to-get-d-bus-connection-operation-not-permitted)~~ (This is for `controlhub`)
* build on mu2edaq or np04-srv-023 instead;
* setup required pugixml (will set up gcc)
* setup boost
* setup gcc (not necessary, but good to have for indicating e qualifier);

#### Patches for v2_6_4:

* patch file created and put in source package (`patch -b -p1 < ipbus.patch`)
* remove `-lboost_python` from pychoal Makefile (scisoft's version of boost does not contain boost_python .so)... this bypass the build, but the python binding may not work...
* adding `#include <boost/core/noncopyable.hpp>` into two header files: ` uhal/uhal/include/uhal/ClientFactory.hpp` and `uhal/uhal/include/uhal/ConnectionManager.hpp`
* ~~omit some install commands in controlhub Makefile which installs configuration files under system path, and restarts rsyslog service.~~ (This is no needed if doing `Set=uhal` installation.)

#### ENVs needed:

     export EXTERN_BOOST_INCLUDE_PREFIX=$BOOST_INC
     export EXTERN_BOOST_LIB_PREFIX=$BOOST_LIB
     export EXTERN_PUGIXML_INCLUDE_PREFIX=$PUGIXML_INC
     export EXTERN_PUGIXML_LIB_PREFIX=$PUGIXML_LIB

#### Clean-ups

There is a bug in its Makefile for `install` target. It puts several copies of itself under nested levels of sub-directories (e.g. include/install/include/install/include/install/includ), this happends for both `lib` and `include`. I did not track down the bug, but just to clean things up, I simply removed the first level sub-dir under `lib` and `include`.

### Building pugixml

* [repo and build instruction](https://pugixml.org/docs/quickstart.html)
* e qualified build, with `-D CMAKE_CXX_FLAGS="-std=c++17"` for `e17`
* with ` -DBUILD_STATIC_LIBS=OFF `
* use `source define_basics NO_TYPE ` for no `debug/prof` builds in ssibuildshims.

### Buliding netio

* [repo and build instruction](https://gitlab.cern.ch/atlas-tdaq-felix/netio)
* Depends on tbb, so need to be s qualified as tbb is in art bundle;
* e and s qualified builds;
* extra qualifiers needed (prof, debug)
* using cmake
* disbale FELIX_BUILD, so it does not depend on zmq etc, only need one .so and netio.hpp
* requires gcc and tbb
* requires `libfabric-devel`; yum packages available on CentOS7. Install via YUM.
* Dev head no longer has `main_unsub.cpp`, so `netio_unsub` executable is no longer produced.

### Building rogue

* [build instruction](https://slaclab.github.io/rogue/installing/build.html);
* [git repo](https://github.com/slaclab/rogue)
* e and s qualified builds; (squalified may not need, non python build does not depend on boost)
* (requires python3 >= 3.6 for python bindings...) setup this for build only...
* requires bzip2-devel for compiling from source installation;
* requires zmq
* use boost v1_69_0
* using cmake;
* REQUIRES GIT TO BUILD... (in CMakeList.txt, it gets version number from git tag... and states "git is requried to build rogue"...)
* Need to patch CMakeList.txt to remove `python3` from `find_package(Boost ... python3)`
* `export BOOST_PATH=$BOOST_FQ_DIR`
* `cmake ../rogue -DROGUE_INSTALL=custom -DROGUE_DIR=/path/to/custom/dir`
* previous rogue UPS package contains bzip2's header and share lib in it... not planning to make a bzip2 package but install the rpm package across the cluster. (It appears that `bzip2-devel` is already installed on all np04 nodes.)

#### Steps to build rogue on np04-srv-023

* Note: you have to use `np04-srv-023` where I have removed the system package `zeromq3-devel` and `zeromq3`. The reason is the default CMake setting in rogue finds the system zeromq first, but we want to use the UPS product `v4_3_2` instead.
* The steps to build it on `np04-srv-023` are as the following:

# setup ups, making gcc, cmake available
source /nfs/sw/artdaq/products/setup
# add the source products path to the $PRODUCTS env
export PRODUCTS=/nfs/sw/ups_source_packages/products:$PRODUCTS

cd /nfs/sw/ups_source_packages/products/rogue/v3_7_0

# remove existing built products (otherwise the build script will
# fail at "git clone" stage)
rm -rf Linux64bit+3.10-2.17-e1*

# run the build script
./build_rogue.sh /nfs/sw/ups_source_packages/products/ e17 prof s83

# run the build script again with "tar" option; this will create
# a tarball in the source prodcuts directory level.
./build_rogue.sh /nfs/sw/ups_source_packages/products/ e17 prof s83 tar

### Building zmq

* This is the `libzmq` core package;
* [repo](https://github.com/zeromq/libzmq)
* straight forward installation with `configure --prefix=${pkgdir}` followed by `make` and `make install`;
* Removed `cppzmq` header installation from previous build script.

### Making cppzmq

* [repo](https://github.com/zeromq/cppzmq)
* This is a header-only package;
* Created a `NULL` flavor;
* No dependencies of `zmq`, since it should work with different versions of `zmq`;
* But one has to either have system `zmq` or setup one version of `zmq` UPS product before using `cppzmq`.

### Building pyzmq

* [repo](https://github.com/zeromq/pyzmq)
* setup required python and zmq version in ups table file;

python setup.py configure --zmq=${ZMQ_FQ_DIR} || ssi_die "python setup.py build failed"
python setup.py build
python -m compileall ./build/lib.linux-x86_64-2.7
mv build/lib.linux-x86_64-2.7/*  ${pkgdir}/lib

### Building czmq

* [repo and build instructions](https://github.com/zeromq/czmq#building-and-installing)

export CXXFLAGS="$cxxflg"
export CFLAGS="${cflg} -I${ZMQ_INC}"
export CC=${cc}
export CXX=${cxx}
export LD_FLAGS=-L${ZMQ_LIB}
export PKG_CONFIG=$ZMQ_LIB/pkgconfig
./configure --prefix=${pkgdir} && make -j 8 && make install

### Building protobuf

* Used exisiting scisoft script to build newer version of protobuf;
* [official release](https://github.com/protocolbuffers/protobuf/releases);
* Added `./autogen.sh` in build script;
* Skipped `make check`, bug existed in `v3_10_0`, missing `third_party/googletest/googletest` directory.

### Making ftd2xx

* [Official download](https://www.ftdichip.com/Drivers/D2XX.htm)
* Tarball contains a full release, simply move header files and share libs into directories structured accordoing to UPS standards.

### Building protodune-timing

* [repo](https://gitlab.cern.ch/protoDUNE-SP-DAQ/timing-board-software)
* v5_1_0 tag name is `v5b0`;
* Builds with Makefiles;
* Depends on `uhal v2_6_4 e17:prof:s85|s83`
* [Build instructions](https://twiki.cern.ch/twiki/bin/view/CENF/PDuneSPTimingInstallSW) in Twiki;
* Rewrite build procedure using `ssibuildshims`
* build `core` subdirectory only;
* changed c++ standard to `c++14` (e15) or `c++17` (e17).

### Building ptmp
* https://github.com/brettviren/ptmp tag 0.6.1
* UPS build notes https://github.com/brettviren/ptmp/blob/master/docs/ups.org
* Example build command line: `./tools/create-ups-product.sh /nfs/sw/work_dirs/phrodrig/tmp/ups v0_6_0 e15 czmq protobuf zmq` after setting up the appropriate packages;
* it should now depend on `cppzmq` as well.
* Now using `ssibuildshims`, the following is the configuration and install part of the script.

cflg="-g -ggdb3 -O2"

if [[ "${basequal}" == e1[45] ]]
then
  cxxflg="${cflg} -std=c++14 -Wno-deprecated-declarations"
elif [[ "${basequal}" == e1[79] ]]
then
  cxxflg="${cflg} -std=c++17 -Wno-deprecated-declarations"
else
  echo "ERROR: unrecognized qualifier ${basequal}"
  usage
  exit 1
fi

#make -j ${ncore} || ssi_die "ERROR: make failed."
./tools/waf configure --jobs=${ncore} --cxxflags="${cxxflg}" --prefix=${pkgdir} \
      --with-libzmq-lib=$ZMQ_LIB --with-libzmq-include=$ZMQ_INC \
      --with-libczmq-lib=$CZMQ_LIB --with-libczmq-include=$CZMQ_INC \
      --with-protobuf=$PROTOBUF_FQ_DIR || ssi_die "ERROR: failed to configure with waf!"

./tools/waf --notests install || ssi_die "ERROR: failed to install with waf!"

### Building ptmp-tcs

* [repo](https://github.com/brettviren/ptmp-tcs)
* Same idea as ptmp. Set up ptmp, then `./tools/create-ups-product.sh /nfs/sw/work_dirs/phrodrig/tmp/ups v0_2_0 e15 czmq protobuf zmq ptmp`
* similar as `ptmp`, the following is the configuration part with `waf`.

#make -j ${ncore} || ssi_die "ERROR: make failed."
./tools/waf configure --jobs=${ncore} --cxxflags="${cxxflg}" --prefix=${pkgdir} \
      --with-ptmp-include=$PTMP_INC --with-ptmp-lib=$PTMP_LIB \
      --with-libzmq-lib=$ZMQ_LIB --with-libzmq-include=$ZMQ_INC \
      --with-libczmq-lib=$CZMQ_LIB --with-libczmq-include=$CZMQ_INC \
      --with-protobuf=$PROTOBUF_FQ_DIR || ssi_die "ERROR: failed to configure with waf!"

### Building jsoncpp

* Used existing scisoft build script for jsoncpp;
* built newer version `v1_8_4` with `e17:prof` and `e17:debug`;
* had problems with building the newer versions `v1_9_0` and up (not going to resolve now).

### Building protodune_wibsoft

* Previous [Build instructions](https://twiki.cern.ch/twiki/bin/view/CENF/WIBArtDAQOp)
* Rewrote build procedure using `ssibuildshims`...
* Generated patch files for revision 351.

### Building libxml2

* latest version is `v2_9_9`;
* [Official repo](https://gitlab.gnome.org/GNOME/libxml2/);
* renamed source directory name in tarball (to follow `pkg-dot-version` name convention);
* used `autogen.sh` instead of `configure` in SciSoft's build script.

### Building QAT

*

---

## Random notes

* jsoncpp build script is designed for e qualified debug and prof build (extra qualifiers); with cmake...
* [ipbb framework](https://indico.fnal.gov/event/18746/contribution/4/material/slides/0.pdf) -- building felix firmware
* [ipbus release](https://github.com/ipbus/ipbus-software/releases)