# daq-sw-librarians discussion on 07/02

## Release Organization
==Persons primarilly working on this: Pengfei==

### Use seperate subdirectories for each DAQ release

### Motivation:

Having separate directories for each DAQ release, in which only the version of packages used in the specific release are included, benifits both developer/user and release manager to get a clearer view of softwares used in certain release and minimize risks of using incompatible versions of a package in a given release.

### Proposed solution:
The following proposed solution is for the near-term usage of UPS as our software packaging technique. It does not apply to other packaging techniques, e.g. spack. (`spack view` in combination with some homebrew scripts might be a nice solution to achieve similar goals. We will explore that shortly.)

In the context of using UPS products, the release subdirectory can contain symbolic links to the versions of packages used or the actual package contents.

The following directory structure illustrates one proposed setup with symlinks under `releases` subdirectory to the chosen versions of UPS products under `/cvmfs/dune.opensciencegrid.org/DUNE/products`.

By default, the UPS products directory a user/developer needs to use is just the release subdirectory, in which only one version for each package is avalilable (i.e. `ups list -aK+` shows only one version for each package). Technichally, this is achived by only including the `releases/v1.0.0/packages` and `releases/v1.1.0/externals` in the `PRODUCT` environment variable.

If different versions of packages need to be used for development/test purpose, one can add the `/cvmfs/dune.opensciencegrid.org/dunedaq/DUNE/products` to the `PRODUCT` env to get access to those.

/cvmfs/dune.opensciencegrid.org/dunedaq/DUNE/
└── products
│   ├── appfwk
│   │   ├── v1.0.0
│   │   └── v1.0.0.version
│   ├── boost
│   │   ├── v1_70_0
│   │   ├── v1_70_0.version
│   │   ├── v1_73_0
│   │   └── v1_73_0.version
│   ├── ers
│   │   ├── v0_26_00
│   │   └── v0_26_00.version
│   ├── gcc
│   │   ├── v8_2_0
│   │   └── v8_2_0.version
│   ├── setup
│   ├── setups
│   ├── setups_layout
│   ├── setups.new
│   ├── ...
│   ├── ...
│   ├── ...
│   ├── TRACE
│   │   ├── v3_15_09
│   │   └── v3_15_09.version
│   └── ups
│       ├── current.chain
│       ├── v6_0_8
│       └── v6_0_8.version
└── releases
    └── v1.0.0
        ├── externals
        │   ├── boost
        │   │   ├── v1_73_0 -> ../../../../products/boost/v1_73_0
        │   │   └── v1_73_0.version -> ../../../../products/boost/v1_73_0.version/
        │   ├── gcc
        │   │   ├── v8_2_0 -> ../../../../products/gcc/v8_2_0
        │   │   └── v8_2_0.version -> ../../../../products/gcc/v8_2_0.version/
        │   ├── setup
        │   ├── setups
        │   ├── setups_layout
        │   ├── setups.new
        │   ├── ...
        │   ├── ...
        │   ├── ...
        │   ├── TRACE
        │   │   ├── v3_15_09 -> ../../../../products/TRACE/v3_15_09
        │   │   └── v3_15_09.version -> ../../../../products/TRACE/v3_15_09.version/
        │   └── ups
        │       ├── current.chain -> ../../../../products/ups/current.chain
        │       ├── v6_0_8 -> ../../../../products/ups/v6_0_8
        │       └── v6_0_8.version -> ../../../../products/ups/v6_0_8.version/
        └── packages
            ├── appfwk
            │   ├── v1.0.0 -> ../../../../products/appfwk/v1.0.0
            │   └── v1.0.0.version -> ../../../../products/appfwk/v1.0.0.version
            ├── ...
            ├── ...
            ├── ...
            └── ers
                ├── v0_26_00 -> ../../../../products/ers/v0_26_00
                └── v0_26_00.version -> ../../../../products/ers/v0_26_00.version/

## Package dependency
==Persons primarilly working on this: Pengfei==

Although the release organization described above indicates the dependencies of packages and their versions by a given DAQ release, it does not fully solve the problem of handling package depencies. Package dependencies need to be present in various levels, places, and envrionments; there might be multiple types of dependencies for different types of pacakges or envrionments.

### Types of package dependency

Package dependency might be different for the following envrionments:

1. software developing/building envrionment;
2. software testing/debugging envrionment;
3. production envrionment.

These envrionments may be further broken down into finer categories by system and client types, e.g. DUNE, protoDUNE, DAQ teststand, CentOS7, CentOS8 etc.

Some packages may not be needed under certain environments, e.g. `cmake` package can be skipped needed under a production environment. In addition, different variants (e.g. `debug` or `opt`) of a specific version of a package might be used under different environments.

Sometimes a package may require a specific version/variant of a package. These happens more frequently among DAQ packages than thrid party packages. But sometimes only a minimum version of a package is required. For example, a bug fix or new feature in `AppFwk` may affect several downstream DAQ packages, and a minimal version of AppFwk may be requried by those packages.

### Proposed solution

To fully handle package dependency described above, we can work from the following places where information related to package dependecy is presented.

#### Presence of package dependency

1. Associated with each package and handled at the software packaging level, e.g. in the UPS table file or spack `package.py` file of a package;
2. inside a release manifest file, or an umbrella release package containing/depending on a list of packages and their corresponding versions if specific versions are needed;
3. `CMakeLists.txt` file.

Though only one version of each package gets put under a given release directory. In each of the places above, any requirement of packages versions should only be specified if it is really necessary. In those rare cases, a minimal version requirement should always be used instead of a specific version if possible.

Dependencies of a package should be specified in a file associated with the package itself. Although the dependencies may also get handled at the DAQ release level. This becomes useful if the individual package would be used outside of a DAQ release. Dependencies speficied with the package should be consistent with those at the release level. But in case of conflicts, e.g. when a developer tests a new version of a package in a given release, the package level dependency specification takes precedence.

Ideally, if different dependency specifications are needed under different environments, they should be associated with each package. This is achievable with spack modules. But it might be difficult with UPS. At a minimum, the dependency specification associated with each package should reflect the dependency required by a production environment. Additional dependencies under debugging or developing environment can be specified at the DAQ release level.

There might be several groups of packages connected by dependency in a given DAQ release. Those groups get coupled only at the release level.

DAQ packages are more closely coupled together than third party packages. Although the treatment of DAQ packages may be the same as third party packages in the dependency specification associated with the package, or at the DAQ release level, this difference will be reflected inside CMake files, as DAQ packages utilizes common CMake modules. Physically, DAQ packages are separated from third party packages described in the release organization section above. Any dependencies specied in CMake files should be propagated/included at the package and release level.

## Tools for development

1. DAQ release manifest file (DAQ release package) where a list of packages (and versions/variants) is provided for a given release, this is the list consitent with the packages get put under the release directory.
2. Inside the manifest file, or in separate files, defines list of packages needed for different envrionments;
3. A tool for creating a DAQ release according to the manifest file (DAQ release package);
4. A tool for validating dependencies between packages in a given DAQ release.

## Proposed CMake functionality

==Persons primarily working on this: John. Note that function names and signatures are subject to change==

As established a few weeks ago, each DAQ package will have a single `CMakeLists.txt` file associated with it. For the v1 release we went with a provisional model in which each DAQ package (and there were only two, `appfwk` and `ers`) was treated as a CMake subproject of a superproject. This allowed us to take advantage of CMake scoping rules in that we could put functionality we wanted the DAQ packages to have in common into the superproject `CMakeLists.txt` file, knowing this functionality would automatically be picked up in the scope of the DAQ packages' individual `CMakeLists.txt` files.

Moving forward, we no longer have this ability, since the plan is to scrap to the superproject/subproject model. If no further action were taken after eliminating this model, it would mean that each time a new DAQ package was created, it would be up to the `CMakeLists.txt` file's author(s) to remember what C++ specifications to require, where/how to install the package's code, etc. While it's important that all DAQ package developers get reasonably proficient with CMake, it's unreasonable to expect them to do the above.

For this reason, we want to make a set of CMake functions available which abstract out some of the CMake details each DAQ package relies on. The following two pitfalls should be avoided, however:

* Replacing a well-known, well-documented CMake function with a DUNE-specific equivalent which just changes, rather than simplifies, the interface
* Creating a function which abstracts away so much low-level functionality that people find they have to resort to basic CMake calls most of the time

With that said, there are some functions that strike me as uncontroversially useful. They're presented roughly in the order they'd be called in a `CMakeLists.txt` file. I go with a naming convention in which each function is prefaced by `daq_`, so we know that it's one of our own and not a basic CMake call or a function from a third-party module.

1.`daq_setup_environment`()

To the extent that there are certain standards every DAQ package should follow, they should be put into a function which contains the CMake code to ensure those standards. Some obvious ones are: requiring that the C++ code is standard, extensionless C+\+17, compiling libraries as shared and not static, and performing the CMake calls needed to support post-build runs with the linter tools. It may be that in this particular case, a macro would be more appropriate than a function.

2. `daq_setup_dependencies( NAME_OF_MANIFEST )`

This would take the dependencies specified in the manifest mentioned earlier in this document and translate them into `find_package()` calls. Conceptually quite simple, it would remove the need for `CMakeLists.txt` file authors to write a corresponding `find_package()` call for each entry in the manifest...or to change `find_package()` calls when the manifest changes. This function likely wouldn't be used in the early stages of a DAQ package's development, but would replace the `find_package` calls at some point before the DAQ package's first official versioning. This would almost certainly be a macro and not a function. (Update, Jul-8-2020: Pengfei argues we shouldn't have the manifest input into the raw CMake code, which renders this function pointless)

3. `daq_point_build_to(SUBDIR)`

This function already exists, though we'd rename it with the standard `daq_` preface. It's purpose is to tell CMake that when a build is performed, the location of the executables, etc. should be in a subdirectory of the build area of the same name as the subdirectory in the code tree. Moving forward, we'd extend this so it would apply similar rules to where the executables, etc. would appear in the installation area.

4. `daq_add_unit_test(TESTNAME OPTIONAL_LIB1 OPTIONAL_LIB2 ...)`

Similar to the `add_unit_test` function we already have, but the interface allows you to link in additional libraries as needed rather than expect you to define a CMake variable called `DAQ_LIBRARIES_PACKAGE` to accomplish the same. This function handles the Boost unit test code the developer would otherwise need to worry about.

5. `daq_install_package( NAME_OF_MANIFEST POSSIBLE_DEPENDENCY)`

The set of actions we'd want to perform for each DAQ package could potentially get pretty lengthy, and hence a `daq_install_package` function could be very useful. Obviously, the simplest thing this function would do is put the executables, compiled shared object libraries and public headers into an installation area.

But there are other things it can do. It can try to use the package manifest to automatically generate a`<package>`Config.cmake file which calls to `find_package` subsequently use. It could even try to do things like automatically generate a ups table or a `package.py` file for Spack.

Note there's a tension between relieving package developers from as much manual labor as possible (including labor which requires manually transcribing package versions), and turning `daq_install_package` into an unwieldy, "Swiss Army Knife" function. daq-buildtools' Issue #13, filed Jul-1-2020, covers the broad topic of installation, though it may even be appropriate to break it up into smaller, more manageable issues, to be reviewed and merged individually. However, this is more a developer concern than a user concern - all the user needs to worry about is to remember calling `daq_install_package`.

### Other thoughts:

In v1, there were CMake variables defined which specified packages which were universally useful to DUNE DAQ libraries (`ers`, `pthread`), and then to DUNE DAQ executables (`ers` and `pthread` as well, but with Boost's program options library added in). We simply defined these variables (`DAQ_LIBRARIES_UNIVERSAL`, `DAQ_LIBRARIES_UNIVERSAL_EXE`) in the superproject CMakeLists.txt file for writers of subproject CMakeLists.txt files to access if they so wished. Moving forward it's probably a better idea to have a canonical DAQ library composed of other libraries (`ers`, etc.) which can be linked in to a target s.t. the needed include paths of those component libraries are automatically picked up by the target.

There's also the question of priorities, i.e., what functions should be written first. I'm very much open to input on this, but here's one idea. Notice the emphasis on writing the "basic" version of some of these functions before going back and making the implementation more elaborate. Presented in order of implementation.

* `daq_setup_environment`. Without the canonical-daq-library idea (yet) incorporated
* `daq_install_package`. Without the fancy ups/Spack file creation; i.e., just the basics.
* `daq_point_build_to`. Should be quick.
* `daq_add_unit_test`. Also should be quick.
* `daq_setup_dependencies`. Need to figure out the format of the manifest, will coordinate with Pengfei. How to convey the semantics of "we don't care about the version of the package" vs. "we want EXACTLY this version" vs. "we want AT LEAST this version"?

After implementing the basic set of functions, can go back and flesh things out. The advantage of this approach is that by the time I go back, we'll probably have a better grasp on, say, the ups vs. Spack vs. ?? question, which means I'll know better how to extend `daq_install_package`

## Example CMakeLists.txt file

Here's what the `CMakeLists.txt` file for *appfwk* would look like given the functions described above:

cmake_minimum_required(VERSION 3.14)
project(appfwk)

find_package(daq-buildtools REQUIRED)
include(DAQ)  # Needed for the daq_* calls below

daq_setup_environment()


# May not use daq_setup_dependencies; Pengfei argues that CMake-level
# code should be self-contained, and that a manifest should be 
# downstream of it.

#daq_setup_dependencies("appfwk_manifest.txt")

find_package(cetlib REQUIRED)
find_package(folly REQUIRED)
find_package(ers REQUIRED)
find_package(Boost COMPONENTS unit_test_framework program_options REQUIRED)
find_package(TRACE REQUIRED)


# We could expect that cetlib::cetlib, cetlib_except::cetlib_except, and 
# Folly::folly targets would all have been exported thanks to the 
# find_package() calls in daq_setup_dependencies

set(APPFWK_LIBRARIES cetlib::cetlib cetlib_except::cetlib_except Folly::folly )

##############################################################################

daq_point_build_to( src )

# Note that linking against appfwk means that the target will be able to pick 
# up the header paths it needs. This approach can also be found on Phil's 
# "Modern CMake" pull request. 

# If we always want to call target_include_directories in the manner below,
# it may be appropriate to have a daq_add_package_library function to do this

# In the target_link_libraries call below, pretend that "daq_libs" contains 
# third-party libraries universally useful to any DUNE DAQ library

add_library(appfwk SHARED src/QueueRegistry.cpp src/DAQProcess.cpp src/DAQModule.cpp)
target_link_libraries(appfwk ${APPFWK_LIBRARIES} daq_libs)
target_include_directories(appfwk
  PUBLIC
  $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
  $<INSTALL_INTERFACE:include>
)

add_library(appfwk_QueryResponseCommandFacility_duneCommandFacility src/QueryResponseCommandFacility.cpp)
target_link_libraries(appfwk_QueryResponseCommandFacility_duneCommandFacility appfwk)

macro(MakeDataTypeLibraries)
   set(oneValueArgs CPPTYPE PREFIX)
   cmake_parse_arguments(MAKE_DATATYPE "" "${oneValueArgs}" "" ${ARGN})

   configure_file(src/FanOutDAQModule.cpp.in src/${MAKE_DATATYPE_PREFIX}FanOutDAQModule.cpp)
   add_library(${MAKE_DATATYPE_PREFIX}FanOutDAQModule_duneDAQModule SHARED src/${MAKE_DATATYPE_PREFIX}FanOutDAQModule.cpp)
   target_link_libraries(${MAKE_DATATYPE_PREFIX}FanOutDAQModule_duneDAQModule appfwk)

endmacro()

MakeDataTypeLibraries(CPPTYPE std::vector<int> PREFIX VectorInt)
MakeDataTypeLibraries(CPPTYPE std::string PREFIX String)
MakeDataTypeLibraries(CPPTYPE dunedaq::appfwk::NonCopyableType PREFIX NonCopyableType)

##############################################################################
daq_point_build_to( apps )

# Pretend that "daq_exe_libs" contains libraries any DUNE DAQ application 
# would want
add_executable(daq_application apps/daq_application.cxx)
target_link_libraries(daq_application daq_exe_libs appfwk)

##############################################################################
daq_point_build_to( test )

add_library(appfwk_DummyModule_duneDAQModule test/DummyModule.cpp)

add_library(appfwk_FakeDataConsumerDAQModule_duneDAQModule test/FakeDataConsumerDAQModule.cpp)
target_link_libraries(appfwk_FakeDataConsumerDAQModule_duneDAQModule appfwk)

add_library(appfwk_FakeDataProducerDAQModule_duneDAQModule test/FakeDataProducerDAQModule.cpp)
target_link_libraries(appfwk_FakeDataProducerDAQModule_duneDAQModule appfwk)


add_executable(queue_IO_check test/queue_IO_check.cxx)
target_link_libraries(queue_IO_check daq_exe_libs appfwk)

add_executable(dummy_test_app test/dummy_test_app.cxx)
target_link_libraries(dummy_test_app daq_exe_libs appfwk appfwk_DummyModule_duneDAQModule)


file(COPY test/producer_consumer_dynamic_test.json DESTINATION test)

##############################################################################
daq_point_build_to( unittest )

daq_add_unit_test(DAQModule_test appfwk)
daq_add_unit_test(DAQSink_DAQSource_test appfwk)
daq_add_unit_test(FanOutDAQModule_test appfwk)
daq_add_unit_test(FollyQueue_test appfwk)
daq_add_unit_test(StdDeQueue_test appfwk)
daq_add_unit_test(ThreadHelper_test appfwk)

##############################################################################

# The second argument is a boolean signifying whether we expect this package 
# might ever be used as another package's dependency, or if we're just using 
# it for its executables 

# There's a lot going on under the covers, but it's always the same lot, thus
# the function

daq_install_package("appfwk_manifest.txt" TRUE)