## ProtoDUNE Single Phase Run II ### DAQ Computing planning ##### *Pengfei Ding, Alec Habig, Bonnie King, Geoffrey Savage* #### January 23 2020 --- ### Outline * Schedule * Dedicated time for changes or squeeze in when we can? * Computing hardware * Replace/remove "obsolete" devices * Transition to FELIX readout * New PD readout? * Operating system * Infrastructure * Software distribution --- Goal before Run II: Bring ProtoDUNE-SP closer to DUNE DAQ configuration --- Questions to answer: * What work is most useful/highest priority? * When can we do it (if travel, when is the best time)? * Can we fund upgrades? --- ### Schedule * How do we work with operations/Serhan? * Cold box running continues * Can some of this work be done during DAQ Mondays in the coming months? * What does the schedule allow for and how much time do we have? * Recommision DAQ (TPC readout [almost] fully FELIX based) * Introduce redesigned components as available --- Slides from F. Resnati, S. Tufanli on [ProtoDUNE-SP end of the year and plans for 2020](https://indico.fnal.gov/event/22704/contribution/1/material/slides/0.pdf). --- ## Schedule * Any dedicated improvement time after mid-April? * LAr emptying begins 20 April 2020 according to latest schedule (I've seen, from [PD-SP Ops meeting](https://indico.fnal.gov/event/22704/)) * Refilling in 2021 (month?) * recommission DAQ (TPC readout [almost] fully FELIX based) * introduce redisigned components as available --- ### Hardware - remove obsolete devices * Dells from 35Ton * 007, 008 - nfs servers * Replace with new servers or NAS * 009, 010 - monitoring servers * Adjust architecture to use these as gateways? * Logins only through a few computers that updated regularly --- ### Hadware - remove obsolete devices * onl nodes - usb connections for programming * Can the USB programmer(s) be replaced with workstation or small form factor computer? * Dells from just before protoDUNE SP run - 011 to 019 * Don't support FELIX * Support for PD readout * Use these in place of 35ton Dells? --- ### Hardware - data disk nodes * Software RAID with JBODs (Just a Bunch Of Disks) * Move to hardware raid * Adjust UPS configuration * Computers are attached to UPS, disks are not * Get UPS for JBODs? --- ### Hardware - data disk nodes * Rebuild raid arrays * Using RAID 5 now - go to raid 10? * RAID 10 has better write performance, 1/2 the space * if using regular RAID in DUNE, recommend RAID10 or RAID6 (even more write penalty than 5) with hot spares --- ### Hardware - network, cabling, PDUs * Add a DAQ network * Additional network switches * re-cable and relabel all cables * all devices, including cables, registered into a hardware database * When will this be available? * If not available can store in simple db (sqlite etc) * Network PDUs for remote power control --- ### Hardware * Budget-dependent wishlist: * New network switch with deeper buffers * iDRAC licenses for newer Dells * Network controlled PDUs * New servers * prioritize which ones to replace --- ### Operating System * Build dedicated kickstart server * Stop using central CERN kickstart server * Full OS Reinstalls * Ansible role audit * OS decision * CentOS7 is EOL June 2024, will last beyond Run II, but * 8 is closer to the OS likely to be used in DUNE * 9 (or later) will be out by then --- ### Infrastructure * enhanced server metrics * Introduce a monitoring check/alerting system (other than Prometheus)? * At FNAL, this is check_mk (with DAQ and some server metrics recorded by Ganglia, or now netdata/prometheus with Grafana) * check_mk used strictly for sysadmin operations --- ### Infrastructure * Own DNS server * is this possible/allowed/a good idea? * Own kickstart server * Any of these infrastructure items could run on existing machines --- ### Software distribution * spackdev/spack/CVMFS software deployment? --- More discussion to come at the collaboration meeting and DAQ Workshop