# LArSoft on Theta ###### tags: `larsoft` `theta` `adsp` [ToC] We'll place everything in `/grand/neutrino_osc_ADSP/`, which is a Lustre area. Lustre is basically parallel IO, each file gets chunked and written to a bunch of different high performance drives. The idea is that when everyone is reading and writing, the actual work isn’t bottlenecked and everyone gets good performance. In reality, there is a metadata server that knows where the files actually live (file A chunked on drives 1, 2…,50…). The number of servers that file get written to is called the `stripe count`. ```bash=0 cd /grand/neutrino_osc_ADSP/software/larsoft/ mkdir products chmod g+w products/ # Set number of severs files in products are written to lfs setstripe -c 40 products/ # Create a folder where we can run pullProducts mkdir tars_products cd tars_products # Place the pullProduct script here and run it: ./pullProducts ../products/ slf7 sbnd-v09_24_02 e20 prof ./pullProducts ../products/ slf7 icarus-v09_25_00 e20 prof ``` To get it to work for ICARUS, the following is also required: ```bash= ./pullProducts ../products/ slf7 sbn-v09_25_00 e20 prof ``` TODO: following error appears when setting up icaruscode: ```bash= Singularity> source /grand/neutrino_osc_ADSP/software/larsoft/products/setup Singularity> setup icaruscode v09_25_00 -q e20:prof ERROR: Found no match for product 'icarus_data' ERROR: Action parsing failed on "setupRequired( icarus_data v09_25_00 )" ``` although I can still run prodsingle jobs...not sure why `icarus_data` is not in the pullProducts manifest. ## Singularity Corey built a singularity image from a docker image made by [FNAL](https://hub.docker.com/r/fermilab/fnal-wn-sl7/): `singularity build fnal-wn-sl7.sing docker://fermilab/fnal-wn-sl7:latest`. We will bind-mount the `grand` area at runtime, so we can setup larsoft. To test it with a prodsingle file: ```bash=0 singularity exec -B /grand /grand/neutrino_osc_ADSP/containers/fnal-wn-sl7.sing bash Singularity> source /grand/neutrino_osc_ADSP/software/larsoft/products/setup Singularity> setup sbndcode v09_24_02 -q e20:prof Singularity> Singularity> lar -c prodsingle_mu_bnblike.fcl -n 10 ``` ## Flux Files ### SBND Fluxes SBND flux files in neutrino mode are in `/cvmfs/sbnd.osgstorage.org/pnfs/fnal.gov/usr/sbnd/persistent/stash/fluxFiles/bnb/BooNEtoGSimple/configH-v1/march2021/neutrinoMode/`. They have been copyied to Theta at this location: ```bash= /grand/neutrino_osc_ADSP/files/flux/sbnd/neutrino_mode/ ``` In total, they take 69 GB of disk space. ### ICARUS Fluxes To be added ### MicroBooNE Fluxes To be added