---
# System prepended metadata

title: LArSoft on Theta
tags: [theta, adsp, larsoft]

---

# LArSoft on Theta

###### tags: `larsoft` `theta` `adsp` 

[ToC]

We'll place everything in `/grand/neutrino_osc_ADSP/`, which is a Lustre area.
Lustre is basically parallel IO, each file gets chunked and written to a bunch of different high performance drives. The idea is that when everyone is reading and writing, the actual work isn’t bottlenecked and everyone gets good performance. In reality, there is a metadata server that knows where the files actually live (file A chunked on drives 1, 2…,50…). The number of servers that file get written to is called the `stripe count`.

```bash=0
cd /grand/neutrino_osc_ADSP/software/larsoft/
mkdir products
chmod g+w products/
# Set number of severs files in products are written to
lfs setstripe -c 40 products/
# Create a folder where we can run pullProducts
mkdir tars_products
cd tars_products
# Place the pullProduct script here and run it:
./pullProducts ../products/ slf7 sbnd-v09_24_02 e20 prof
./pullProducts ../products/ slf7 icarus-v09_25_00 e20 prof
```

To get it to work for ICARUS, the following is also required:
```bash=
./pullProducts ../products/ slf7 sbn-v09_25_00 e20 prof
```
TODO: following error appears when setting up icaruscode:
```bash=
Singularity> source /grand/neutrino_osc_ADSP/software/larsoft/products/setup
Singularity> setup icaruscode v09_25_00 -q e20:prof
ERROR: Found no match for product 'icarus_data'
ERROR: Action parsing failed on "setupRequired( icarus_data v09_25_00 )"
```
although I can still run prodsingle jobs...not sure why `icarus_data` is not in the pullProducts manifest.



## Singularity

Corey built a singularity image from a docker image made by [FNAL](https://hub.docker.com/r/fermilab/fnal-wn-sl7/): `singularity build fnal-wn-sl7.sing docker://fermilab/fnal-wn-sl7:latest`.
We will bind-mount the `grand` area at runtime, so we can setup larsoft.

To test it with a prodsingle file:

```bash=0
singularity exec -B /grand /grand/neutrino_osc_ADSP/containers/fnal-wn-sl7.sing bash
Singularity> source /grand/neutrino_osc_ADSP/software/larsoft/products/setup
Singularity> setup sbndcode v09_24_02 -q e20:prof
Singularity> 
Singularity> lar -c prodsingle_mu_bnblike.fcl -n 10
```



## Flux Files

### SBND Fluxes
SBND flux files in neutrino mode are in `/cvmfs/sbnd.osgstorage.org/pnfs/fnal.gov/usr/sbnd/persistent/stash/fluxFiles/bnb/BooNEtoGSimple/configH-v1/march2021/neutrinoMode/`. They have been copyied to Theta at this location:

```bash=
/grand/neutrino_osc_ADSP/files/flux/sbnd/neutrino_mode/
```
In total, they take 69 GB of disk space.

### ICARUS Fluxes

To be added

### MicroBooNE Fluxes

To be added





