# Running MCC11 jobs on Cori

###### tags: `Offline`

## Running MCC11 jobs inside docker container

Before trying MCC11 jobs on Cori, I would like to go through the full `gen-g4-detsim-reco` chain in a SL7 docker container on a local machine.

Here I am using `docker-bd.fnal.gov` (use `duneimagegpvm01.fnal.gov` in the near future).

1. Copy over `dunepro.Production.proxy` to your local machine;
2. Start a generic sl7 container ` docker run --rm -it -v $PWD:/scratch -v /cvmfs:/cvmfs fermilab/fnal-wn-sl7`
3. Setup software environment in the container and obtain valid voms proxy

export X509_USER_PROXY=/scratch/dunepro.Production.proxy
voms-proxy-init -noregen -rfc -voms dune:/dune/Role=Production
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
setup dunetpc  v08_24_00 -q e17:prof
setup duneutil  v08_24_00 -q e17:prof

4. Run a `gen` stage job with `lar -c prodgenie_nue_dune10kt_1x2x6.fcl -n 2 -o /scratch/local_v082400_n2_gen.root |& tee -a /scratch/run_local_v082400_n2_gen.log`
5. Run the downstream `g4` stage job with `lar -c standard_g4_dune10kt_1x2x6.fcl -s /scratch/local_v082400_n2_gen.root -n 2 -o /scratch/local_v082400_n2_g4.root |& tee /scratch/run_local_v082400_n2_g4.log`
6. Run the downstream `detsim` stage job with `lar -c standard_detsim_dune10kt_1x2x6.fcl -s /scratch/local_v082400_n2_g4.root -n 2 -o /scratch/local_v082400_n2_detsim.root |& tee /scratch/run_local_v082400_n2_detsim.log`
7. Run the downstream `reco` stage job with `lar -c standard_reco_dune10kt_1x2x6.fcl -s /scratch/local_v082400_n2_detsim.root -n 2 -o /scratch/local_v082400_n2_reco.root |& tee /scratch/run_local_v082400_n2_reco.log`

In my test, all of the above jobs finished with:

> Art has completed and will exit with status 0.
>

## Making shifter image available on Cori

This can be easily done with `shifterimg pull fermilab/fnal-wn-sl7` on Cori's login node.

## Running MCC11 jobs inside shifter image on Cori interactively (more details will be added here. Haven't tested all steps)

1. Copy over dunepro's proxy to Cori;
2. Submit an interactive job;
3. Startup the shifter container;
4. Setup software enviroment and obtain voms proxy;
5. Follow similar step as above to run the MCC11 jobs stage-by-stage.

## Things need to be addressed before production run on Cori

1. Use flux files from stash-cache instead of the default `/pnfs` location;
2. Stream output files to dCache instead of writing to local disk;
3. Are we keeping all the files from intermidiate stages?