Running MCC11 jobs on Cori

tags: Offline

Running MCC11 jobs inside docker container

Before trying MCC11 jobs on Cori, I would like to go through the full gen-g4-detsim-reco chain in a SL7 docker container on a local machine.

Here I am using docker-bd.fnal.gov (use duneimagegpvm01.fnal.gov in the near future).

1. Copy over dunepro.Production.proxy to your local machine;
2. Start a generic sl7 container docker run --rm -it -v $PWD:/scratch -v /cvmfs:/cvmfs fermilab/fnal-wn-sl7
3. Setup software environment in the container and obtain valid voms proxy

export X509_USER_PROXY=/scratch/dunepro.Production.proxy
voms-proxy-init -noregen -rfc -voms dune:/dune/Role=Production
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
setup dunetpc  v08_24_00 -q e17:prof
setup duneutil  v08_24_00 -q e17:prof

4. Run a gen stage job with lar -c prodgenie_nue_dune10kt_1x2x6.fcl -n 2 -o /scratch/local_v082400_n2_gen.root |& tee -a /scratch/run_local_v082400_n2_gen.log
5. Run the downstream g4 stage job with lar -c standard_g4_dune10kt_1x2x6.fcl -s /scratch/local_v082400_n2_gen.root -n 2 -o /scratch/local_v082400_n2_g4.root |& tee /scratch/run_local_v082400_n2_g4.log
6. Run the downstream detsim stage job with lar -c standard_detsim_dune10kt_1x2x6.fcl -s /scratch/local_v082400_n2_g4.root -n 2 -o /scratch/local_v082400_n2_detsim.root |& tee /scratch/run_local_v082400_n2_detsim.log
7. Run the downstream reco stage job with lar -c standard_reco_dune10kt_1x2x6.fcl -s /scratch/local_v082400_n2_detsim.root -n 2 -o /scratch/local_v082400_n2_reco.root |& tee /scratch/run_local_v082400_n2_reco.log

In my test, all of the above jobs finished with:

Art has completed and will exit with status 0.

Making shifter image available on Cori

This can be easily done with shifterimg pull fermilab/fnal-wn-sl7 on Cori's login node.

Running MCC11 jobs inside shifter image on Cori interactively (more details will be added here. Haven't tested all steps)

1. Copy over dunepro's proxy to Cori;
2. Submit an interactive job;
3. Startup the shifter container;
4. Setup software enviroment and obtain voms proxy;
5. Follow similar step as above to run the MCC11 jobs stage-by-stage.

Things need to be addressed before production run on Cori

1. Use flux files from stash-cache instead of the default /pnfs location;
2. Stream output files to dCache instead of writing to local disk;
3. Are we keeping all the files from intermidiate stages?