---
# System prepended metadata

title: Script to split a SAM dataset definition

---

# Script to split a SAM dataset definition

Here is the [gist of the script](https://gist.github.com/dingp/36aa66247fd3d4fa70e307a491a17e7a
).

The script will take an exisiting SAM dataset definition as input, and produce a list of datasets, each containing a subset of files in the input dataset.

It does so by following these steps:
1. Count total number of files in the input dataset. If `--max-files-per-set` is provided, calculate how many subsets to be created, and ignore `--nsubsets` if necessary;
2. if `--nsubsets` is specified, and the estimated number of files per subset is smaller than `--max-files-per-set`, the number specified by `--nsubsets` will be the number of subsets created;
3. take a snapshot of the input dataset and create a new dataset definition with constraints on the snapshot ID. This is the superset of the new subsets;
4. create each subsets by specifying the snapshot ID and the range of snapshot file number.

## Environment setup and authentication

You can do the setup in either of the following ways on a DUNE GPVM node. 

### method 1 (if you are using `dunesw`)

```bash
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
export DUNESW_VERSION=v09_72_01d00 # change this to the version available on cvmfs.
setup dunesw $DUNESW_VERSION -q e20:prof
setup_fnal_security
```

### method 2 (if you are setting up `samweb_client` only)

```bash
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
setup sam_web_client
export SAM_EXPERIMENT=dune

kx509
export ROLE=Analysis
voms-proxy-init -rfc -noregen -voms=dune:/dune/Role=$ROLE -valid 120:00
```

## Usage of the script

### Script helper

The helper of the script can be found below. The script will firstly print out the details of the input dataset, and the list of new datasets to be created. By default, the script will ask for confirmation before proceeding to create the new datasets. Use `-y/--yes` to bypass the confirmation prompt.

```bash
-bash-4.2$ ./split_sam_dataset.py -h
usage: split_sam_dataset.py [-h] [--dataset-name DATASET_NAME] [--prefix PREFIX]
                            [--suffix SUFFIX] [-y] [--max-files-per-set MAX_FILES_PER_SET]
                            [--nsubsets NSUBSETS]

Split SAM dataset into subsets.

optional arguments:
  -h, --help            show this help message and exit
  --dataset-name DATASET_NAME
                        SAM dataset name;
  --prefix PREFIX       Prefix to the dataset name after the split;
  --suffix SUFFIX       Suffix to the dataset name after the split;
  -y, --yes             batch mode, bypassing the prompt;
  --max-files-per-set MAX_FILES_PER_SET
                        Maixum number of files per subset (takes precendence over --nsubsets
                        if speficied);
  --nsubsets NSUBSETS   Number of subsets to be split into.

```

### Example 1 - Split a dataset into 10 subsets

```bash
./split_sam_dataset.py --prefix=dingpf --nsubsets 10 --dataset-name prodcosmics_corsika_protodunedp_mcc10
```

```bash
-bash-4.2$ ./split_sam_dataset.py --prefix=dingpf --nsubsets 10 --dataset-name prodcosmics_corsika_protodunedp_mcc10
--------------------------------------------------------------------------------
Definition Name: prodcosmics_corsika_protodunedp_mcc10
  Definition Id: 74222
  Creation Date: 2018-08-12T05:56:38+00:00
       Username: dunepro
          Group: dune
     Dimensions: data_tier simulated and file_format artroot and application detsim and version v06_70_01 and dune.campaign mcc10 and file_name
cosmics_protodunedp_%.root
--------------------------------------------------------------------------------
 Total file count: 1065
Number of subsets: 10
--------------------------------------------------------------------------------
The following datasets will be created:
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_all_10_subsets (1065 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_1_out_of_10    (107 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_2_out_of_10    (107 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_3_out_of_10    (107 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_4_out_of_10    (107 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_5_out_of_10    (107 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_6_out_of_10    (106 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_7_out_of_10    (106 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_8_out_of_10    (106 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_9_out_of_10    (106 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_10_out_of_10    (106 files)

Do you want to continue? (y/n): y
Continuing...
--------------------------------------------------------------------------------
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_all_10_subsets -- 1065 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_1_out_of_10 -- 107 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_2_out_of_10 -- 107 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_3_out_of_10 -- 107 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_4_out_of_10 -- 107 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_5_out_of_10 -- 107 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_6_out_of_10 -- 106 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_7_out_of_10 -- 106 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_8_out_of_10 -- 106 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_9_out_of_10 -- 106 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052057_10_out_of_10 -- 106 files
```

### Example 2 -- Split a dataset into subsets with a maximum number of 100 files per subset.
```bash
-bash-4.2$ ./split_sam_dataset.py --prefix=dingpf --max-files-per-set 100 --nsubsets 10 --data
set-name prodcosmics_corsika_protodunedp_mcc10
--------------------------------------------------------------------------------
WARNING: --max-files-per-set is set.
WARNING: need to create at least 11 subsets.
WARNING: ignore --nsubsets=10.
--------------------------------------------------------------------------------
Definition Name: prodcosmics_corsika_protodunedp_mcc10
  Definition Id: 74222
  Creation Date: 2018-08-12T05:56:38+00:00
       Username: dunepro
          Group: dune
     Dimensions: data_tier simulated and file_format artroot and application detsim and version v06_70_01 and dune.campaign mcc10 and file_name
cosmics_protodunedp_%.root
--------------------------------------------------------------------------------
 Total file count: 1065
Number of subsets: 11
--------------------------------------------------------------------------------
The following datasets will be created:
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_all_11_subsets (1065 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_1_out_of_11    (97 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_2_out_of_11    (97 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_3_out_of_11    (97 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_4_out_of_11    (97 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_5_out_of_11    (97 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_6_out_of_11    (97 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_7_out_of_11    (97 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_8_out_of_11    (97 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_9_out_of_11    (97 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_10_out_of_11    (96 files)
dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_11_out_of_11    (96 files)

Do you want to continue? (y/n): y
Continuing...
--------------------------------------------------------------------------------
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_all_11_subsets -- 1065 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_1_out_of_11 -- 97 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_2_out_of_11 -- 97 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_3_out_of_11 -- 97 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_4_out_of_11 -- 97 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_5_out_of_11 -- 97 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_6_out_of_11 -- 97 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_7_out_of_11 -- 97 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_8_out_of_11 -- 97 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_9_out_of_11 -- 97 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_10_out_of_11 -- 96 files
Created -- dingpf_prodcosmics_corsika_protodunedp_mcc10_202330052100_11_out_of_11 -- 96 files

```