# Datasets handling The framework assigns a specific meaning to: *datasets*, *part*, *samples*, *sub-samples*. * **Dataset**: a dataset is a set of NanoAOD file containing events generated with the same configuration. Each dataset has a set of metadata, a unique datataking period, an *isMC* attribute. Each dataset groups files from one or more "CMS datasets" names (e.g. `"/ttHTobb_M125_TuneCP5_13TeV-powheg-pythia8/RunIISummer.....-v2/NANOAODSIM`). Dataset can have a **part** metadata, which is usually also part of the name: for example the dataset `WjetsHT200-500_2016APV` should have the metadata **part=HT200-500**: inside the framework the part metadata can be used to define customizations. ::::{important} The dataset name must be unique inside the framework and it is used to identify the output objects. The `build_datasets.py` script, defined below, makes sure that the final dataset name is the composition of the user defined label, the datataking period (and the era for data), the part metadata. :::: * **Sample**: a sample is a set of events representing a common physics process. The *sample name* can be seen as the **category** of events: multiple *datasets* may have the same *sample* name. Inside the framework the sample label is used to categorize the events and to customize weights, variables and categories. * **Subsample**: a subsample is a subset of a sample: it can be seen as a categorization applied only to events part of a specific sample. Subsamples are configured in a specific way using Cut objets: look at the [Configuration](./configuration.md) docs for more details. ## Datasets definition files Input datasets for the analyses are defined in a JSON file following the syntax below: ```python { "DYJetsToLL_M-50": { "sample": "DYJetsToLL", "json_output": "datasets/DYJetsToLL_M-50.json", "files": [ { "das_names": [ "/DYJetsToLL_M-50_TuneCP5_13TeV-amcatnloFXFX-pythia8/RunIISummer20UL18NanoAODv9-106X_upgrade2018_realistic_v16_L1v1-v2/NANOAODSIM" ], "metadata": { "year": "2018", "isMC": true, "xsec": 6077.22, "part": "M-50", }, "dbs_instance": "prod/global", # No need to include it, using prod/global by default } ], }, "DATA_SingleMuon": { "sample": "DATA_SingleMuon", "json_output": "datasets/DATA_SingleMuon.json", "files": [ { "das_names": [ "/SingleMuon/Run2018A-UL2018_MiniAODv2_NanoAODv9-v2/NANOAOD" ], "metadata": { "year": "2018", "isMC": false, "primaryDataset": "SingleMuon", "era": "A", }, }, { "das_names": [ "/SingleMuon/Run2018B-UL2018_MiniAODv2_NanoAODv9-v2/NANOAOD" ], "metadata": { "year": "2018", "isMC": false, "primaryDataset": "SingleMuon", "era": "B", }, "dbs_instance": "prod/global", }, ], }, } ``` The framework uses this definition to build the actual dataset json file. * The user defined label makes the dataset unique. * The `sample` key is the one used internally in the framework to identify the sample type (see explanation above). * The `json_output` defines the output location for the list of files. The same dataset can contain different group of dataset (**DAS**) names, each with a **separate metadata dictionary**. Each group will be interpreted by the `build_datasets` script to create unique set of files, with a unique label build as `{user_defined_label}__{part}__{year}_{Era}`. ## Create the Dataset JSON Files To build a JSON file with a list of datasets, run the following script: ```bash ___ _ __ _____ __ __ / _ )__ __(_) /__/ / _ \___ _/ /____ ____ ___ / /_ / _ / // / / / _ / // / _ `/ __/ _ `(_-