SH Namespace Reference

Classes
class	DiskList
	an interface for listing directory contents, locally or on a file server More...
class	DiskListEOS
	a DiskList implementation for EOS More...
class	DiskListLocal
	a DiskList implementation for local directories More...
class	DiskListSRM
	a DiskList implementation for the SRM protocol More...
class	DiskListXRD
	a DiskList implementation for the XRD protocol More...
class	DiskOutput
	a class representing an output location for files More...
class	DiskOutputLocal
	an implementation of DiskOutput for local disks More...
class	DiskOutputXRD
	an implementation of DiskOutput using the XRD protocol More...
class	DiskWriter
	an interface that manages writing a single output file More...
class	DiskWriterLocal
	an implementation of DiskWriter for local files More...
class	DiskWriterXRD
	an implementation of DiskWriter for the XRD protocol More...
struct	RucioListDidsEntry
	one entry from the rucio-list-dids command More...
struct	RucioListFileReplicasEntry
	one entry from the rucio-list-file-replicas command More...
struct	RucioDownloadResult
	the result from rucio_download More...
class	Meta
	A base class for classes that implement arbitrary meta-data. More...
class	MetaData
	A class implementing a templatized version of the meta-data. More...
struct	MetaDataQuery
	an object containing the result of an AMI meta-data query More...
struct	MetaDataSample
	all the meta-data fields that we may try to read from AMI. More...
struct	MetaFields
	This class contains the known meta-data names. More...
struct	MetaNames
	the names of common meta-data More...
class	MetaObject
	A class that manages meta-data to be associated with an object. More...
class	MetaVector
	This class defines a templatized version of the meta-data in vector form. More...
class	Sample
	a base class that manages a set of files belonging to a particular data set and the associated meta-data. More...
class	SampleComposite
	This module defines an implementation of Sample that contains composite samples. More...
class	SampleGrid
	This class implements a Sample located on the grid. More...
class	SampleHandler
	A class that manages a list of Sample objects. More...
class	SampleHist
	A sample that represents a single histogram file. More...
class	SampleLocal
	A Sample based on a simple file list. More...
class	SampleMeta
	A Sample that consists only of Meta-Information. More...
class	SamplePtr
	A smart pointer class that holds a single Sample object. More...
struct	ScanDir
	the class used for scanning local directories and file servers for samples More...
class	TagList
	A class that manages a list of tags for identifying samples. More...
Functions
void	fetchMetaData (MetaDataQuery &query)
void	fetchMetaData (SH::SampleHandler &sh, bool override)
bool	checkVomsProxy ()
	return whether we have a valid VOMS proxy available
void	ensureVomsProxy ()
	ensure that we have a valid VOMS proxy available
std::vector< std::string >	faxListFilesGlob (const std::string &name, const std::string &filter)
	list the FAX URLs for all the files in the dataset or dataset container matching the given filter (as glob expression)
std::vector< std::string >	faxListFilesRegex (const std::string &name, const std::string &filter)
	list the FAX URLs for all the files in the dataset or dataset container matching the given filter (as regular expression)
std::vector< RucioListDidsEntry >	rucioListDids (const std::string &dataset)
	run rucio-list-dids for the given dataset
std::vector < RucioListFileReplicasEntry >	rucioListFileReplicas (const std::string &dataset)
	run rucio-list-file-replicas for the given dataset
std::map< std::string, std::unique_ptr< MetaObject > >	rucioGetMetadata (const std::set< std::string > &datasets)
	run rucio-get-metadata for the given list of datasets
RucioDownloadResult	rucioDownload (const std::string &location, const std::string &dataset)
	run rucio-download
std::vector< RucioDownloadResult >	rucioDownloadList (const std::string &location, const std::vector< std::string > &datasets)
	run rucio-download with multiple datasets
std::string	dbg (const Meta &, unsigned)
std::string	dbg (const TagList &obj, unsigned)
void	scanDir (SampleHandler &sh, DiskList &list, const std::string &pattern, const std::string &samplePattern, const std::string &samplePostfix)
void	scanDir (SampleHandler &sh, const std::string &dir)
void	scanDir (SampleHandler &sh, const std::string &dir, const std::string &prefix)
void	scanFiles (SampleHandler &sh, DiskList &list, const std::string &pattern)
Sample *	makeFromTChain (const std::string &name, const TChain &chain)
void	scanSingleDir (SampleHandler &sh, const std::string &name, DiskList &list, const std::string &pattern)
void	scanDQ2 (SampleHandler &sh, const std::string &pattern)
void	scanRucio (SampleHandler &sh, const std::string &pattern, bool alwaysQuery=false)
	make a list of grid datasets using the given pattern
void	addGrid (SampleHandler &sh, const std::string &ds)
void	makeGridDirect (SampleHandler &sh, const std::string &disk, const std::string &from, const std::string &to, bool allow_partial)
void	scanForTrees (SampleHandler &sh, const Sample &sample, const std::string &pattern)
void	scanForTrees (SampleHandler &sh, const std::string &pattern)
void	readFileList (SampleHandler &sh, const std::string &name, const std::string &file)
void	printDuplicateEvents (const Sample &sample)
void	printDuplicateEventsSplit (const SampleHandler &sh)
void	printDuplicateEventsJoint (const SampleHandler &sh)
void	mergeSamples (SampleHandler &sh, const std::string &sampleName, const std::string &pattern)
void	readSusyMeta (const SampleHandler &sh, const std::string &inputFile)
void	readSusyMetaDir (const SampleHandler &sh, const std::string &inputDir)
std::unique_ptr< TFile >	openFile (const std::string &name, const MetaObject &options)
	open a file with the given options
std::unique_ptr< SampleLocal >	mergeFiles (const Sample &sample, const std::string &location, bool overwrite)
	merge all the files in the sample into a single file in the given location
SampleHandler	mergeFiles (const SampleHandler &sh, const std::string &location, bool overwrite)
	merge all the files in the sample handles into a single file per sample in the given location
void	scanNEvents (SampleHandler &sh)
void	scanNEvents (Sample &sample)
SampleHandler	splitSample (Sample &sample, const Long64_t nevt)

Detailed Description

This module provides a lot of global definitions, forward declarations and includes that are used by all modules. As such it doesn't fall into the user vs. expert classification. This module is considered to be in the pre-alpha stage.

This module defines utility functions used for joining samples. The interface provided in this module is intended for the general user. The module is considered to be in the pre-alpha stage.

This module defines utility functions used for splitting samples. The interface provided in this module is intended for the general user. The module is considered to be in the pre-alpha stage.

Function Documentation

void SH::addGrid	(	SampleHandler &	sh,
		const std::string &	ds
	)

effects: add a grid dataset for dataset ds guarantee: strong failures: out of memory II requires: ds.find ("*") == std::string::npos

bool SH::checkVomsProxy ( )

return whether we have a valid VOMS proxy available

Guarantee: basic

Failures: grid utility failures

void SH::ensureVomsProxy ( )

ensure that we have a valid VOMS proxy available

First this checks whether we have a valid PROXY, and if not it sets up a new certificate.

Guarantee: basic

Failures: grid utility failures
failure to set up new VOMS proxy

std::vector< std::string > SH::faxListFilesGlob	(	const std::string &	name,
		const std::string &	filter
	)

list the FAX URLs for all the files in the dataset or dataset container matching the given filter (as glob expression)

Guarantee: strong

Failures: grid utility failures

Precondition:: !name.empty(); name.find('*') == std::string::npos; !filter.empty()

std::vector< std::string > SH::faxListFilesRegex	(	const std::string &	name,
		const std::string &	filter
	)

list the FAX URLs for all the files in the dataset or dataset container matching the given filter (as regular expression)

Guarantee: strong

Failures: grid utility failures

Precondition:: !name.empty(); name.find('*') == std::string::npos; !filter.empty()

void SH::fetchMetaData	(	SampleHandler &	sh,
		bool	override
	)

effects: fetch information on all the samples in the sample handler. if override is specified it will override the existing meta-data fields, otherwise they will only be used if the given fields do not yet exist guarantee: basic

void SH::fetchMetaData ( MetaDataQuery & query )

effects: fetch information on all the samples/datasets specified guarantee: basic

Sample * SH::makeFromTChain	(	const std::string &	name,
		const TChain &	chain
	)

effects: create a sample with the given name from the given TChain object guarantee: strong failures: out of memory II

void SH::makeGridDirect	(	SampleHandler &	sh,
		const std::string &	disk,
		const std::string &	from,
		const std::string &	to,
		bool	allow_partial
	)

effects: update all grid samples in the sample handler that are located on the given disk to be opened directly. for that it replaces the from part of the file with the to part. if allow_partial is set, it allows for partial datasets (meaning you may lose some files that are not there). guarantee: basic, may convert only some samples failures: out of memory III failures: dq2-ls errors failures: from not part of file name

SampleHandler SH::mergeFiles	(	const SampleHandler &	sh,
		const std::string &	location,
		bool	overwrite
	)

merge all the files in the sample handles into a single file per sample in the given location

This is mostly meant to allow merging histogram files (particularly in EventLoop), but in some cases it can also be used to merge n-tuple files together.

Returns:: a sample handler containing the merged files

Guarantee: basic

Failures: i/o errors
out of memory III

std::unique_ptr< SampleLocal > SH::mergeFiles	(	const Sample &	sample,
		const std::string &	location,
		bool	overwrite
	)

merge all the files in the sample into a single file in the given location

This is mostly meant to allow merging histogram files (particularly in EventLoop), but in some cases it can also be used to merge n-tuple fiels together.

Returns:: a sample containing the merged file

Guarantee: basic

Failures: i/o errors
out of memory III

void SH::mergeSamples	(	SampleHandler &	sh,
		const std::string &	sampleName,
		const std::string &	pattern
	)

effects: remove all samples matching the name pattern, and join them into a single sample named sampleName guarantee: strong failures: out of memory II failures: i/o errors

std::unique_ptr< TFile > SH::openFile	(	const std::string &	name,
		const MetaObject &	options
	)

open a file with the given options

for now this is to allow opening files with retries, but in the future there may be other options as well.

Guarantee: strong

Failures: i/o errors
file not found

Postcondition:: result != nullptr

void SH::printDuplicateEvents ( const Sample & sample )

effects: check the given sample for duplicate events and then print them out guarantee: basic, may print partially failures: out of memory III failures: i/o errors

void SH::printDuplicateEventsJoint ( const SampleHandler & sh )

effects: check for duplicate events between all the samples and then print them out guarantee: basic, may print partially failures: out of memory III failures: i/o errors

void SH::printDuplicateEventsSplit ( const SampleHandler & sh )

effects: check each sample for duplicate events and then print them out guarantee: basic, may print partially failures: out of memory III failures: i/o errors

void SH::readFileList	(	SampleHandler &	sh,
		const std::string &	name,
		const std::string &	file
	)

effects: read a file list from a text file guarantee: strong failures: out of memory III failures: i/o errors

void SH::readSusyMeta	(	const SampleHandler &	sh,
		const std::string &	inputFile
	)

effects: read the susy-meta-data file and add its information to the samples from the given sample handler guarantee: basic failures: i/o errors warning: there are no checks for duplicate and inconsistent sample definitions

void SH::readSusyMetaDir	(	const SampleHandler &	sh,
		const std::string &	inputDir
	)

effects: read an entire directory of susy-meta-data files and add their information to the samples from the given sample handle. for that it assumes that all files ending in ".txt" are susy-meta-data files. guarantee: basic failures: i/o errors warning: there are no checks for duplicate and inconsistent sample definitions

RucioDownloadResult SH::rucioDownload	(	const std::string &	location,
		const std::string &	dataset
	)

run rucio-download

Guarantee: basic

Failures: grid utility failures

std::vector< RucioDownloadResult > SH::rucioDownloadList	(	const std::string &	location,
		const std::vector< std::string > &	datasets
	)

run rucio-download with multiple datasets

Guarantee: basic

Failures: grid utility failures

std::map< std::string, std::unique_ptr< MetaObject > > SH::rucioGetMetadata ( const std::set< std::string > & datasets )

run rucio-get-metadata for the given list of datasets

Guarantee: strong

Failures: grid utility failures

Precondition:: !datasets.empty(); !dataset.empty() (for each dataset)

std::vector< RucioListDidsEntry > SH::rucioListDids ( const std::string & dataset )

run rucio-list-dids for the given dataset

Guarantee: strong

Failures: grid utility failures

Precondition:: !dataset.empty()

std::vector< RucioListFileReplicasEntry > SH::rucioListFileReplicas ( const std::string & dataset )

run rucio-list-file-replicas for the given dataset

Guarantee: strong

Failures: grid utility failures

Precondition:: !dataset.empty()

void SH::scanDir	(	SampleHandler &	sh,
		DiskList &	list,
		const std::string &	pattern = `".root"`,
		const std::string &	samplePattern = `"*"`,
		const std::string &	samplePostfix = `""`
	)

effects: scan the given directory and add all subdirectories as samples that contain root files. if prefix is provided, use that instead of dir when constructing the file names. guarantee: basic, only some samples might be added failures: low level errors IV failures: directory not found failures: duplicate samples rationale: the prefix option is for the Tier 3 prototype, where you scan the local directory, but then access the files through xrootd

void SH::scanDQ2	(	SampleHandler &	sh,
		const std::string &	pattern
	)

effects: make a list from DQ2 using the given pattern guarantee: basic, may add partially failures: out of memory III failures: dq2-ls errors

void SH::scanFiles	(	SampleHandler &	sh,
		DiskList &	list,
		const std::string &	pattern = `".root"`
	)

effects: scan the given directory tree and make a separate sample for each file (using the file name before any "." as the sample name). guarantee: basic, only some samples might be added failures: out of memory III failures: i/o errors failures: duplicate samples

void SH::scanForTrees	(	SampleHandler &	sh,
		const Sample &	sample,
		const std::string &	pattern = `".*"`
	)

effects: scan for trees in the given sample (or sample handler), and create a separate sample for each tree. if pattern is specified it is applied to the allowed tree names. guarantee: strong failures: out of memory III failures: i/o errors

void SH::scanNEvents ( Sample & sample )

effects: scan the given sample and store the number of entries per file in the meta-data guarantee: strong failures: out of memory failures: read errors failures: invalid sample type

void SH::scanNEvents ( SampleHandler & sh )

effects: scan each sample in the sample handler and store the number of entries per file in the meta-data guarantee: basic, may only scan some failures: out of memory failures: read errors failures: invalid sample type

void SH::scanRucio	(	SampleHandler &	sh,
		const std::string &	pattern,
		bool	alwaysQuery = `false`
	)

make a list of grid datasets using the given pattern

This will first look for dataset containers, and if none are found, it will instead look for datasets. If those are also not found it will produce an error.

For patterns that don't contain a wild-card (i.e. that can only match one dataset/container) it instead adds the dataset directly without querying rucio. This is a performance optimization based on the assumption that you probably checked the dataset names beforehand, and that if you made a mistake it will pop up at a later stage anyways. If you want to query rucio even in those cases set alwaysQuery to true.

Guarantee: basic, may add some datasets

Failures: no pattern match
rucio failures
out of memory III

void SH::scanSingleDir	(	SampleHandler &	sh,
		const std::string &	name,
		DiskList &	list,
		const std::string &	pattern = `".root"`
	)

effects: scan the given directory tree and turn it into a single sample of the given name guarantee: strong failures: out of memory III

SampleHandler SH::splitSample	(	Sample &	sample,
		Long64_t	nevt
	)

effects: split the given sample into a set of samples, with each sample containing either exactly one file or at most nevt events side effects: if scanNEvents hasn't been run on this sample, run it. guarantee: strong failures: out of memory failures: scanning errors

SH Namespace Reference

Classes

Functions

Detailed Description

Function Documentation