DataPipe is the sub-system of the DPPS (Data Processing and Preservation System) responsible for DL0-DL3 data processing for CTAO and is comprised of ctapipe, pyirf, a benchmarking suite, DIRAC workflows, and other software.
Covered subjects
The meeting will be organized around these five DataPipe top-level workflows
- Reconstruction Model Training
- IRF Generation
- Benchmarking
- Observation Data Processing
- Quality Metrics
Attendee Survey
A moderately long survey can be found here
Structure
This event will days with interleaved introductory presentations, hands on demos of existing software, discussions about future needs, and hacking sessions relevant for the subjects of the day.
The discussions will aim to cover the following items
- Functionality: what is done, what needs to be done?
- Lessons from past experience: What problems and solutions in LST, NectarCam, HESS/MAGIC/VERITAS/etc are relevant for DataPipe?
- Tools, Workflows & Configuration: what executables/standard-scripts are needed and how we will automate this with DIRAC or other systems? Not just for one bin in observation space, but thinking toward the final goal of hundreds... What physics studies are needed to define the standard configuration?
- Verification: are the requirements and/or concepts well-enough specified? Or do we need to better explain things? Do we all agree what is needed and what is the scope?
- Validation/Benchmarking: how will we check the requirements are met? How do we check if a release is ok?
- Interfaces: how do internal DataPipe sub-systems or external systems outside of DataPipe connect to (and depend on) each other, and what needs to be defined (data formats, etc)
