Integration & Validation of a Standardized Library & File Format for PyDARNMUSIC & DARNtids: Migrating From Legacy Pickle Files to HDF5 & Implementing a Comprehensive Testing Suite

The PyDARN Multiple Signal Classification (PyDARN-MUSIC) algorithm is a general, widely used sign processing algorithm for determining the parameters of multiple wavefronts arriving at an antenna array from measurements made on the signals received at the array elements. It has been used in previous Super Dual Auroral Radar Network (SuperDARN) studies to determine the size, strength, speed, and direction of Medium Scale Traveling Ionospheric Disturbances (MSTIDs). DARNtids (Dual Auroral Radar Network Traveling Ionospheric Disturbances) is a python-based open-source software toolkit that first detects level of MSTID activity in SuperDARN data and then runs the MSTID MUSIC analysis software on event periods with significant MSTIDs. DARNtids can run the SuperDARN MUSIC algorithm automatically on thousands  individual analysis periods. Both PyDARN-MUSIC and DARNtids use pickle files for storing Python objects  byte streams. Pickle files are fragile and cannot be easily manipulated and used on modern systems. They are specific to Python, produce massive serialized values, and are slower than alternative options. As result, both repositories have experienced software engineering challenges related to portability, interoperability, testability, and maintainability, thereby forcing runs to take significantly longer than necessary, data to be stored inefficiently, and the debugging process to be difficult. We present the methodology taken to seamlessly replace dependencies on pickle files in PyDARN-MUSIC and DARNtids integrating the Hierarchical Data Format Version 5 (HDF5) file format and library. We further discuss validation techniques via the newly implemented PyDARN-MUSIC test suite to ensure numerical data accuracy between existing pickle file results and new HDF5 result.