MINOS running on Stampede

The HEP computing model is constantly evolving, and one change that is currently taking place is increased use of High Performance Computing (HPC) resources. Some of these HPC resources include supercomputing sites such as NERSC, as well as the EXtreme Science and Engineering Discovery Environment (XSEDE). XSEDE is actually a collection of several HPC resources, including the Stampede cluster at the Texas Advanced Computing Center.

The challenge of utilizing HPC resources is to interface with their job submission systems in a way that does not require any additional work by FIFE users, and to make sure that all usage is accounted for accurately. Proper accounting and access control is especially important at those resources that operate on an allocation basis, such as Stampede. Since job pilot resources on GPGrid are available for any group to use, SCD members had to make some modifications to the job submission infrastructure to ensure that only experiments with allocations are able to run jobs at these HPC resources. The changes are transparent to users.

The additional work has finally paid off. SCD has been working with MINOS to move some of their processing to Stampede, as a MINOS collaborator has an allocation there. The submission infrastructure is now in place and has been successfully tested. MINOS has been able to send jobs there, and they are working on modifying their workflow to run offsite. The modifications will also enable experiments with future allocations at Stampede or any number of other HPC resources to seamlessly submit jobs to GPGrid, the Open Science Grid, or their HPC allocations. As HEP increases its use of HPC resources, these tools will become even more valuable. If you are interested in learning more about accessing HPC resources or obtaining an allocation, please contact the FIFE group.

–Ken Herner