Nearly all forms of scientific computing at Fermilab require some form of non-volatile storage. While the primary storage format for scientific data at Fermilab is tape-backed mass storage systems (MSS, consisting of Enstore and dCache), there are a variety of other storage solutions available, depending on the type of scientific computing that needs to be accomplished. Network attached storage (NAS), which at Fermilab is primarily BlueArc systems, provides a good platform of POSIX-compliant storage for interactive computing. It does not, however, provide a robust platform for large-scale parallel access from grid jobs. Furthermore, NAS space is not easily accessible from off-site computing, such as jobs run on remote grid sites and clouds via GPGrid or the HEPCloud interface.
Up until 2016, most experiments at Fermilab had an allocation of NAS space that was also mounted on Fermigrid worker nodes. To ensure that the NAS remains performant for interactive use, GridFTP transfers from BlueArc space to worker nodes are throttled, but this throttling can cause delays to the starting of grid jobs in excess of 6 hours. SCD is undergoing a campaign to unmount and make inaccessible all NAS space from grid worker nodes. Users will continue to have access to MSS space from their grid jobs, including non-tape backed (dCache) MSS that provide a performant solution for short-term storage. Code and software, which in many cases was previously mounted on an “app” area on the NAS, should now be accessed on interactive and grid jobs via CVMFS repositories. The resulting storage architecture, including solutions and preferred use cases, is shown in the figure above.
The Data Movement and Storage department and the User Support for Distributed Computing department are working with experiment computing liaisons to ensure that this transition goes smoothly with the goal of having no dependency on the NAS for Grid computing by the beginning of 2018.
More details on the storage architecture can be found in this document: https://goo.gl/a16OxJ
Information on the BlueArc unmounting and how to avoid dependencies in grid jobs will be presented at the FIFE Workshop 2017.