Data federation strategies for ATLAS using XRootD

Citations Over TimeTop 10% of 2014 papers

Abstract

In the past year the ATLAS Collaboration accelerated its program to federate data storage resources using an architecture based on XRootD with its attendant redirection and storage integration services. The main goal of the federation is an improvement in the data access experience for the end user while allowing more efficient and intelligent use of computing resources. Along with these advances come integration with existing ATLAS production services (PanDA and its pilot services) and data management services (DQ2, and in the next generation, Rucio). Functional testing of the federation has been integrated into the standard ATLAS and WLCG monitoring frameworks and a dedicated set of tools provides high granularity information on its current and historical usage. We use a federation topology designed to search from the site's local storage outward to its region and to globally distributed storage resources. We describe programmatic testing of various federation access modes including direct access over the wide area network and staging of remote data files to local disk. To support job brokering decisions, a time-dependent cost-of-data-access matrix is made taking into account network performance and key site performance factors. The system's response to production-scale physics analysis workloads, either from individual end-users or ATLAS analysis services, is discussed.

Related Papers

• Information granularity,information entropy and decision tree(2009)
• → Temporal Granularity(1995)10 cited
• → A Study of a New Colour Granularity Formula(1981)1 cited
• Study of Approximate Periodicity Mining with Multi-granularity Time(2010)
• Susquehanna Chorale Spring Concert "Roots and Wings"(2017)