Publication: Data Handling Optimization in Russian Data Lake Prototype
Дата
2023
Авторы
Journal Title
Journal ISSN
Volume Title
Издатель
Аннотация
Abstract CERN experiments are preparing for the HL-LHC era, which will bring an unprecedented volume of scientific data. These data will need to be stored and processed by thousands of physicists, but expected resource growth is nowhere near the extrapolated requirements of existing models, in terms of both storage volume and compute power. Opportunistic CPU resources such as HPCs and university clusters can provide extra CPU cycles, but there is no opportunistic storage. In this article, we will present the main architectural ideas, deployment details, and test results, with emphasis on our research to build a prototype of a distributed data processing and storage system with a focus on optimizing the efficiency of resources by reducing overhead costs for accessing the data. The described prototype was built using the geographically distributed WLCG sites and university clusters in Russia.
Описание
Ключевые слова
Task Scheduling , Parallel Computing , Distributed Storage , High-Performance Computing , Computational Grids
Цитирование
Data Handling Optimization in Russian Data Lake Prototype / Alekseev, A. [et al.] // Journal of Physics: Conference Series. - 2023. - 2438. - № 1. - 10.1088/1742-6596/2438/1/012021
URI
https://www.doi.org/10.1088/1742-6596/2438/1/012021
https://www.scopus.com/record/display.uri?eid=2-s2.0-85149722430&origin=resultslist
http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=Alerting&SrcApp=Alerting&DestApp=WOS_CPL&DestLinkType=FullRecord&UT=WOS:001026601300021
https://openrepository.mephi.ru/handle/123456789/30034
https://www.scopus.com/record/display.uri?eid=2-s2.0-85149722430&origin=resultslist
http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=Alerting&SrcApp=Alerting&DestApp=WOS_CPL&DestLinkType=FullRecord&UT=WOS:001026601300021
https://openrepository.mephi.ru/handle/123456789/30034