Publication: Logs analysis to search for anomalies in the functioning of large technology platforms
Дата
2019
Авторы
Journal Title
Journal ISSN
Volume Title
Издатель
Аннотация
© 2005–ongoing JATIT and LLS. All rights reserved.Today, with the widespread use of machine learning methods in various fields of human activity, the detection of rare events still remains one the most challenging tasks. This is due to the fact that there is very little information to learn computers to detect deviations from normal operation, although it has to deal with the processing of very large amounts of data that characterize the ongoing processes. This occurs, for example, in high energy physics, when searching for and studying new particles. The similar situation occurs when detecting pre-anomalous situations in the complex high-tech equipment operation. Logs are the only source of information to detect the processes running on such equipment, therefore many IT companies use them to analyze the functioning of their software and hardware technologies. This allows viewing the logs starting from very beginning to the point of failure completion, consistently figuring out the possible causes of the incident. In most companies, this process is not automated, because there is no single established approach to analyze logs of different configurations of stored metric values and different filling intensities. In addition, historical logs are not used to predict the sequence of events that lead to anomalies in the operation of any software technologies. The present article deals with the problem of detecting states and predicting the nearest behavior of large technological platforms by directional analysis of their logs. Usually, logs of large technology platforms represent data sets of very high dimensionality that does not allow modern algorithms in the allowable time limits to draw the necessary conclusions about the behavior of platforms and form sequence of control actions, if necessary. To solve this problem, the article compares the effectiveness of existing algorithms, traditionally used unsupervised learning, because the available data for learning are too small, as well as algorithms working with big data. Pilot implementations of all algorithms involved in solving the problem, performed in Python programming language, have been studied in a single environment. Based on their comparison, the most efficient algorithm was chosen, when recognizing different types of events based on real data. The solution of the chosen algorithm was implemented using Apache Spark framework. Additional investigation has shown that the selected algorithm can work in real time mode.
Описание
Ключевые слова
Цитирование
Dunaev, M. Logs analysis to search for anomalies in the functioning of large technology platforms / Dunaev, M., Zaytsev, K. // Journal of Theoretical and Applied Information Technology. - 2019. - 97. - № 11. - P. 3111-3123