NONPARAMETRIC ASSESSMENT OF THE PREDICTIVE ACCURACY OF MODELS USING TIME SERIES OF METHANE CONCENTRATIONS IN THE ATMOSPHERIC AIR OF THE ARCTIC ISLAND OF BELY

Over the past few decades, the number, variety, and complexity of time series forecasting models have grown. There has also been an increased interest among researchers in comparing and assessing the predictive accuracy and performance of models, and in determining which models are more accurate. The researchers propose using the hypothesis testing approach to assess the performance of a time series forecast model. They obtained data for the study while monitoring the dynamics of ground-level concentrations of the main greenhouse gases on the Arctic Island of Bely, Yamalo-Nenets Autonomous Area, Russia. A total of three models based on autoregressive neural networks with exogenous input (NARX) were considered to predict changes in methane concentration in the surface layer of atmospheric air. The performance of the models was assessed using 8 “traditional” indices (correlation and determination coefficients, Wilmott goodness-of-fit indices, mean absolute error, relative mean square error, etc.) and the proposed permutational approach using the hypothesis testing method. In general, based on the data under study, the estimates of the permutation approach and other indicators of accuracy and error coincide, but there are discrepancies. Effect sizes do not always determine the statistical significance of differences.

1. Summary for Policymakers. Climate Change 2021: The Physical Science Basis. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. V. Masson-Delmotte, P. Zhai, A. Pirani, S. L. Connors, C. Péan, S. Berger, N. Caud, Y. Chen, L. Goldfarb, M. I. Gomis, M. Huang, K. Leitzell, E. Lonnoy, J. B. R. Matthews, T. K. Maycock, T. Waterfield, O. Yelekçi, R. Yu., B. Zhou (eds.). Cambridge, New York, Cambridge Univ. Press, 2021, pp. 3—32. Available at: https://doi.org/10.1017/9781009157896.001.

2. Summary for Policymakers. Climate Change 2023: Synthesis Report. Contribution of Working Groups I, II and III to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. H. Lee and J. Romero (eds). Geneva, 2023, pp. 1—34. DOI: 10.59327/IPCC/AR6-9789291691647.001.

3. Gurlev I. V., Makosko A. A., Malygin I. G. Ecological problems of the Russian Arctic Zone on the example of the Yamal-Nenets Autonomous Area. Arctic: Ecology and Economy, 2024, vol. 14, no. 3, pp. 370—383. DOI: 10.25283/2223-4594-2024-3-370-383. (In Russian).

4. Bogoyavlensky V. I., Nikonov R. A., Bogoyavlensky I. V. New data on intensive Earth degassing in the Arctic in the north of Western Siberia: thermokarst lakes with gas blowout craters and mud volcanoes. Arctic: Ecology and Economy, 2023, vol. 13, no. 3, pp. 353—368. DOI: 10.25283/2223-4594-2023-3-353-368. (In Russian).

5. Rakhmatova A., Sergeev A., Shichkin A., Buevich A., Baglaeva E. Three-day forecasting of greenhouse gas CH4 in the atmosphere of the Arctic Belyy Island using discrete wavelet transform and artificial neural networks. Neural Comput & Applic, 2021, vol. 33, pp. 10311—10322. Available at: https://doi.org/10.1007/s00521-021-05792-3.

6. Radersma R., Sheldon B. C. A new permutation technique to explore and control for spatial autocorrelation. Methods Ecol Evol, 2015, vol. 6, pp. 1026—1033. Available at: https://doi.org/10.1111/2041-210X.12390.

7. Ross S. M. Simulation, bootstrap statistical methods, and permutation tests. Introduction to Probability and Statistics for Engineers and Scientists. [S. l.], 2021, pp. 619—647. DOI: 10.1016/B978-0-12-824346-6.00024-7.

8. Berry K. J., Johnston J. E., Mielke P. W. A Primer of Permutation Statistical Methods. Cham, Switzerland,  Springer-Verlag, 2019, 476 p. DOI: 10.1007/978-3-030-20933-9.

9. Berry K. J., Johnston J. E., Mielke P. W., Johnston L. A. Permutation methods. Part II. Wiley Interdisciplinary Reviews: Computational Statistics, 2018, vol. 10 (3), e1429. DOI: 10.1002/wics.1429.

10. Poddubny V. A., Nagovitsyna E. S., Markelov Y. I., Buevich A. G., Antonov K. L., Omel’kova E. V., Manzhurov I. L. Assessment of the spatial distribution of methane concentration in the Barents and Kara Seas in the summer 2016—2017. Russian Meteorology and Hydrology, 2020, vol. 45, no. 3, pp. 193—200. Available at: https://doi.org/10.15372/AOO20230204.

11. Subbotina I. E., Buevich A. G., Sergeev A. P., Shichkin A. V., Baglaeva E. M., Remezova M. S. Two-step combined algorithm for improving the accuracy of predicting methane concentration in atmospheric air based on the NARX neural network and subsequent prediction of residuals. Arctic: Ecology and Economy, 2020, no. 2 (38), pp. 59—67. Available at: https://doi.org/10.25283/2223-4594-2020-2-59-67. (In Russian).

12. Buevich A., Sergeev A., Shichkin A., Baglaeva E. A two-step combined algorithm based on NARX neural network and the subsequent prediction of the residues improves prediction accuracy of the greenhouse gases concentrations. Neural Computing and Applications, 2021, vol. 33 (1), pp. 1547—1557. DOI: 10.1007/s00521-020-04995-4.

13. Cai L. Multi-response permutation procedure as an alternative to the analysis of variance: An SPSS implementation. Behavior Research Methods, 2006, vol. 38 (1), pp. 51—59. Available at: https://doi.org/10.3758/BF03192749.

14. Fişek M. H., Barlas Z. Permutation tests for goodness-of-fit testing of mathematical models to experimental data. Social Science Research, 2013, vol. 42 (2), pp. 482—495. DOI: 10.1016/j.ssresearch.2012.09.010.

15. Hart J. D. A., Weiss M. N., Brent L. J. N., Franks D. W. Common permutation methods in animal social network analysis do not control for non-independence. Behav Ecol Sociobiol, 2022, vol. 76, p. 151. Available at: https://doi.org/10.1007/s00265-022-03254-x.

16. Kończak G. Applications of Permutation Methods in the Analysis of Associations. Argumenta Oeconomica Cracoviensia, 2020, vol. 1 (22), pp. 31—45. Available at: https://doi.org/10.15678/AOC.2020.2203.

17. Mielke P. W., Berry K. J. Permutation Methods. A Distance Function Approach. Springer Series in Statistics. New York, Springer, 2001, 446 p. DOI: 10.1007/978-1-4757-3449-2/.

18. Taylor A. B., MacKinnon D. P. Four applications of permutation methods to testing a single-mediator model. Behavior Research Methods, 2012, vol. 44 (3), pp. 806—844. DOI: 10.3758/s13428-011-0181-x/.

19. Fekete A., Fuchs S., Garschagen M., Hutter G., Klepp S., Lüder C., Neise T., Sett D., von Elverfeldt K., Wannewitz M. Adjustment or transformation? Disaster risk intervention examples from Austria, Indonesia, Kiribati and South Africa. Land Use Policy, 2022, vol. 120, p. 106230. Available at: https://doi.org/10.1016/j.landusepol.2022.106230.

20. Willmott C. J., Ackleson S., Davis R., Feddema J., Klink K., Legates D., O’Donnell J., Rowe C. Statistics for the Evaluation and Comparison of Models. J. of Geophysical Research, 1985, vol. 90, no. Ñ5, pp. 8995—9005. Available at: https://doi.org/10.1029/JC090iC05p08995.

21. Trofimova I., Balybina A. Classification of climates and climatic regionalization of the West-Siberian plain. Geography and Natural Resources, 2014, vol. 35, pp. 114—122. Available at: https://doi.org/10.1134/S1875372814020024.

22. Daneshfaraz R., Norouzi R., Abbaszadeh H., Azamathulla H. Theoretical and experimental analysis of applicability of sill with different widths on the gate discharge coefficients. Water Science & Technology Water Supply, 2022, vol. 22, pp. 7767—7781. DOI: 10.2166/ws.2022.354.

23. Edel’geriev R. S. K., Romanovskaya A. A. New Approaches to the Adaptation to Climate Change: The Arctic Zone of Russia. Russ. Meteorol. Hydrol., 2020, vol. 45, pp. 305—316. Available at: https://doi.org/10.3103/S1068373920050015/.

24. Bogoyavlensky V. I., Bogoyavlensky I. V., Kargina T. N., Nikonov R. A. Digital technologies for remote detection and monitoring of the development of heaving mounds and craters of catastrophic gas blowouts in the Arctic. Arctic: Ecology and Economy, 2020, ¹ 4 (40), pp. 90—105. Available at: https://doi.org/10.25283/2223-4594-2020-4-90-105. (In Russian).

25. Tan Z., Zhuang Q. Arctic lakes are continuous methane sources to the atmosphere under warming conditions. Environmental Research Letters, 2015, vol. 10. P. 054016.

26. Zona D., Gioli B., Commane R. et. al. Cold season emissions dominate the Arctic tundra methane budget. Proceedings of the National Academy of Sciences USA, 2016, vol. 113, pp. 40—45.