rst Информационно-экономические аспекты стандартизации и технического регулирования Informatsionno-ekonomicheskiye aspekty standartizatsii i tekhnicheskogo regulirovaniya 2311-1348 ФГБУ «Институт стандартизации» KXRTNC 89068821 Research Article Информационные системы и процессы information systems and processes МЕТОДЫ ОЦЕНКИ КАЧЕСТВА СИСТЕМ КОМПЬЮТЕРНОГО ЗРЕНИЯ: ЭВОЛЮЦИЯ ПОДХОДОВ, ТЕНДЕНЦИИ И ОГРАНИЧЕНИЯ METHODS FOR EVALUATING THE QUALITY OF COMPUTER VISION SYSTEMS: EVOLUTION OF APPROACHES, TRENDS, AND LIMITATIONS Шарова Д. Е. Sharova D. E.

Шарова Д. Е., Начальник управления Департамент информационных технологий города Москвы

Москва, Россия

Sharova D. E., Head of Division Department of Information Technologies of the City of Moscow

Moscow, Russia

 Гарбук С. В. Garbuk S. V.

Гарбук С. В., Директор ФГБУН ВИНИТИ РАН

Москва, Россия

Garbuk S. V., Director All-Russian Institute of Scientific and Technical Information of the Russian Academy of Sciences (VINITI RAS)

Moscow, Russia

 Департамент информационных технологий города Москвы Россия Department of Information Technologies of the City of Moscow Russian Federation 2026 96 88 35 41 Copyright © Шарова Д. Е., Гарбук С. В., 2026 2026 Шарова Д. Е., Гарбук С. В. Sharova D. E., Garbuk S. V. Данная работа распространяется под лицензией Creative Commons Attribution 4.0. https://iea.gostinfo.ru/article/view/21

Рост применения систем компьютерного зрения на базе искусственного интеллекта сопровождается расширением спектра методов оценки их качества. При этом существующие подходы существенно различаются по охватываемым аспектам и часто не обеспечивают полноты представления о работе системы в реальных условиях. В статье представлен обзор основных направлений оценки качества: классических статистических метрик (точность, чувствительность, площадь под ROC-кривой), показателей устойчивости и обобщаемости, эксплуатационных характеристик, а также методов анализа доверия, объяснимости и взаимодействия с пользователем. Особое внимание уделено роли существенных факторов эксплуатации (СФЭ) - изменяемых условий применения систем (тип данных, оборудование, сценарии использования, структура пользовательских действий), влияющих на воспроизводимость метрик и достоверность испытаний. Показано, что даже высокие значения точности или площади под ROC-кривой не гарантируют качества в эксплуатации при изменении распределения данных, условий съемки или параметров подготовки изображений. Отмечается, что современные метрики и показатели качества охватывают точность, устойчивость к дрейфу данных, интерпретируемость результатов, доверие пользователя, скорость и предсказуемость работы системы...

The growing use of computer vision systems based on artificial intelligence has led to the emergence of diverse approaches for evaluating their quality. However, these methods differ significantly in scope and often fail to reflect the full spectrum of system behavior under real-world conditions. This article provides a structured review of the main groups of quality assessment techniques, including classical statistical metrics (accuracy, precision, recall, ROC-AUC, MCC), robustness and generalization indicators, operational performance characteristics, as well as approaches addressing explainability, user interaction, and trustworthiness.A particular focus is placed on significant operational factors - variable conditions of system use such as data source, acquisition parameters, equipment, and user workflow - that substantially affect the reproducibility of metrics and the reliability of testing. The analysis shows that high accuracy or AUC scores do not guarantee stable performance when data distributions shift or when operating conditions deviate from those used during model development...

 ИСКУССТВЕННЫЙ ИНТЕЛЛЕКТ КОМПЬЮТЕРНОЕ ЗРЕНИЕ ОЦЕНКА КАЧЕСТВА ПОКАЗАТЕЛИ КАЧЕСТВА ИНФОРМАЦИОННЫЕ ПРОЦЕССЫ COMPUTER VISION QUALITY ASSESSMENT SIGNIFICANT OPERATIONAL FACTORS METRICS QUALITY INDICATORS References Powers D.M.W. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation // Journal of Machine Learning Technologies. 2011. Vol. 2, no. 1. P. 37–63. Powers D.M.W. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation // Journal of Machine Learning Technologies. 2011. Vol. 2, no. 1. P. 37–63. Chicco D., Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation // BMC Genomics. 2020. Vol. 21, no. 1. P. 6. Chicco D., Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation // BMC Genomics. 2020. Vol. 21, no. 1. P. 6. Fawcett T. An introduction to ROC analysis // Pattern Recognition Letters. 2006. Vol. 27, no. 8. P. 861–874. Fawcett T. An introduction to ROC analysis // Pattern Recognition Letters. 2006. Vol. 27, no. 8. P. 861–874. Garbuk S. V. Intellimetry as a Way to Ensure AI Trustworthiness // 2018 International Conference on Artificial Intelligence Applications and Innovations. IC-AIAI. 2018. P. 27–30. Garbuk S. V. Intellimetry as a Way to Ensure AI Trustworthiness // 2018 International Conference on Artificial Intelligence Applications and Innovations. IC-AIAI. 2018. P. 27–30. Brodersen K.H., Ong C.S., Stephan K.E., Buhmann J.M. The balanced accuracy and its posterior distribution // In 2010 20th international conference on pattern recognition. IEEE. 2010. P. 3121–3124. Brodersen K.H., Ong C.S., Stephan K.E., Buhmann J.M. The balanced accuracy and its posterior distribution // In 2010 20th international conference on pattern recognition. IEEE. 2010. P. 3121–3124. Chicco D., Tötsch N., Jurman G. The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation // BioData Mining. 2021. Vol. 14, no. 1. P. 13. Chicco D., Tötsch N., Jurman G. The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation // BioData Mining. 2021. Vol. 14, no. 1. P. 13. Warrens M. J. Cohen's kappa is a weighted average // Statistical Methodology. 2010. Vol. 8, no. 6. P. 473–484. Warrens M. J. Cohen's kappa is a weighted average // Statistical Methodology. 2010. Vol. 8, no. 6. P. 473–484. Grandini M., Bagli E., Visani G. Metrics for Multi-Class Classification: an Overview. 2020. arXiv: 2008.05756. Grandini M., Bagli E., Visani G. Metrics for Multi-Class Classification: an Overview. 2020. arXiv: 2008.05756. Recht B., Roelofs R., Schmidt L., Shankar V. Do ImageNet Classifiers Generalize to ImageNet // Proceedings of the 36th International Conference on Machine Learning. 2019. Vol. 97. P. 5389–5400. Recht B., Roelofs R., Schmidt L., Shankar V. Do ImageNet Classifiers Generalize to ImageNet // Proceedings of the 36th International Conference on Machine Learning. 2019. Vol. 97. P. 5389–5400. Taori R., Dave A., Shankar V. et al. Measuring Robustness to Natural Distribution Shifts in Image Classification // Advances in Neural Information Processing Systems. 2020. Vol. 33. P. 18583–18599. Taori R., Dave A., Shankar V. et al. Measuring Robustness to Natural Distribution Shifts in Image Classification // Advances in Neural Information Processing Systems. 2020. Vol. 33. P. 18583–18599. Gulrajani I., Lopez-Paz D. In Search of Lost Domain Generalization. 2020. arXiv:2007.01434v1 Gulrajani I., Lopez-Paz D. In Search of Lost Domain Generalization. 2020. arXiv:2007.01434v1 Goodfellow I.J., Shlens J., Szegedy C. Explaining and Harnessing Adversarial Examples. 2014. arXiv:1412.6572v3 Goodfellow I.J., Shlens J., Szegedy C. Explaining and Harnessing Adversarial Examples. 2014. arXiv:1412.6572v3 Lu J., Liu A., Dong F. et. al. Learning under Concept Drift: A Review // Transactions on Knowledge and Data Engineering. IEEE. 2019. Vol. 31, no. 12. P. 2346–2363. Lu J., Liu A., Dong F. et. al. Learning under Concept Drift: A Review // Transactions on Knowledge and Data Engineering. IEEE. 2019. Vol. 31, no. 12. P. 2346–2363. Amershi S., Weld D., Vorvoreanu M. et al. Guidelines for Human-AI Interaction // CHI Conference on Human Factors in Computing Systems. 2019. No. 3. P. 1–13. Amershi S., Weld D., Vorvoreanu M. et al. Guidelines for Human-AI Interaction // CHI Conference on Human Factors in Computing Systems. 2019. No. 3. P. 1–13. Sendak M., D’Arcy J., Kashyap S. et al. A Path for Translation of Machine Learning Products into Healthcare Delivery // EMJ Innovations. 2020. Sendak M., D’Arcy J., Kashyap S. et al. A Path for Translation of Machine Learning Products into Healthcare Delivery // EMJ Innovations. 2020. Holstein K., Wortman Vaughan J., Daumé H. et al. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? // CHI Conference on Human Factors in Computing Systems. 2019. No. 600. P. 1–16. Holstein K., Wortman Vaughan J., Daumé H. et al. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? // CHI Conference on Human Factors in Computing Systems. 2019. No. 600. P. 1–16. Buçinca Z., Malaya M., Glassman E. Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems // Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 2020. P. 454–464. Buçinca Z., Malaya M., Glassman E. Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems // Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 2020. P. 454–464. Lai V., Tan C. On Human Predictions with Explanations and Predictions without Explanations // Proceedings of the Conference on Fairness, Accountability, and Transparency. 2019. P. 29–38. Lai V., Tan C. On Human Predictions with Explanations and Predictions without Explanations // Proceedings of the Conference on Fairness, Accountability, and Transparency. 2019. P. 29–38. El Arab R.A., Al Moosa O.A. Systematic review of cost effectiveness and budget impact of artificial intelligence in healthcare // NPJ Digital Medicine. 2025. Vol. 8, no.1. P. 548. El Arab R.A., Al Moosa O.A. Systematic review of cost effectiveness and budget impact of artificial intelligence in healthcare // NPJ Digital Medicine. 2025. Vol. 8, no.1. P. 548. Enholm I.M., Papagiannidis E., Mikalef P. et al. Artificial Intelligence and Business Value: a Literature Review // Information Systems Frontires. 2022. Vol. 24. P. 1709–1734. Enholm I.M., Papagiannidis E., Mikalef P. et al. Artificial Intelligence and Business Value: a Literature Review // Information Systems Frontires. 2022. Vol. 24. P. 1709–1734. Cabitza F, Campagner A, Balsano C. Bridging the "last mile" gap between AI implementation and operation: "data awareness" that matters // Annals of Translational Medicine. 2020. Vol. 8, no. 7. P. 501. Cabitza F, Campagner A, Balsano C. Bridging the "last mile" gap between AI implementation and operation: "data awareness" that matters // Annals of Translational Medicine. 2020. Vol. 8, no. 7. P. 501. Ribeiro M., Singh S., Guestrin C. “Why Should I Trust You?” Explaining the Predictions of Any Classifier // KDD Conference Proceedings. 2016. P. 1135–1144. Ribeiro M., Singh S., Guestrin C. “Why Should I Trust You?” Explaining the Predictions of Any Classifier // KDD Conference Proceedings. 2016. P. 1135–1144. Lundberg S., Lee S. A Unified Approach to Interpreting Model Predictions // Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017. P. 4768–4777. Lundberg S., Lee S. A Unified Approach to Interpreting Model Predictions // Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017. P. 4768–4777. Doshi-Velez F., Kim B. Towards A Rigorous Science of Interpretable Machine Learning. 2017. arXiv: 1702.08608. Doshi-Velez F., Kim B. Towards A Rigorous Science of Interpretable Machine Learning. 2017. arXiv: 1702.08608. Poursabzi-Sangdeh F., Goldstein D., Hofman J. et.al. Manipulating and Measuring Model Interpretability // CHI Conference on Human Factors in Computing Systems. 2018. Art. no. 237. P. 1–52. Poursabzi-Sangdeh F., Goldstein D., Hofman J. et.al. Manipulating and Measuring Model Interpretability // CHI Conference on Human Factors in Computing Systems. 2018. Art. no. 237. P. 1–52. Webb G.I., Hyde R., Cao H. et.al. Characterizing concept drift // Data Mining and Knowledge Discovery. 2016. Vol. 30, no. 4. P. 964–994. Webb G.I., Hyde R., Cao H. et.al. Characterizing concept drift // Data Mining and Knowledge Discovery. 2016. Vol. 30, no. 4. P. 964–994. De Lange M., Aljundi R., Masana M. et.al. A continual learning survey: Defying forgetting in classification tasks // IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021. Vol. 44, no. 7. P. 3366–3385. De Lange M., Aljundi R., Masana M. et.al. A continual learning survey: Defying forgetting in classification tasks // IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021. Vol. 44, no. 7. P. 3366–3385. Гарбук С.В. Качественный анализ рисков в жизненном цикле систем искусственного интеллекта // В кн. Безопасность России. Правовые, социально-экономические и научно-технические аспекты. Тематический блок «Национальная безопасность». Системная инженерия в проблемах национальной безопасности. Научн. рук. чл.-корр. РАН Н.А. Махутов. С. 643–659. М.: МГОФ «Знание». 2025. 898 с. Garbuk S.V. Qualitative risk analysis in the life cycle of artificial intelligence systems. In: Security of Russia. Legal, socio-economic and scientific-technical aspects. Thematic section “National Security”. Systems engineering in national security problems. Scientific editor: N.A. Makhutov. Moscow: MGOF “Znanie”. 2025. PP. 643–659. 898 p. (in Russian) Гарбук С.В. Метод оценки влияния параметров стандартизации на эффективность создания и применения систем искусственного интеллекта // Информационно-экономические аспекты стандартизации и технического регулирования. 2022. № 3(67). С. 4–14. Garbuk S.V. A method for assessing the impact of standardization parameters on the effectiveness of creating and applying artificial intelligence systems // Information and economic aspects of standardization and technical regulation. 2022. Vol. 3, no. 67. P. 4–14.

Конфликт интересов. Авторы заявляют об отсутствии конфликта интересов.

The authors declare that there are no conflicts of interest present.