МЕТОДЫ ОЦЕНКИ КАЧЕСТВА СИСТЕМ КОМПЬЮТЕРНОГО ЗРЕНИЯ: ЭВОЛЮЦИЯ ПОДХОДОВ, ТЕНДЕНЦИИ И ОГРАНИЧЕНИЯ

• 2026 № 1 (88) С. 35-41

🇷🇺 На русском

Для цитирования

Шарова Д.Е, Гарбук С.В. Методы оценки качества систем компьютерного зрения: эволюция подходов, тенденции и ограничения //Информационно-экономические аспекты стандартизации и технического регулирования. 2026. № 1(88). С. 35–41.

Аннотация

Рост применения систем компьютерного зрения на базе искусственного интеллекта сопровождается расширением спектра методов оценки их качества. При этом существующие подходы существенно различаются по охватываемым аспектам и часто не обеспечивают полноты представления о работе системы в реальных условиях. В статье представлен обзор основных направлений оценки качества: классических статистических метрик (точность, чувствительность, площадь под ROC-кривой), показателей устойчивости и обобщаемости, эксплуатационных характеристик, а также методов анализа доверия, объяснимости и взаимодействия с пользователем. Особое внимание уделено роли существенных факторов эксплуатации (СФЭ) - изменяемых условий применения систем (тип данных, оборудование, сценарии использования, структура пользовательских действий), влияющих на воспроизводимость метрик и достоверность испытаний. Показано, что даже высокие значения точности или площади под ROC-кривой не гарантируют качества в эксплуатации при изменении распределения данных, условий съемки или параметров подготовки изображений. Отмечается, что современные метрики и показатели качества охватывают точность, устойчивость к дрейфу данных, интерпретируемость результатов, доверие пользователя, скорость и предсказуемость работы системы...

Ключевые слова

ИСКУССТВЕННЫЙ ИНТЕЛЛЕКТ КОМПЬЮТЕРНОЕ ЗРЕНИЕ ОЦЕНКА КАЧЕСТВА ПОКАЗАТЕЛИ КАЧЕСТВА ИНФОРМАЦИОННЫЕ ПРОЦЕССЫ

Об авторах

Шарова Д. Е.

Шарова Д. Е. — Начальник управления, Департамент информационных технологий города Москвы, ( Москва, Россия )

Гарбук С. В.

Гарбук С. В. — Кандидат технических наук, Директор, ФГБУН ВИНИТИ РАН, ( Москва, Россия )

Список литературы

1. Powers D.M.W. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation // Journal of Machine Learning Technologies. 2011. Vol. 2, no. 1. P. 37–63.
2. Chicco D., Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation // BMC Genomics. 2020. Vol. 21, no. 1. P. 6.
3. Fawcett T. An introduction to ROC analysis // Pattern Recognition Letters. 2006. Vol. 27, no. 8. P. 861–874.
4. Garbuk S. V. Intellimetry as a Way to Ensure AI Trustworthiness // 2018 International Conference on Artificial Intelligence Applications and Innovations. IC-AIAI. 2018. P. 27–30.
5. Brodersen K.H., Ong C.S., Stephan K.E., Buhmann J.M. The balanced accuracy and its posterior distribution // In 2010 20th international conference on pattern recognition. IEEE. 2010. P. 3121–3124.
6. Chicco D., Tötsch N., Jurman G. The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation // BioData Mining. 2021. Vol. 14, no. 1. P. 13.
7. Warrens M. J. Cohen's kappa is a weighted average // Statistical Methodology. 2010. Vol. 8, no. 6. P. 473–484.
8. Grandini M., Bagli E., Visani G. Metrics for Multi-Class Classification: an Overview. 2020. arXiv: 2008.05756.
9. Recht B., Roelofs R., Schmidt L., Shankar V. Do ImageNet Classifiers Generalize to ImageNet // Proceedings of the 36th International Conference on Machine Learning. 2019. Vol. 97. P. 5389–5400.
10. Taori R., Dave A., Shankar V. et al. Measuring Robustness to Natural Distribution Shifts in Image Classification // Advances in Neural Information Processing Systems. 2020. Vol. 33. P. 18583–18599.
11. Gulrajani I., Lopez-Paz D. In Search of Lost Domain Generalization. 2020. arXiv:2007.01434v1
12. Goodfellow I.J., Shlens J., Szegedy C. Explaining and Harnessing Adversarial Examples. 2014. arXiv:1412.6572v3
13. Lu J., Liu A., Dong F. et. al. Learning under Concept Drift: A Review // Transactions on Knowledge and Data Engineering. IEEE. 2019. Vol. 31, no. 12. P. 2346–2363.
14. Amershi S., Weld D., Vorvoreanu M. et al. Guidelines for Human-AI Interaction // CHI Conference on Human Factors in Computing Systems. 2019. No. 3. P. 1–13.
15. Sendak M., D’Arcy J., Kashyap S. et al. A Path for Translation of Machine Learning Products into Healthcare Delivery // EMJ Innovations. 2020.
16. Holstein K., Wortman Vaughan J., Daumé H. et al. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? // CHI Conference on Human Factors in Computing Systems. 2019. No. 600. P. 1–16.
17. Buçinca Z., Malaya M., Glassman E. Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems // Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 2020. P. 454–464.
18. Lai V., Tan C. On Human Predictions with Explanations and Predictions without Explanations // Proceedings of the Conference on Fairness, Accountability, and Transparency. 2019. P. 29–38.
19. El Arab R.A., Al Moosa O.A. Systematic review of cost effectiveness and budget impact of artificial intelligence in healthcare // NPJ Digital Medicine. 2025. Vol. 8, no.1. P. 548.
20. Enholm I.M., Papagiannidis E., Mikalef P. et al. Artificial Intelligence and Business Value: a Literature Review // Information Systems Frontires. 2022. Vol. 24. P. 1709–1734.
21. Cabitza F, Campagner A, Balsano C. Bridging the "last mile" gap between AI implementation and operation: "data awareness" that matters // Annals of Translational Medicine. 2020. Vol. 8, no. 7. P. 501.
22. Ribeiro M., Singh S., Guestrin C. “Why Should I Trust You?” Explaining the Predictions of Any Classifier // KDD Conference Proceedings. 2016. P. 1135–1144.
23. Lundberg S., Lee S. A Unified Approach to Interpreting Model Predictions // Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017. P. 4768–4777.
24. Doshi-Velez F., Kim B. Towards A Rigorous Science of Interpretable Machine Learning. 2017. arXiv: 1702.08608.
25. Poursabzi-Sangdeh F., Goldstein D., Hofman J. et.al. Manipulating and Measuring Model Interpretability // CHI Conference on Human Factors in Computing Systems. 2018. Art. no. 237. P. 1–52.
26. Webb G.I., Hyde R., Cao H. et.al. Characterizing concept drift // Data Mining and Knowledge Discovery. 2016. Vol. 30, no. 4. P. 964–994.
27. De Lange M., Aljundi R., Masana M. et.al. A continual learning survey: Defying forgetting in classification tasks // IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021. Vol. 44, no. 7. P. 3366–3385.
28. Гарбук С.В. Качественный анализ рисков в жизненном цикле систем искусственного интеллекта // В кн. Безопасность России. Правовые, социально-экономические и научно-технические аспекты. Тематический блок «Национальная безопасность». Системная инженерия в проблемах национальной безопасности. Научн. рук. чл.-корр. РАН Н.А. Махутов. С. 643–659. М.: МГОФ «Знание». 2025. 898 с.
29. Гарбук С.В. Метод оценки влияния параметров стандартизации на эффективность создания и применения систем искусственного интеллекта // Информационно-экономические аспекты стандартизации и технического регулирования. 2022. № 3(67). С. 4–14.

🇬🇧 In English

METHODS FOR EVALUATING THE QUALITY OF COMPUTER VISION SYSTEMS: EVOLUTION OF APPROACHES, TRENDS, AND LIMITATIONS

For citation

Sharova D.E., Garbuk S.V. Methods For Evaluating The Quality Of Computer Vision Systems: Evolution Of Approaches, Trends, And Limitations. Information and Economic Aspects of Standardization and Technical Regulation. 2026; 1(88): 35–41. (In Russ.).

Abstract

The growing use of computer vision systems based on artificial intelligence has led to the emergence of diverse approaches for evaluating their quality. However, these methods differ significantly in scope and often fail to reflect the full spectrum of system behavior under real-world conditions. This article provides a structured review of the main groups of quality assessment techniques, including classical statistical metrics (accuracy, precision, recall, ROC-AUC, MCC), robustness and generalization indicators, operational performance characteristics, as well as approaches addressing explainability, user interaction, and trustworthiness.A particular focus is placed on significant operational factors - variable conditions of system use such as data source, acquisition parameters, equipment, and user workflow - that substantially affect the reproducibility of metrics and the reliability of testing. The analysis shows that high accuracy or AUC scores do not guarantee stable performance when data distributions shift or when operating conditions deviate from those used during model development...

Keywords

COMPUTER VISION QUALITY ASSESSMENT SIGNIFICANT OPERATIONAL FACTORS METRICS QUALITY INDICATORS

About the authors

Sharova D. E.

Sharova D. E. — Head of Division, Department of Information Technologies of the City of Moscow, ( Moscow, Russia )

Garbuk S. V.

Garbuk S. V. — Ph.D. (Engineering), Director, All-Russian Institute of Scientific and Technical Information of the Russian Academy of Sciences (VINITI RAS), ( Moscow, Russia )

References

1. Powers D.M.W. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation // Journal of Machine Learning Technologies. 2011. Vol. 2, no. 1. P. 37–63.
2. Chicco D., Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation // BMC Genomics. 2020. Vol. 21, no. 1. P. 6.
3. Fawcett T. An introduction to ROC analysis // Pattern Recognition Letters. 2006. Vol. 27, no. 8. P. 861–874.
4. Garbuk S. V. Intellimetry as a Way to Ensure AI Trustworthiness // 2018 International Conference on Artificial Intelligence Applications and Innovations. IC-AIAI. 2018. P. 27–30.
5. Brodersen K.H., Ong C.S., Stephan K.E., Buhmann J.M. The balanced accuracy and its posterior distribution // In 2010 20th international conference on pattern recognition. IEEE. 2010. P. 3121–3124.
6. Chicco D., Tötsch N., Jurman G. The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation // BioData Mining. 2021. Vol. 14, no. 1. P. 13.
7. Warrens M. J. Cohen's kappa is a weighted average // Statistical Methodology. 2010. Vol. 8, no. 6. P. 473–484.
8. Grandini M., Bagli E., Visani G. Metrics for Multi-Class Classification: an Overview. 2020. arXiv: 2008.05756.
9. Recht B., Roelofs R., Schmidt L., Shankar V. Do ImageNet Classifiers Generalize to ImageNet // Proceedings of the 36th International Conference on Machine Learning. 2019. Vol. 97. P. 5389–5400.
10. Taori R., Dave A., Shankar V. et al. Measuring Robustness to Natural Distribution Shifts in Image Classification // Advances in Neural Information Processing Systems. 2020. Vol. 33. P. 18583–18599.
11. Gulrajani I., Lopez-Paz D. In Search of Lost Domain Generalization. 2020. arXiv:2007.01434v1
12. Goodfellow I.J., Shlens J., Szegedy C. Explaining and Harnessing Adversarial Examples. 2014. arXiv:1412.6572v3
13. Lu J., Liu A., Dong F. et. al. Learning under Concept Drift: A Review // Transactions on Knowledge and Data Engineering. IEEE. 2019. Vol. 31, no. 12. P. 2346–2363.
14. Amershi S., Weld D., Vorvoreanu M. et al. Guidelines for Human-AI Interaction // CHI Conference on Human Factors in Computing Systems. 2019. No. 3. P. 1–13.
15. Sendak M., D’Arcy J., Kashyap S. et al. A Path for Translation of Machine Learning Products into Healthcare Delivery // EMJ Innovations. 2020.
16. Holstein K., Wortman Vaughan J., Daumé H. et al. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? // CHI Conference on Human Factors in Computing Systems. 2019. No. 600. P. 1–16.
17. Buçinca Z., Malaya M., Glassman E. Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems // Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 2020. P. 454–464.
18. Lai V., Tan C. On Human Predictions with Explanations and Predictions without Explanations // Proceedings of the Conference on Fairness, Accountability, and Transparency. 2019. P. 29–38.
19. El Arab R.A., Al Moosa O.A. Systematic review of cost effectiveness and budget impact of artificial intelligence in healthcare // NPJ Digital Medicine. 2025. Vol. 8, no.1. P. 548.
20. Enholm I.M., Papagiannidis E., Mikalef P. et al. Artificial Intelligence and Business Value: a Literature Review // Information Systems Frontires. 2022. Vol. 24. P. 1709–1734.
21. Cabitza F, Campagner A, Balsano C. Bridging the "last mile" gap between AI implementation and operation: "data awareness" that matters // Annals of Translational Medicine. 2020. Vol. 8, no. 7. P. 501.
22. Ribeiro M., Singh S., Guestrin C. “Why Should I Trust You?” Explaining the Predictions of Any Classifier // KDD Conference Proceedings. 2016. P. 1135–1144.
23. Lundberg S., Lee S. A Unified Approach to Interpreting Model Predictions // Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017. P. 4768–4777.
24. Doshi-Velez F., Kim B. Towards A Rigorous Science of Interpretable Machine Learning. 2017. arXiv: 1702.08608.
25. Poursabzi-Sangdeh F., Goldstein D., Hofman J. et.al. Manipulating and Measuring Model Interpretability // CHI Conference on Human Factors in Computing Systems. 2018. Art. no. 237. P. 1–52.
26. Webb G.I., Hyde R., Cao H. et.al. Characterizing concept drift // Data Mining and Knowledge Discovery. 2016. Vol. 30, no. 4. P. 964–994.
27. De Lange M., Aljundi R., Masana M. et.al. A continual learning survey: Defying forgetting in classification tasks // IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021. Vol. 44, no. 7. P. 3366–3385.
28. Garbuk S.V. Qualitative risk analysis in the life cycle of artificial intelligence systems. In: Security of Russia. Legal, socio-economic and scientific-technical aspects. Thematic section “National Security”. Systems engineering in national security problems. Scientific editor: N.A. Makhutov. Moscow: MGOF “Znanie”. 2025. PP. 643–659. 898 p. (in Russian)
29. Garbuk S.V. A method for assessing the impact of standardization parameters on the effectiveness of creating and applying artificial intelligence systems // Information and economic aspects of standardization and technical regulation. 2022. Vol. 3, no. 67. P. 4–14.