Detection of human body parts on the image using the neural networks and the attention model

Authors

  • Viktoria V. Sorokina Belarusian State University, 4 Niezaliežnasci Avenue, Minsk 220030, Belarus
  • Sergey V. Ablameyko Belarusian State University, 4 Niezaliežnasci Avenue, Minsk 220030, Belarus; United Institute of Informatics Problems, National Academy of Sciences of Belarus, 6 Surhanava Street, Minsk 220012, Belarus

Keywords:

human body parts detection, attention model, convolutional neural network, Smart Cropping

Abstract

Human body parts detection is a challenging task, which has a lot of applications. In this paper, we propose an algorithm to detect human body parts on images using the OpenPose neural network and the attention model. The novelty of the proposed algorithm is that it is based on a convolutional neural network that uses non-parametric representation to associate the body parts with people in an image in combination with the attention model that learns to focus on specific regions of the input image. The algorithm is part of the Smart Cropping system developed by the authors with the aim to cut necessary pieces of clothing in images and prepare e-commerce catalogues.

Author Biographies

  • Viktoria V. Sorokina, Belarusian State University, 4 Niezaliežnasci Avenue, Minsk 220030, Belarus

    postgraduate student at the department of web-technologies and computer simulation, faculty of mechanics and mathematics

  • Sergey V. Ablameyko, Belarusian State University, 4 Niezaliežnasci Avenue, Minsk 220030, Belarus; United Institute of Informatics Problems, National Academy of Sciences of Belarus, 6 Surhanava Street, Minsk 220012, Belarus

    academician of the National Academy of Sciences of Belarus, doctor of science (engineering), full professor; professor at the department of web-technologies and computer simulation, faculty of mechanics and mathematics, Belarusian State University; and chief researcher at the department of intelligent information systems, United Institute of Informatics Problems, National Academy of Sciences of Belarus

References

  1. Yucheng Chen, Yingli Tian, Mingyi He. Monocular human pose estimation: a survey of deep learning-based methods. Computer Vision and Image Understanding. 2020;192:102897. DOI: 10.1016/j.cviu.2019.102897.
  2. Rolley-Parnell E-J, Kanoulas D, Laurenzi A, Delhaisse B, Rozo L, Caldwell DG, et al. Bi-manual articulated robot teleoperation using an external RGB-D range sensor. In: 15th International conference on control, automation, robotics and vision; 2018 November 18–21; Singapore. [S. l.]: Institute of Electrical and Electronics Engineers; 2018. p. 298–304. DOI: 10.1109/ICARCV.2018.8581174.
  3. Murdock H. The ultimate eCommerce product image guide for 2021 [Internet]. [S. l.]: Threekit Inc.; 2020 January 30 [cited 2021 March 25]. Available from: https://www.threekit.com/blog/ecommerce-product-image-guide-2020.
  4. Ablameyko SV, Krasnoproshin VV, Obraztsov VA. [Models and technologies of pattern recognition with application in data mining]. Vestnik BGU. Seriya 1. Fizika. Matematika. Informatika. 2011;3:62–72. Russian.
  5. Zhao Liu, Jianke Zhu, Jiajun Bu, Chun Chen. A survey of human pose estimation: the body parts parsing based methods. Journal of Visual Communication and Image Representation. 2015;32:10–19. DOI: 10.1016/j.jvcir.2015.06.013.
  6. Luvizon DC, Picard D, Tabia H. 2D/3D pose estimation and action recognition using multitask deep learning. In: 2018 IEEE /CVF conference on computer vision and pattern recognition; 2018 June 18–22; Salt Lake City, USA. Los Alamitos: Conference Publishing Services, IEEE Computer Society; 2018. p. 5137–5146. DOI: 10.1109/CVPR.2018.00539.
  7. Insafutdinov E. DeeperCut: a deeper, stronger, and faster multi-person pose estimation model. In: Leibe B, Matas J, Sebe N, Welling M, editors. Computer vision – ECCV 2016. 14th European conference; 2016 October 11–14; Amsterdam, The Netherlands. Part 6. Cham: Springer; 2016. p. 34–50 (Lecture notes in computer science; volume 9910). DOI: 10.1007/978-3-319-46466-4_3.
  8. Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, Cewu Lu. RMPE: regional multi-person pose estimation. In: 2017 IEEE International conference on computer vision (ICCV); 2017 October 22–29; Venice, Italy. [S. l.]: Institute of Electrical and Electronics Engineers; 2017. p. 2353–2362. DOI: 10.1109/ICCV.2017.256.
  9. Xiao Chu, Wei Yang, Wanli Ouyang, Cheng Ma, Yuille AL, Xiaogang Wang. Multi-context attention for human pose estimation. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR); 2017 July 21–26; Honolulu, USA. [S. l.]: Institute of Electrical and Electronics Engineers; 2017. p. 5669–5678. DOI: 10.1109/CVPR.2017.601.
  10. He K, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. In: 2017 IEEE International conference on computer vision (ICCV); 2017 October 22–29; Venice, Italy. [S. l.]: Institute of Electrical and Electronics Engineers; 2017. p. 2980–2988. DOI: 10.1109/ICCV.2017.322.
  11. Toshev A, Szegedy C. DeepPose: human pose estimation via Deep Neural Networks. In: 2014 IEEE conference on computer vision and pattern recognition; 2014 June 23–28; Columbus, USA. [S. l.]: Institute of Electrical and Electronics Engineers; 2017. p. 1653–1660. DOI: 10.1109/CVPR.2014.214.
  12. Tompson J, Jain A, LeCun Y, Bregler C. Join training of a convolutional network and a graphical model for human pose estimation. In: 28th annual conference on Neural Information Processing Systems; 2014 December 8–13; Montreal, Canada. Red Hook: Curran Associates Inc.; 2015. p. 1799–1807 (Advances in Neural Information Processing Systems; volume 27).
  13. Tompson J, Goroshin R, Jain A, LeCun Y, Bregler C. Efficient object localization using convolutional networks. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR); 2015 June 7–12; Boston, USA. [S. l.]: Institute of Electrical and Electronics Engineers; 2015. p. 648–656. DOI: 10.1109/CVPR.2015.7298664.
  14. Yang Y, Ramanan D. Articulated human detection with flexible mixtures of parts. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2013;35(12):2878–2890. DOI: 10.1109/TPAMI.2012.261.
  15. Cao Z, Simon T, Wei S, Sheikh Y. Realtime multi-person 2D pose estimation using part affinity fields. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR); 2017 July 21–26; Honolulu, USA. [S. l.]: Institute of Electrical and Electronics Engineers; 2017. p. 1302–1310. DOI: 10.1109/CVPR.2017.143.
  16. Bahdanau D, Cho KH, Bengio Y. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473v7 [Preprint]. 2016 [cited 2019 April 5]: [15 p.]. Available from: https://arxiv.org/abs/1409.0473v7.
  17. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556v6 [Preprint]. 2015 [cited 2019 April 5]: [14 p.]. Available from: https://arxiv.org/abs/1409.1556v6.
  18. Wang F, Tax DMJ. Survey on the attention based RNN model and its applications in computer vision. arXiv:1601.06823v1 [Preprint]. 2016 [cited 2019 April 5]: [42 p.]. Available from: https://arxiv.org/abs/1601.06823v1.
  19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR); 2016 June 26 – Jule 1; Las Vegas, USA. Los Alamitos: Conference Publishing Services, IEEE Computer Society; 2016. p. 770–778. DOI: 10.1109/CVPR.2016.90.
  20. Sim T, Baker S, Bsat M. The CMU Pose, Illumination, and Expression (PIE) database. In: Proceedings of Fifth IEEE International conference on automatic face gesture recognition; 2002 May 21–22; Washington, USA. [S. l.]: Institute of Electrical and Electronics Engineers; 2002. p. 53–58. DOI: 10.1109/AFGR.2002.1004130.

Downloads

Additional Files

Published

2022-08-03

Issue

No. 2 (2022): Journal of the Belarusian State University. Mathematics and Informatics

Section

Theoretical Foundations of Computer Science

License

The authors who are published in this journal agree to the following:

  1. The authors retain copyright on the work and provide the journal with the right of first publication of the work on condition of license Creative Commons Attribution-NonCommercial. 4.0 International (CC BY-NC 4.0).
  2. The authors retain the right to enter into certain contractual agreements relating to the non-exclusive distribution of the published version of the work (e.g. post it on the institutional repository, publication in the book), with the reference to its original publication in this journal.
  3. The authors have the right to post their work on the Internet (e.g. on the institutional store or personal website) prior to and during the review process, conducted by the journal, as this may lead to a productive discussion and a large number of references to this work. (See The Effect of Open Access.)

How to Cite

[1]
Sorokina, V.V. and Ablameyko, S.V. 2022. Detection of human body parts on the image using the neural networks and the attention model. Journal of the Belarusian State University. Mathematics and Informatics. 2 (Aug. 2022), 94–106. DOI:https://doi.org/10.33581/2520-6508-2022-2-94-106.