Resolving Hand over Face Occlusion

  • Paul Smith
  • Niels da Vitoria Lobo
  • Mubarak Shah
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3766)


This paper presents a method to segment the hand over complex backgrounds, such as the face. The similar colors and texture of the hand and face make the problem particularly challenging. Our method is based on the concept of an image force field. In this representation each individual image location consists of a vector value which is a nonlinear combination of the remaining pixels in the image. We introduce and develop a novel physics based feature that is able to measure regional structure in the image thus avoiding the problem of local pixel based analysis, which break down under our conditions. The regional image structure changes in the occluded region during occlusion. Elsewhere the regional structure remains relatively constant. We model the regional image structure at all image locations over time using a Mixture of Gaussians (MoG) to detect the occluded region in the image. We have tested the method on a number of sequences demonstrating the versatility of the proposed approach.


Regional Structure Gesture Recognition Complex Background Hand Shape Hand Gesture Recognition 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ahmad, T., Taylor, C., Lanitis, A., Cootes, T.: Tracking and recognising hand gestures, using statistical shape models. Image and Vision Computing (1997)Google Scholar
  2. 2.
    Bretzner, L., Laptev, I., Lindeberg, T.: Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering. Automatic Face and Gesture Recognition (2002)Google Scholar
  3. 3.
    Brèthes, L., Menezes, P., Lerasle, F., Hayet, J.: Face tracking and hand gesture recognition for human-robot interaction. In: International Conference on Robotics and Automation (2004)Google Scholar
  4. 4.
    Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis.TPAMI (2002)Google Scholar
  5. 5.
    Cui, Y., Weng, J.: Learning-based hand sign recognition. Automatic Face and Gesture Recognition (1995)Google Scholar
  6. 6.
    Cui, Y., Weng, J.: Hand sign recognition from intensity image sequences with complex backgrounds. Automatic Face and Gesture Recognition (1996)Google Scholar
  7. 7.
    Davis, J., Shah, M.: Recognizing hand gestures. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 800, Springer, Heidelberg (1994)Google Scholar
  8. 8.
    Fei, H., Reid, I.D.: Joint bayes filter: A hybrid tracker for non-rigid hand motion recognition. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3023, pp. 497–508. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  9. 9.
    Hamada, Y., Shimada, N., Shirai, Y.: Hand shape estimation under complex backgrounds for sign language recognition. Automatic Face and Gesture Recognition (2004)Google Scholar
  10. 10.
    Hurley, D.J., Nixon, M.S., Carter, J.N.: Force field energy functionals for image feature extraction. IVC (2002)Google Scholar
  11. 11.
    Jeong, M.H., Kuno, Y., Shimada, N., Shirai, Y.: Recognition of shape-changing hand gestures. IEICE Transactions Division D E85-D (10), 1678–1687 (2002)Google Scholar
  12. 12.
    Sherrah, J., Gong, S.: Resolving visual uncertainty and occlusion through probabilistic reasoning. In: BMVC (2000)Google Scholar
  13. 13.
    Stauffer, C., Grimson, E.: Learning patterns of activity using real-time tracking. PAMI (2000)Google Scholar
  14. 14.
    Stenger, B., Thayananthan, A., Torr, P., Cipolla, R.: Hand pose estimation using hierarchical detection. In: Intl. Workshop on Human-Computer Interaction (2004)Google Scholar
  15. 15.
    Triesch, J., von der Malsburg, C.: A system for person-independent hand posture recognition against complex backgrounds. TPAMI (2001)Google Scholar
  16. 16.
    Triesch, J., von der Malsburg, C.: Classification of hand postures against complex backgrounds using elastic graph matching. Image and Vision Computing (2002)Google Scholar
  17. 17.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: CVPR (2001)Google Scholar
  18. 18.
    Zhou, H., Huang, T.S.: Tracking articulated hand motion with eigen dynamics analysis. In: ICCV (2003)Google Scholar
  19. 19.
    Zhu, X., Yang, J., Waibel, A.: Segmenting hands of arbitrary color. Automatic Face and Gesture Recognition (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Paul Smith
    • 1
  • Niels da Vitoria Lobo
    • 1
  • Mubarak Shah
    • 1
  1. 1.Computer Vision Lab, School of Computer ScienceUniversity of Central Florida 

Personalised recommendations