Interactive Point-and-Click Segmentation for Object Removal in Digital Images

  • Frank Nielsen
  • Richard Nock
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3766)


In this paper, we explore the problem of deleting objects in still pictures. We present an interactive system based on a novel intuitive user-friendly interface for removing undesirable objects in digital pictures. To erase an object in an image, a user indicates which object is to be removed by simply pinpointing it with the mouse cursor. As the mouse cursor rolls over the image, the current implicit selected object’s border is highlighted, providing a visual feedback. In case the computer-segmented area does not match the users’ perception of the object, users can further provide a few inside/outside object cues by clicking on a small number of object or nonobject pixels. Experimentally, a small number of such cues is generally enough to reach a correct matching, even for complex textured images. Afterwards, the user removes the object by clicking the left mouse button, and a hole-filling technique is initiated to generate a seamless background portion. Our image manipulation system consists of two components: (i) fully automatic or partially user-steered image segmentation based on an improved fast statistical region-growing segmentation, and (ii) texture synthesis or image inpainting of irregular shaped hole regions. Experiments on a variety of photographs display the ability of the system to handle complex scenes with highly textured objects.


Object Boundary Texture Synthesis Image Inpainting Mouse Cursor Alpha Matte 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barrett, W.A., Cheney, A.S.: Object-based image editing. In: ACM SIGGRAPH, pp. 777–784 (2002)Google Scholar
  2. 2.
    Criminisi, P.: Perez, and K. Toyama. Region filling and object removal by exemplar-based inpainting. IEEE Transactions on Image Processing 13(9), 1200–1212 (2004)CrossRefGoogle Scholar
  3. 3.
    Falcão, X., Lotufo, R.A., Araújo, G.: The image foresting transformation. Technical Report #IC-00-12 (2000)Google Scholar
  4. 4.
    Gleicher, M.: Image snapping. In: ACM SIGGRAPH, pp. 183–190 (1995)Google Scholar
  5. 5.
    Li, Y., Sun, J., Tang, C.-K., Shum, H.-Y.: Lazy snapping. ACM Trans. Graph. 23(3), 303–308 (2004)CrossRefGoogle Scholar
  6. 6.
    Lin, S., Zhang, Q., Shi, J.: Alpha estimation in perceptual color space. In: Proc. IEEE ICASSP (2005)Google Scholar
  7. 7.
    Mortensen, E.N., Barrett, W.A.: Intelligent scissors for image composition. In: ACM SIGGRAPH, pp. 191–198 (1995)Google Scholar
  8. 8.
    Mortensen, E.N., Barrett, W.A.: Toboggan-based intelligent scissors with a four-parameter edge model. In: IEEE CVPR, pp. 2452–2458 (1999)Google Scholar
  9. 9.
    Nielsen, F.: Visual Computing: Geometry, Graphics, and Vision. Charles River Media, Hingham (2005) ISBN: 1-58450-427-7Google Scholar
  10. 10.
    Nock, R., Nielsen, F.: Semi-supervised statistical region refinement for color image segmentation. Pattern Recognition 38(6), 835–846 (2005)zbMATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    Pérez, P., Blake, A., Gangnet, M.: Jetstream: Probabilistic contour extraction with particles. In: IEEE ICCV, pp. 524–531 (2001)Google Scholar
  12. 12.
    Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. 22(3), 313–318 (2003)CrossRefGoogle Scholar
  13. 13.
    Rosenfeld, A.: Picture processing by computer. Academic Press, London (1969)zbMATHGoogle Scholar
  14. 14.
    Rother, C., Kolmogorov, V., Blake, A.: Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)CrossRefGoogle Scholar
  15. 15.
    Sun, J., Jia, J., Tang, C.-K., Shum, H.-Y.: Poisson matting. ACM Trans. Graph. 23(3), 315–321 (2004)CrossRefGoogle Scholar
  16. 16.
    Tan, K.-H., Ahuja, N.: Selecting objects with freehand sketches. In: IEEE ICCV, pp. 337–344 (2001)Google Scholar
  17. 17.
    Tarjan, R.E.: Efficiency of a good but not linear set union algorithm. J. ACM 22 (2), 215–225 (1975)zbMATHCrossRefMathSciNetGoogle Scholar
  18. 18.
    Telea, A.: An image inpainting technique based on the fast marching method. Journal of Graphics Tools 9(1), 23–34 (2004)Google Scholar
  19. 19.
    Wei, L.-Y., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: ACM SIGGRAPH, pp. 479–488 (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Frank Nielsen
    • 1
  • Richard Nock
    • 2
  1. 1.Sony Computer Science Laboratories, Inc. 
  2. 2.Université Antilles-Guyane 

Personalised recommendations