Multiple-Gaze Geometry: Inferring Novel 3D Locations from Gazes Observed in Monocular Video

Ernesto Brau, Jinyan Guan, Tanya Jeffries, Kobus Barnard

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations


We develop using person gaze direction for scene understanding. In particular, we use intersecting gazes to learn 3D locations that people tend to look at, which is analogous to having multiple camera views. The 3D locations that we discover need not be visible to the camera. Conversely, knowing 3D locations of scene elements that draw visual attention, such as other people in the scene, can help infer gaze direction. We provide a Bayesian generative model for the temporal scene that captures the joint probability of camera parameters, locations of people, their gaze, what they are looking at, and locations of visual attention. Both the number of people in the scene and the number of extra objects that draw attention are unknown and need to be inferred. To execute this joint inference we use a probabilistic data association approach that enables principled comparison of model hypotheses. We use MCMC for inference over the discrete correspondence variables, and approximate the marginalization over continuous parameters using the Metropolis-Laplace approximation, using Hamiltonian (Hybrid) Monte Carlo for maximization. As existing data sets do not provide the 3D locations of what people are looking at, we contribute a small data set that does. On this data set, we infer what people are looking at with 59% precision compared with 13% for a baseline approach, and where those objects are within about 0.58 m.

Original languageEnglish (US)
Title of host publicationComputer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings
EditorsVittorio Ferrari, Cristian Sminchisescu, Yair Weiss, Martial Hebert
Number of pages19
ISBN (Print)9783030012243
StatePublished - 2018
Event15th European Conference on Computer Vision, ECCV 2018 - Munich, Germany
Duration: Sep 8 2018Sep 14 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11208 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Other15th European Conference on Computer Vision, ECCV 2018


  • 3D gaze estimation
  • 3D temporal scene understanding
  • Discovering objects
  • MCMC
  • Model selection
  • Monocular video

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'Multiple-Gaze Geometry: Inferring Novel 3D Locations from Gazes Observed in Monocular Video'. Together they form a unique fingerprint.

Cite this