Differentiable Deflectometric Eye Tracking

Eye tracking is an important tool in many scientific and commercial domains. State-of-the-art eye tracking methods are either reflection-based and track reflections of sparse point light sources, or image-based and exploit 2D features of the acquired eye image. In this work, we attempt to significantly improve reflection-based methods by utilizing pixel-dense deflectometric surface measurements in combination with optimization-based inverse rendering algorithms. Utilizing the known geometry of our deflectometric setup, we develop a differentiable rendering pipeline based on PyTorch3D that simulates a virtual eye under screen illumination. Eventually, we exploit the image-screen-correspondence information from the captured measurements to find the eye's rotation, translation, and shape parameters with our renderer via gradient descent. We demonstrate real-world experiments with evaluated mean relative gaze errors below 0.45 ° at a precision better than 0.11 °. Moreover, we show an improvement of 6X over a representative reflection-based state-of-the-art method in simulation. In addition, we demonstrate a special variant of our method that does not require a specific pattern and can work with arbitrary image or video content from every screen (e.g., in a VR headset).