What You Always Wanted to Know About Eye Tracking - Part 2: Data Accuracy

After discussing the issue of fixation detection in the first part of the series, this post will provide some additional basic knowledge about the eye tracking technology before diving into some more advanced conceptual aspects of eye tracking in part three.

An essential part of the eye tracking setup is the camera that captures light reflected by the eye in order to analyze the information and, on that basis, determine the gaze direction. If the eye tracker is not head-mounted, the camera is often positioned statically, e.g., above or below a computer monitor on which the "test material" is displayed.

Head Movements and Chinrests

The precision of measurement is influenced by movements of the head. Movements of the head towards or away from the camera can - temporarily - move the captured image out of focus, until the auto-focus of the camera has corrected the situation. Until this has happened, accuracy of data can be impaired.
In addition to movements of the head towards or away from the camera, sideway movements can, obviously, make the camera lose the image completely, which is usually addressed by making the camera follow the captured image up to a certain extent ("camera tracking"). When the head movement exceeds the range that camera tracking can compensate for (due to larger movements of the head), the camera can either remain at the position where the image was last captured or return to a default position. As soon as a reflection is captured again, the data acquisition continues. In the meantime, no data can be collected.

Head movements are especially likely whenever the person whose gaze is tracked deals with interactive systems, i.e. does not only sit passive in front of a screen, e.g. only reading, with as little movement as possible. This means that, with interactive material, head movements and the corrective mechanisms described above are likely to come into play. This also implies that, until the respective corrections are completed, there can be "impurities" in the data collected, i.e. either inaccurate data or missing data points.

One approach for improving measurement precision consists in fixating the head, e.g., by using a "chin rest". While this effectively eliminates most head movements, it also results in an even more unnatural situation for the person whose gaze is tracked. (Imagine sitting in front of your computer without being able to move your head.) In addition to being fixated, talking is hardly possible with a chin rest, which limits the kinds of studies that can be done.

Systematic Error and Implicit Required Fixation Locations

In addition, even with fixating the head, measured data can contain errors. While unsystematic "noise" can usually be reduced through averaging, systematic errors (i.e. a systematic offset in the measured data points) cannot be eliminated through calculating averages (because the errors of the individual data points don't cancel each other). Such systematic errors are described, e.g., by Hornof and Halverson (2002), who dealt with ways of addressing this error type: „Some systematic error can be identified and manually removed if, when fixations superimposed on the visual stimuli are studied, a consistent pattern of systematic error can be identified from trial to trial. But systematic error cannot be reduced by averaging across gazepoints“ (p. 592). Such systematic errors can exist even right after calibration has taken place: „Even after calibration, eye trackers often maintain a systematic error such that fixations are recorded a small distance from the fixation point“ (p. 592). In addition, measurement precision can deteriorate during measurement: „Drift error may result from, for example, head movements and changes in the position of eye glasses and contact lenses“ (Surakka, Illi & Isokoski, 2003, p. 478). „As with most instrumentation, the initial calibration may deteriorate during an experiment“ (Hornof & Halverson, p. 593). On the basis of such insight, Hornof and Halverson state that formal criteria for eye tracking studies are recommendable, which allow insight into data accuracy and also enable decisions regarding new calibrations that are triggered during measurement whenever the desired level of accuracy is in danger: „An objective verification step and criterion contribute to the accuracy and usefulness of the data recorded, the confidence in the conclusions drawn from the data and the reproducibility of the experiment“ (p. 593).

As one approach for doing so, Hornof and Halverson describe "implicit required fixation locations" to identify systematic errors and - if required - automatically trigger a re-calibration:
"Implicit RFLs can be used to monitor eye tracker calibration in real time and to invoke recalibration when necessary. Task requirements, when combined with human perceptual and motor characteristics, often constrain where fixations will occur. In many human-device task executions, there are a number of locations that the participant must fixate in order to accomplish a task. .... The target of a mouse movement, if small enough, would make a very good implicit RFL. .... Incorporating mouse movements into an experimental task on a computer provides an opportunity for implicit RFLs." (p. 594)
This means that for a given task, areas of the screen (or parts of the "visual scene" that is relevant for the task) should be determined that are very likely to be fixated for the person to successfully complete the task. These can, e.g., consist of text that has to be read, as it is likely that the text has to be fixated (in contrast to perceiving it parafoveally).
Hornof and Halverson made use of two Implicit RFLs in their experimental setting. To initiate an experimental trial, participants had to click a button to start it, which served as Implicit RFL 1 to check accuracy of calibration and trigger a new calibration in case accuracy was below a pre-determined threshold. During each trial, participants had to find and click on a target stimuli, which also served as Implicit RFL 2. Implicit RFL 2 was not, however, used to be able to trigger a new calibration, but instead to check for the accuracy of the calibration that had taken place previously.

What should have become clear from the aspects discussed in this post is that, just because eye tracking provides insight into an - otherwise mostly "hidden" - aspect of human behavior with a very high temporal resolution, this does not necessarily mean that it is per se an accurate method. As with every kind of measurement, data accuracy is an aspect that must explicitly be taken care of by the researcher. When conducting eye tracking studies, data accuracy should therefore be

• "arranged for" with proper calibration
• monitored during measurement
• improved by triggering a re-calibration in case accuracy drops below a pre-determined threshold
• reported with the results

After covering some basic aspects of "eye tracking mechanics", the next part of this series will focus on conceptual issues around eye tracking measurements.


References
Hornof, A.J. & Halverson, T. (2002). Cleaning up systematic error in eye-tracking data by using required fixation locations. Behavior Research Methods, Instruments & Computers, 34(4), 592-604.

Surakka, V., Illi, M. & Isokoski, P. (2003). Voluntary eye movements in human-computer interaction. In J. Hyönä, R. Radach & H. Deubel (Eds.), The mind’s eye: Cognitive and applied aspects of eye movement research (p. 473-491). Amsterdam: Elsevier.