July 03, 2005

Eye Tracking as Silver Bullet for Usability Evaluations?

When reading articles on usability evaluation, talking to other usability professionals and clients, one can get the impression that eye tracking is perceived as silver bullet for conducting usability evaluations. After all, the method allows us to directly assess how users perceive an interface, which parts they pay attention to and which parts they miss. And all this just by getting them seated in front of the interface and tracking their gaze. Right?
Well...no, actually. Matters are a bit more complicated here.
The misconceptions about eye tracking are - at least in part - due to the fact that visual perception seems to be a fairly straightforward process at the first glance (no pun intended...). I fixate an object, I perceive it and then can use it in my working memory. So measuring fixations should be the direct way to the contents of my working memory. But this model is flawed. Here are some reasons why eye tracking is not as straightforward as it may seem.

  • On a very basic level, even measuring fixations is by no means a "standardized" procedure. In fact, the fixations a system will determine depend heavily on the fixation algorithm used. So the same raw data may result in different fixation patterns and scanpaths when analyzed on different systems. Imagine such a dependency on the concrete measuring instrument used in the case of a nurse measuring the temperature of a patient...
  • Even if we assume that we got the fixation determination issue solved, it is still unclear what a fixation actually means. Studies in experimental cognitive psychology show that it is absolutely possible to fixate an object without consciously perceiving it - and not to fixate an object and being very well aware of it (parafoveal vision). So eye tracking is not the "attention measurement" as which it is often entitled.
  • OK, now let's say fixation determination is standardized and fixation equals attention. Then everything’s fine, right? - Nope.
    With the concept of "attention" the fun just starts. When consulting your library (or psychologist friend) be prepared to bring a lot of time. There are different opinions on the relation between "attention" and behaviour (and some even say that one should discard the concept of attention altogether). Basically, there is no general "law" that relates attention (or fixations in our case) to actions. Different people may do different things with an interface, even after showing similar fixation patterns. This may be bad news to people having their ads investigated in eye tracking studies to increase sales.
  • Apart from all this, not every person can be eye tracked. Drop out rates of 20% are standard. This may quickly become an issue when people are being paid for taking part in an evaluation. And it doesn't exactly make eye tracking the preferred method for projects with a tight schedule.

So is eye tracking all bad? Should it be used at all? This can of course be answered with: "It depends". When using eye tracking in usability evaluations, having some guidelines in mind can be helpful.

  • Document the specific parameters of your eye tracking system and don't switch systems (or parameters, to be more specific) when making comparative eye tracking studies.
  • Have specific questions and consult the literature for relevant findings. Don't do eye tracking just "to have a look". You won't get reasonable results when you don't have an idea what you are looking for, or in which parts of the data to look for it.
  • Don't make eye tracking your only data source. It is helpful relating it to similar research issues and validate it against other sources, such as performance data.
  • Don't believe that you are necessarily measuring the users' "attention" or that it is the ultimate measure. The subjective impression the user has of an interface and the elements of high interest may not be clearly reflected in eye tracking data.
  • Qualitative data may be more interesting that quantitative data. Instead of measuring a lot of users' eye movements for quantitative analysis, it could be more effective tracking a few representative users working on selected parts of the interface. For a skilled researcher this may provide considerable insight and serve as a basis for a more detailed investigation of issues (maybe with completely different methods).

So, eye tracking is no silver bullet for investigating usability issues. It may be an excellent marketing instrument, but inappropriately used, it may backfire.

2 comments :

Raj said...

Your aritcle on eye trackign is elobrate. I google it and got your page. I ahve also written on eye tracking. Have a look at my blog. www.suggestusability.com

Mark said...

@raj
Thanks. As it seems, the issue of eye tracking keeps drawing people’s attention.