Critical look at eyetracking in usability tests

Jared Spool, whose well-conceived user testing approaches and analyses always seem to pass the dumbass test, wonders whether eyetracking is worth the expense.

"It's always good when your clients and developers become aware of how users behave. And an eyetracker is a great demonstration of fine grain behavior. In just mere moments, you can easily see how users gaze at the screen. So, I agree eyetrackers have demonstrative value.

"But do they have diagnostic value? Can we actually learn what to change in our designs from them?

"Well, after watching hundreds of eyetracking tests, I can tell you it's still really hard to know what you can learn from them."

A few of you may remember I wrote an analysis of findings from 2004 eyetracking research on hypothetical newspaper Web sites. Spool was one of several people I interviewed for that roundup, and he raised valid concerns about the methodology.

"The big flaw in this study is that it assumes that the actual news has no effect on the reader. But, we all know that isn't true. And study after study shows that it isn't true.

"This is amplified in this particular study because they explicitly 'chose mostly "evergreen" content that wouldn't appear dated, rather than deadline news.' Yet it is that very deadline news that drives people to online news sources on a regular basis.

"I'm having trouble with every finding in this study because the mock-up content was so badly neutered and no controls or measures are reported about the relationship that the users have with the mockup content."

In that case, Spool wasn't cautioning so much about the use of eyetracking tools as the reliance on faux sites. And he had a good point. If it's not a "real" site, and the usage isn't tracked in a "real" environment, how can you be sure the results resemble anything "real"?

Still, I see the 2004 eyetracking findings cited often as design guidance for news sites. And you have to remember how many newspaper executives made dramatic changes to their designs following a round of far more primitive eyetracking research on print papers in the 1980s.

I believe that was the pivot point where newspaper design became more about attracting gaze (the only thing eyetracking honestly measures, right?) than holding attention.

In any event, I came away from writing the 2004 article believing that eyetracking is just one more way to get user feedback, not the way to understand how people use a site, or the whole Web. Spool's latest post just validates that belief.

Budget notwithstanding, I'd have no problem adding eyetracking tests to a Web user test program. But I wouldn't make eyetracking my whole user test program, at the expense of other methods that can be at least as informative.