During usability test sessions, we usually ask the participants to say out loud what they are thinking. What they say ends up being important data to justify what is right or wrong about a software page design. We trust that what they say as veridical, at least as long as it is consistent with our own perceptions of the events.
But what is happening when participants talk? Are they reporting events as they occur? Are they filtering what they perceive and reporting their interpretation of it? Is their interpretation biased by what they expect to see or what they think we expect them to see?
This issue reminds me of a book, The Electric Kool-Aid Acid Test, by Tom Wolfe published in 1968. In this work of new journalism, Wolfe describes the adventures of Ken Kesey. Kesey, the author of One Flew Over the Cuckoo’s Nest, was an early advocate of the use of LSD, before it became illegal. He believed that when you take LSD your brain chemistry changes and you are the closest to “being in the moment” that a human can be. He believed in something called “the gap.” When we perceive an event, is has already happened, hence the gap. There are gaps between the event and when we sense it; between the event and when we perceive it and between the event and our feelings about it. He and the people who gathered around him, called the Merry Pranksters, would take LSD to try to get as close as they could to being in the moment, that is to minimizing the gap between when an event occurs and their perception of it. They lived for living in the moment.
So if there is a gap between events and their perceptions, what is a participant in a usability test talking about? It can’t be what is happening. At best it is their memory of what just happened. Also we know that people can think many times faster than they can talk. So there is plenty of time to interpret what is perceived, especially if you slow down your talking. This gap is why cognitive psychology researchers don’t trust what people say as reflecting what is going on in their heads when they are allowed to report/interpret whatever they choose.
So what’s up with thinking aloud?
Hey, Vel here, bloggin’ about user testing and methods best practices today! Here’s a real
question I got this week. The Dear Vel part was just me adding color – because I fancy myself an
advice giver and all….
Dear Vel,
For my usability report, I am debating whether comments or issues from just one or two users should be included in the report. I can’t really generalize to the user group from just one person, right? So should I include them to be complete or not?
The answer to this question and the larger topic of “how far to go with reporting user comments
in general, and when to classify a trouble spot or frustration point as an issue” could be a book
chapter or more. And it surely is somewhere. Number of users commenting or encountering the
issue is just one factor, along with criticality of the error, frequency of the task in the real world,
and so forth. Another blog post on that perhaps later.
Here are some quick thoughts for your specific question to get you going:
For actual measurable issues encountered by a low percentage of participants (e.g., measured
by user errors, need for assistance, or even low perceptual scores), the criticality of the issue is
usually the most obvious and meaningful gauge of whether to classify and report as a “usability
issue.” A relatively critical error where the user loses data or requires intervention to recover,
even if encountered by only one person, should likely be reported. After all, if that one person got
into that trouble spot, what do you think the chances are that other people out there are going to
get caught in that same spot? Unless you have good cause to explain why you think no one else
will find that an issue, it’s likely other people are going to be affected.
But what about just comments?
In general, whether or not to include comments/verbal issues/potential issues from one person
(or, really, from any number of people) depends on your professional judgment as to the utility of
the comments to the client.
I say all the time that ‘Great ideas can come from anyone’ and ‘Great ideas can come from just
one person.’
Here are some questions to consider… If you can answer yes to any of the following, perhaps the
comment is worth including.
Example case: In one of my recent more benchmark-style tests, there were some sections/tasks
that we spent comparatively very little time on, so when I did receive a spontaneous comment
that was novel and I deemed it to be a potentially good idea to think about for the design, I
definitely wanted to make sure to communicate it to the client. In my professional opinion, for
this particular study and product, I believe that if we had been able to cover more ground on
that task with all participants and explore their needs, it is likely other participants would have
had similar suggestions. I did not want to lose the data by not reporting what I determined were
good suggestions by users. Of course, I did qualify in the report that we did not have time to
fully explore this topic with all users so more investigation would be needed for a complete
recommendation.
Note: Even if the above questions apply, you still may or may not judge that the comment/issue
can solidly be classified as an “Issue” in the issues table, but you can certainly include it in
an “Additional Observations” section and/or part of the detailed qualitative results section of a
report.
Does this help you?
Anyone have additional ideas or want to disagree on something I’ve said?
Happy testing!
great post! this is definitely something I think about when doing user testing… I also tend to go with a “gut feeling” approach – like if only one or two people say something but I feel like others would feel the same way, or it’s truly a unique idea I’ll certainly communicate it to the client. I also sometimes compromise by putting it in a “more issues” section outside the issues table. Also, as you suggested, it really depends what “kind” of issue it is. If it’s a bug or interesting idea, report it. But if it’s just this one person’s opinion about something, and could have been corroborated by others but wasn’t, I’ll leave it out.
During a recent project, a client asked us to evaluate the initial user experience for setting up a telephony product. They were not interested in doing a more traditional usability test as they felt that this “out of the box” experience could not be easily captured or replicated by forcing a user to perform a set of pre-defined tasks. They wanted feedback that was more contextual and ethnographic in nature, but they still wanted the quantitative data such as times, errors, assists, and usability ratings that you get from standard usability testing. The more we thought about this, the more we came to realize that there is a gap in methods from where lab testing ends and contextual/ethnographic studies begin.
Bridging the gap - Event Style Testing
Usability testing is often used for getting feedback or benchmarking the usability of key task flows of a product or service. These key tasks are discrete and isolated to make it easier to get more quantitative data, but this comes at the cost of losing what the natural workflow may be since a structure or order is enforced. The natural workflow is something that is researched through more contextual methods. What we wanted for this project was to be able to allow a natural workflow that would allow users to use multiple tools and do things that were not expected by the product team while being able to get metrics such as time, errors, assists, and perceived usability. With this in mind, we created a method that we’ve dubbed internally as Event Style Usability Testing to help bridge this gap between usability testing and contextual/ethnographic studies.
Components of Event Style Usability Testing
There are a couple of basic components to the use of Event Style Usability Testing
To better understand this, what do I mean by an “event”? Events are very similar to tasks that are used in more standard usability testing with a couple of distinctions. Like tasks, events are created that users are asked to perform or experience. For example, an event would be to send an online payment to a person from an online bank account. If this were simply a task in a usability study, then users may be told specifics about who to send the payment to, for how much, and when it should arrive by. For an event, things are turned into a scenario as much as possible. They are told what they are trying to accomplish, not given specifics. In the case of the event for sending an online payment, users would choose who to send it to, for how much, from what account, etc. Allowing them to shape the event using their own data or specific to their own interests allows interesting insights and observations about what users may do in real situations.
Once the events have been pre-defined, an event list is created for participants to use for the session. Prior to beginning the session, users are asked to read the entire list of events that they will be asked to do. They are then instructed to complete the list of events in whatever order they would like and makes the most sense to them. Having them read through the entire list of events this way allows them simulate more closely what they would do out in the real world. It’s assumed in most systems and software that users generally know what they want to do (or else why are they using the system or software in the first place) but may not know specifically how to do it. Event style testing aims to more closely replicated this.
The rest of the process now relies on the careful observation of the testing facilitator. Since there is no specified order that participants are using to complete events, the facilitator needs to monitor what events the participant is attempting or in the process of doing. This includes capturing time stamps of when an event is started, completed, or stopped and not completed, or resumed after being stopped previously. While observing each event, other usability metrics can be captured as well such as any observed errors, assists needed, and other usability scores.
The last piece to the Event Style Process is being flexible to add new events to record and capture in the middle of a session. When trying to replicate real world workflows, it can be nearly impossible to predict and plan for all paths and avenues that users may try to use to work through events. Also, unforeseen or unanticipated events may occur in context that would not occur in a highly structured usability test. The event style testing allows these newly observed events to be captured mid test so that they can be reported on later.
Benefits of Event Style Testing
The event style testing allows for a number of benefits and deliverables that cannot normally be obtained in a usability test. These include:
Workflow Analysis
Since the order that events take place is recorded during the study, a full task flow can be created and analyzed. This is particularly useful for systems or software where very little is known about how users setup or experience them, especially in complex systems where multiple tools or software may be needed. The resulting task flows can show where various tools and software are frequently used, in what order, and for what purpose. In the case of the telephony product we tested, the task flows highlighted at what points different software products were used for configurations and at what points users stopped to verify correct setups and troubleshoot problems.
Event Style Workflow
Quantitative data collection
While the natural workflow analysis can be observed, quantitative data such as errors, assists, and usability ratings can still be collected and compared across users. This creates a more holistic view of the experience being evaluated from a qualitative and quantitative standpoint, the best of both worlds.
Event Style Usability Testing has become a powerful method in our user experience toolbox to bridge the usability test and ethnographic research divide. We are always looking for new takes on old methods and hope others may find this useful as well!
Wow this is a great resource.. I’m enjoying it.. good article
[...] This post was mentioned on Twitter by Nino Palermo, Etan. Etan said: Article about event based usability testing: http://blog.echouser.com/2010/07/event-style-usability-testing/ [...]
[...] The EchoUser Experience » Event Style Usability Testing [...]
[...] there are still/always good questions to be asked about the way we pursue this goal, I am always interested in seeing other examples of how this kind of work happens both inside and [...]
People (un)like myself « The EchoUser Experience | Thoughts and stories from our work and otherwise. Enjoy.
Well Vel, your either on the bus or your off the bus.
BTW the Merry Pranksters loved to videotape themselves. They enjoyed watching themselves experiencing what had just happened – being two steps removed from reality.
Joe Dumas
Love the philosophical post! Or at least that is my post-experience, filtered-then-translated-into-language impression from across the gap. Guess you’ll never know what I really experienced as I read it. And perhaps that is a good thing! It probably would have a lot of noise (some colors perhaps?) and be mostly useless until processed and translated.
As for the relevance to usability testing, the addition of mind altering drugs to the protocol could introduce a positive(?) biasing effect. And how would we separate the effects of the drug experience from the effects of experiencing the interface? Perhaps if our product targets the ‘perpetually medicated’, you’re onto something here…
Good stuff!
Vel