Hey, Vel here, bloggin’ about user testing and methods best practices today! Here’s a real
question I got this week. The Dear Vel part was just me adding color – because I fancy myself an
advice giver and all….
Dear Vel,
For my usability report, I am debating whether comments or issues from just one or two users should be included in the report. I can’t really generalize to the user group from just one person, right? So should I include them to be complete or not?
The answer to this question and the larger topic of “how far to go with reporting user comments
in general, and when to classify a trouble spot or frustration point as an issue” could be a book
chapter or more. And it surely is somewhere. Number of users commenting or encountering the
issue is just one factor, along with criticality of the error, frequency of the task in the real world,
and so forth. Another blog post on that perhaps later.
Here are some quick thoughts for your specific question to get you going:
For actual measurable issues encountered by a low percentage of participants (e.g., measured
by user errors, need for assistance, or even low perceptual scores), the criticality of the issue is
usually the most obvious and meaningful gauge of whether to classify and report as a “usability
issue.” A relatively critical error where the user loses data or requires intervention to recover,
even if encountered by only one person, should likely be reported. After all, if that one person got
into that trouble spot, what do you think the chances are that other people out there are going to
get caught in that same spot? Unless you have good cause to explain why you think no one else
will find that an issue, it’s likely other people are going to be affected.
But what about just comments?
In general, whether or not to include comments/verbal issues/potential issues from one person
(or, really, from any number of people) depends on your professional judgment as to the utility of
the comments to the client.
I say all the time that ‘Great ideas can come from anyone’ and ‘Great ideas can come from just
one person.’
Here are some questions to consider… If you can answer yes to any of the following, perhaps the
comment is worth including.
Example case: In one of my recent more benchmark-style tests, there were some sections/tasks
that we spent comparatively very little time on, so when I did receive a spontaneous comment
that was novel and I deemed it to be a potentially good idea to think about for the design, I
definitely wanted to make sure to communicate it to the client. In my professional opinion, for
this particular study and product, I believe that if we had been able to cover more ground on
that task with all participants and explore their needs, it is likely other participants would have
had similar suggestions. I did not want to lose the data by not reporting what I determined were
good suggestions by users. Of course, I did qualify in the report that we did not have time to
fully explore this topic with all users so more investigation would be needed for a complete
recommendation.
Note: Even if the above questions apply, you still may or may not judge that the comment/issue
can solidly be classified as an “Issue” in the issues table, but you can certainly include it in
an “Additional Observations” section and/or part of the detailed qualitative results section of a
report.
Does this help you?
Anyone have additional ideas or want to disagree on something I’ve said?
Happy testing!
Much of our work results in our clients having to make changes to an existing product or service. It’s kind of the point of usability and design: unless everyone loves your product, careful research and testing will be sure to raise a few things that could be changed. Whether it’s the color of an icon or the entire product concept, design leaves no stone unturned.
One of the common refrains we hear from clients is this:
“But how will users know what we’ve changed??!”
(implication: they will hate it…)
We have lots of answers to that question, but here’s an example of my favorite (and perhaps the simplest) answer:
The lesson, it would seem, is to just tell them.
I’m running a stealth usability study for an iPhone app at the moment, and we’re on a tight 4 week RITE (Rapid Iterative Test and Evaluation) schedule.
For those not in the know, RITE is a way of condensing the usability and design prototyping process into a short time frame. Instead of testing a dozen users in one big batch, writing a report with design recommendations, submitting the report, making some of the design changes (or not), and then starting the testing all over again, RITE lets us do the same thing in about a month. We test 4 users every week instead of 12 once a month; we make design changes on the fly instead of waiting for a report to be produced; we get all the stakeholders involved so things can happen today and tomorrow, instead of next week or next month. In many ways, RITE is the way that usability – and design more generally – should always be done (in my opinion).
One of the problems with RITE, however (and there are a few, believe you me) is that recruiting users becomes a pressure-filled activity. Instead of having weeks to schedule participants, fill in missing slots, etc., we have days to schedule people, sometimes only one day. For example, I scheduled 2 users yesterday for a session today – pretty nuts if you ask me.
But this entry isn’t so much about RITE, or how to test an iPhone app, or why an external USB camera doesn’t play nice with live streaming and a screencast – I’ll get to all these issues in good time. No, this time I want to focus on a particular user (who shall not be named for obvious reasons), and how persistent she was today. If any of you live in the Bay Area you’ve probably heard that the Bay Bridge is closed, which is wreaking havoc on…well, just about everything. And this poor user-who-shall-not-be-named made a valiant effort today to get across said closed bridge, only to spend another hour and a half trying to get into town from the East Bay. She finally gave up when she realized that making the session would mean leaving her daughter stranded at school without a ride.
So to every user who has ever spent an inordinate amount of time getting to a session, all in the name of a better product, I salute you.
great post! this is definitely something I think about when doing user testing… I also tend to go with a “gut feeling” approach – like if only one or two people say something but I feel like others would feel the same way, or it’s truly a unique idea I’ll certainly communicate it to the client. I also sometimes compromise by putting it in a “more issues” section outside the issues table. Also, as you suggested, it really depends what “kind” of issue it is. If it’s a bug or interesting idea, report it. But if it’s just this one person’s opinion about something, and could have been corroborated by others but wasn’t, I’ll leave it out.
Shawna