Hey, Vel here, bloggin’ about user testing and methods best practices today! Here’s a real question I got this week. The Dear Vel part was just me adding color – because I fancy myself an advice giver and all….

Dear Vel, For my usability report, I am debating whether comments or issues from just one or two users should be included in the report. I can’t really generalize to the user group from just one person, right? So should I include them to be complete or not?

The answer to this question and the larger topic of “how far to go with reporting user comments in general, and when to classify a trouble spot or frustration point as an issue” could be a book chapter or more. And it surely is somewhere. Number of users commenting or encountering the issue is just one factor, along with criticality of the error, frequency of the task in the real world, and so forth. Another blog post on that perhaps later.

Here are some quick thoughts for your specific question to get you going:

For actual measurable issues encountered by a low percentage of participants (e.g., measured by user errors, need for assistance, or even low perceptual scores), the criticality of the issue is usually the most obvious and meaningful gauge of whether to classify and report as a “usability issue.” A relatively critical error where the user loses data or requires intervention to recover, even if encountered by only one person, should likely be reported.

After all, if that one person got into that trouble spot, what do you think the chances are that other people out there are going to get caught in that same spot? Unless you have good cause to explain why you think no one else will find that an issue, it’s likely other people are going to be affected.

But what about just comments?

In general, whether or not to include comments/verbal issues/potential issues from one person (or, really, from any number of people) depends on your professional judgment as to the utility of the comments to the client. I say all the time that ‘Great ideas can come from anyone’ and ‘Great ideas can come from just one person.’

Here are some questions to consider… If you can answer yes to any of the following, perhaps the comment is worth including.

• Is the comment a novel idea you (and the client) perhaps haven’t though of?

• Do you think it potentially merits further research even though you can’t make a “generalization” yet?

• Do you think the comment/idea adds to the overall assessment and understanding of usability?

• Was this a spontaneous comment, and would you have likely received more such comments if you had the time to “drill in” with more participants?

• Was this participant different somehow and might lend a different, important perspective to the results?

Example case: In one of my recent more benchmark- tests, there were some sections/tasks that we spent comparatively very little time on, so when I did receive a spontaneous comment that was novel and I deemed it to be a potentially good idea to think about for the design, I definitely wanted to make sure to communicate it to the client.

In my professional opinion, for this particular study and product, I believe that if we had been able to cover more ground on that task with all participants and explore their needs, it is likely other participants would have had similar suggestions. I did not want to lose the data by not reporting what I determined were good suggestions by users. Of course, I did qualify in the report that we did not have time to fully explore this topic with all users so more investigation would be needed for a complete recommendation.

Note: Even if the above questions apply, you still may or may not judge that the comment/issue can solidly be classified as an “Issue” in the issues table, but you can certainly include it in an “Additional Observations” section and/or part of the detailed qualitative results section of a report.

Does this help you? Anyone have additional ideas or want to disagree on something I’ve said?

Happy testing!