I’ve been doing a lot of phone interviews lately, and as anyone who’s done them before knows, they can be a little tricky. Interviewees are sometimes wary (“Why are you paying me for this research?”), discouraged (“I can’t possible be giving you good answers, I don’t know anything about that topic”), belligerent (“I won’t answer that question!”) and vague (“Yep, sure – whatever you just said.”), or any combination thereof.
Having said that, interviewees can also be genuine and honest, insightful and thoughtful, inspired (and inspiring), as well as way more capable and knowledgeable than they sometimes think they are. When this happens, conducting interviews becomes a breeze and an absolute joy.
Mine is by no means the final word on interviewing and interviewing techniques – I’m not a fan of sticking to the script like some wooden C3PO, nor am I an advocate of being a total cowboy, or unprofessional. I am, however, a disciple of what I call the “Human School.” Here are some of its tenets:
1. Treat them like people (also known as: creating rapport and warm and fuzzies)
This is a big one – and the hardest “rule” to pin down. I hate it when telemarketers launch into a spiel after confirming who I am, or when an interviewer “hmmhmm” and “uh huhs” in agreement with what I’m saying without giving the impression that they’re listening. When I conduct an interview, I do my best to:
- Show them I care: “Is this still a good time?”, “Did you have a good weekend?, “Do you want to get that phone in the background?”. The
- Show them I’m listening: “Oh, so did you mean this?”, or “I wonder if that means X…?”
- Show them I’m human: “Yeah, I totally get what you’re saying, that just happened to me!”, or “Are you serious? :: cracks up ::”
As with most of the Human School these are fast and loose pointers, but I’ve found they go a long way toward establishing a rapport with people, which engenders trust and helps them speak more freely (which is really what you’re after).
2. Speak in natural language (no professional B.S.)
This is another toughie. Interviewers often follow scripts where there are certain things they can say to acknowledge and spur conversation (“OK” “I see”, “Got that”), but I find this so stilted. At it’s most basic level an interview is a conversation between two humans (albeit with one speaking more than the other), so converse. Speak. Engage.
3. Deviate from script – but cover all your bases
Related to the previous tenet, in the Human School it’s OK to deviate from your script, or change the order of your questions. Conversations are fluid, and there’s no need to force a structure when it could be detrimental. So feel free to deviate from your script a little and dive into rat holes – as long as you’re sure to cover all of your questions. I typically give myself 2-3 rat holes per hour-long session, which usually gives me time to cover my questions as well as explore any new areas that might pop up.
4. Don’t be shy
This is the hardest one for me – sometimes the questions I need to ask could be seen as too probing, too personal, too invasive. The key here is to make sure you do two things:
- Frame the conversation properly. Explain how wide-ranging (or not) the interview might be, and that if the interviewee doesn’t like any of your questions to let you know, and you’ll just move right on. I sometimes reiterate that the interview is anonymous (if it’s been explained already during the screening), but I often leave it out because I feel like it sets a negative overtone for the interview.
- Keep tougher questions for until you have rapport. Many interview scripts go like this: Gather demographic information (age, sex, education, financial history, etc.), talk about interest area 1, talk about interest are 2, etc… The thing is, I think asking about someone’s financial history (“How many credit cards do you have? What are their balances?”) is like asking someone if they want kids on a first date. It’s just not cool. So I typically wait until I have a rapport with someone before I start asking tougher/more invasive questions, sometimes even tacking them on at the end like an afterthought (“Oh yeah, I almost forgot to ask – how much are your car payments, again?”). This way I don’t scare them off up front, and I still get the information I need.
Now, a lot of these points won’t necessarily work if you’re after quantitative feedback, or trying to get “benchmark” ratings of some kind. Positive affirmation can be seen as “leading”, and this a huge no-no for many research projects. However, I do my best to go by the rule: “When in doubt, just be human.”
Hey, Vel here, bloggin’ about user testing and methods best practices today! Here’s a real
question I got this week. The Dear Vel part was just me adding color – because I fancy myself an
advice giver and all….
Dear Vel,
For my usability report, I am debating whether comments or issues from just one or two users should be included in the report. I can’t really generalize to the user group from just one person, right? So should I include them to be complete or not?
The answer to this question and the larger topic of “how far to go with reporting user comments
in general, and when to classify a trouble spot or frustration point as an issue” could be a book
chapter or more. And it surely is somewhere. Number of users commenting or encountering the
issue is just one factor, along with criticality of the error, frequency of the task in the real world,
and so forth. Another blog post on that perhaps later.
Here are some quick thoughts for your specific question to get you going:
For actual measurable issues encountered by a low percentage of participants (e.g., measured
by user errors, need for assistance, or even low perceptual scores), the criticality of the issue is
usually the most obvious and meaningful gauge of whether to classify and report as a “usability
issue.” A relatively critical error where the user loses data or requires intervention to recover,
even if encountered by only one person, should likely be reported. After all, if that one person got
into that trouble spot, what do you think the chances are that other people out there are going to
get caught in that same spot? Unless you have good cause to explain why you think no one else
will find that an issue, it’s likely other people are going to be affected.
But what about just comments?
In general, whether or not to include comments/verbal issues/potential issues from one person
(or, really, from any number of people) depends on your professional judgment as to the utility of
the comments to the client.
I say all the time that ‘Great ideas can come from anyone’ and ‘Great ideas can come from just
one person.’
Here are some questions to consider… If you can answer yes to any of the following, perhaps the
comment is worth including.
Example case: In one of my recent more benchmark-style tests, there were some sections/tasks
that we spent comparatively very little time on, so when I did receive a spontaneous comment
that was novel and I deemed it to be a potentially good idea to think about for the design, I
definitely wanted to make sure to communicate it to the client. In my professional opinion, for
this particular study and product, I believe that if we had been able to cover more ground on
that task with all participants and explore their needs, it is likely other participants would have
had similar suggestions. I did not want to lose the data by not reporting what I determined were
good suggestions by users. Of course, I did qualify in the report that we did not have time to
fully explore this topic with all users so more investigation would be needed for a complete
recommendation.
Note: Even if the above questions apply, you still may or may not judge that the comment/issue
can solidly be classified as an “Issue” in the issues table, but you can certainly include it in
an “Additional Observations” section and/or part of the detailed qualitative results section of a
report.
Does this help you?
Anyone have additional ideas or want to disagree on something I’ve said?
Happy testing!
great post! this is definitely something I think about when doing user testing… I also tend to go with a “gut feeling” approach – like if only one or two people say something but I feel like others would feel the same way, or it’s truly a unique idea I’ll certainly communicate it to the client. I also sometimes compromise by putting it in a “more issues” section outside the issues table. Also, as you suggested, it really depends what “kind” of issue it is. If it’s a bug or interesting idea, report it. But if it’s just this one person’s opinion about something, and could have been corroborated by others but wasn’t, I’ll leave it out.
