Done! 5pm today, after I signed off from my EchoUser Analytics account, said goodbye to my 10th participant, I officially finished the testing part (oh yeah, the research and report writing part is yet to come… 
) for this usability benchmarking study on an upcoming Cisco router product.
This new router product will be used to set up networks in small business environment, so that means, if you are reading this blog post right now, my dear readers, most likely, never once in your life, you’ll be put in a position to use a product like this.
So who are the users?
Engineers or technicians who work at Cisco resellers. People who usually have years of experience working in the field, at clients’ sites, helping tens if not thousands of companies setting up their wired and wireless network, configuring the network’s security services, customizing VPN accounts, etc. They have to be the power users of the UI of router products. This is their job, so for one thing, they interact with UIs in this domain on a daily basis; for another thing, it’s their expertise in this domain that makes them special and let them be able to earn their living.
I did expect those power users to go through the router configuration tasks we prepared for them in a logic and efficient way. However, what I did NOT expect was how they felt extremely “rewarded” after they had overcome the obstacles in the UX and UI design of the router product. Normally, users are supposed to feel defeated when they make errors and mistakes when interacting with a system.
One interesting anecdote: there was this one task I asked the users to establish a site-to-site VPN tunnel (simply put, just to let two routers on the Internet to talk to each other). There was this technician from San Jose, who had more than 20 years of experience dealing with router products, sitting in front of the desktop to finish this task. Despite various of hurdles he had to go through to be able to get to the last step of the whole configuration process, what finally threw me off was, in order to complete the task, as the last step, the user needed to click this super tiny and grayed out “connect’ icon which was hidden among a group of other icons on the screen, it was as if a joke the product designer intentionally played on the users to NOT let them accomplish what the users hope to accomplish. After exploring on the configuration page for over 5 minutes, perhaps guided by some supernatural forces, the user actually did find that little “connect” icon, and clicked on it. Mission succeed!
With all those frustrations and confusions I observed in this process, I would expect this San Jose network engineer to give me some feedback like “that process was frustrating” “they could have designed in a better way to make the icon more obvious.” However, in fact what I heard was “that wasn’t too hard”, “glad that I did find that button to click.”
He seemed to have enjoyed this “intellectual battle” with the product designers. On one side, the product designers were creating these super hard “mazes” for power users to solve; on the other side, the more difficult these “mazes” are, the more rewarding these power users would feel, and to some extent they are actually proud of the fact that they are given “mazes” like these to solve. Those “mazes” are the watermarks for their professionalism.
I recently came across this interesting perspective from Alan Cooper‘s book “The Inmates are Running the Asylum,” he categorizes the modern technology users into two groups:
“High cognitive friction polarizes people into two groups. It either makes them feel frustrated and stupid for failing, or giddy with power at overcoming the extreme difficulty. These powerful emotions force people into being either an ‘apologist’ or a ‘survivor.’ They either adopt cognitive friction as a lifestyle, or they go underground and accept it as a necessary evil.”
As Cooper put it “regardless of how hard an interaction is, or how uselessly obscure a feature is, the apologist will unerringly point to the power and functionality of the gadget, blithely ignoring the difficulty of actually using it.” I wouldn’t say that was exactly what had driven the engineer who apparently “suffered” from the user experience of the router product to still give me positive feedback, but I think Cooper has given a quite reasonable explanation to it.
Indeed, engineers like to work in an environment where their special skills and domain knowledge can make a difference. That’s how they differentiate themselves from the crowd. In a sense, they are shouting at the product (the product designers indeed), saying “the tougher the better!” However, should that be the criteria for UX professionals to design products for them? Perhaps not. The increased level of complexity of a system would increase the possibility of human errors, the possibility of project failures and the level of education/training needed to operate a system.
Well, if the price we are going to pay for implementing simpler and more usable UI for power users (like those network engineers) will just be: the lowered self-efficacy for those powerful power users, I’d say “deal.” 
Hey, Vel here, bloggin’ about user testing and methods best practices today! Here’s a real
question I got this week. The Dear Vel part was just me adding color – because I fancy myself an
advice giver and all….
Dear Vel,
For my usability report, I am debating whether comments or issues from just one or two users should be included in the report. I can’t really generalize to the user group from just one person, right? So should I include them to be complete or not?
The answer to this question and the larger topic of “how far to go with reporting user comments
in general, and when to classify a trouble spot or frustration point as an issue” could be a book
chapter or more. And it surely is somewhere. Number of users commenting or encountering the
issue is just one factor, along with criticality of the error, frequency of the task in the real world,
and so forth. Another blog post on that perhaps later.
Here are some quick thoughts for your specific question to get you going:
For actual measurable issues encountered by a low percentage of participants (e.g., measured
by user errors, need for assistance, or even low perceptual scores), the criticality of the issue is
usually the most obvious and meaningful gauge of whether to classify and report as a “usability
issue.” A relatively critical error where the user loses data or requires intervention to recover,
even if encountered by only one person, should likely be reported. After all, if that one person got
into that trouble spot, what do you think the chances are that other people out there are going to
get caught in that same spot? Unless you have good cause to explain why you think no one else
will find that an issue, it’s likely other people are going to be affected.
But what about just comments?
In general, whether or not to include comments/verbal issues/potential issues from one person
(or, really, from any number of people) depends on your professional judgment as to the utility of
the comments to the client.
I say all the time that ‘Great ideas can come from anyone’ and ‘Great ideas can come from just
one person.’
Here are some questions to consider… If you can answer yes to any of the following, perhaps the
comment is worth including.
Example case: In one of my recent more benchmark-style tests, there were some sections/tasks
that we spent comparatively very little time on, so when I did receive a spontaneous comment
that was novel and I deemed it to be a potentially good idea to think about for the design, I
definitely wanted to make sure to communicate it to the client. In my professional opinion, for
this particular study and product, I believe that if we had been able to cover more ground on
that task with all participants and explore their needs, it is likely other participants would have
had similar suggestions. I did not want to lose the data by not reporting what I determined were
good suggestions by users. Of course, I did qualify in the report that we did not have time to
fully explore this topic with all users so more investigation would be needed for a complete
recommendation.
Note: Even if the above questions apply, you still may or may not judge that the comment/issue
can solidly be classified as an “Issue” in the issues table, but you can certainly include it in
an “Additional Observations” section and/or part of the detailed qualitative results section of a
report.
Does this help you?
Anyone have additional ideas or want to disagree on something I’ve said?
Happy testing!
great post! this is definitely something I think about when doing user testing… I also tend to go with a “gut feeling” approach – like if only one or two people say something but I feel like others would feel the same way, or it’s truly a unique idea I’ll certainly communicate it to the client. I also sometimes compromise by putting it in a “more issues” section outside the issues table. Also, as you suggested, it really depends what “kind” of issue it is. If it’s a bug or interesting idea, report it. But if it’s just this one person’s opinion about something, and could have been corroborated by others but wasn’t, I’ll leave it out.
Interesting post and sounds like enjoyable research – I’ve thought on and off about those issues for a while now, especially when I used to work at Microsoft. You’d look at a product like Microsoft Word, that was definitely in most of its iterations not the most user-friendly, but in some sense that made mastery of it a skill that people could incorporate into their professional identities. Being able to use Excel and Word was for a while an essential resume skill. Same with general computer usage at a time when computers were still beyond the average person’s cognitive framework. So if people experience a thrill or gain a marketable skill from beating a difficult system, like you said, does making it simpler and easier have unintended negative consequences?
I believe a lot of it has to do with the professional vs. personal life aspects. In our professional lives we are supposed to master complex systems [i.e. Adobe Creative Suite] whereas in our personal lives, as customers we often prefer service and simplicity [no one wants to "master" their banking website]. Your power users weren’t just power users, but professional power users. Also, difficult interfaces endow their power users with the value of scarcity, even in a personal social realm [i.e. if cars were easy to fix then your amateur mechanic neighbor wouldn't have as much social value].
Anyways, thanks for writing this up!
Morgan