The EchoUser research team had quite a busy December. Our schedules were filled with recruiting users, drafting test plans, moderating usability sessions, writing reports, and, last but not least, arranging check-in meetings with clients throughout the project cycle.
Clients — regardless of their UX background — would raise questions and concerns about UX methodology in those meetings to make sure that their studies were on the right track and that they would get valuable and defensible data from the projects.
In the two usability projects I am on (both benchmark studies), I came across the following two interesting questions from our clients. Though the two questions seemed to have come from two different angles, they both point to one of the key issues in doing usability studies: how to interpret usability data with a small number of users. I thought I’d share the two client questions and hope to elicit some extended discussions here.
Client Question 1: How many participants is enough for a benchmark usability study? Eight, 10, or 12?
A lot of times, the question actually becomes, “Do we need a single-digit participant number or a double-digit one?” Clients want the usability study results to be defensible both from a statistical and a PR standpoint. When time and resources allow and it’s easy to recruit target participants, the question of “Should we get two more participants for the study?” has an easy solution: Let’s just do two more sessions. However, in a scenario in which qualified participants are very difficult to find or recruit (for instance, the study requires a highly specific user profile) or time and resources are limited, how many participants are needed? Is it worthwhile to spend two more weeks on the study just to make it to a total of 10 participants?
The bigger issue: What is the rationale we should use to validate the number of participants for a usability study?
If we go back to the classic model from Nielsen, five users are enough to uncover 85% of usability issues. That has been the UX industry standard’ for a long time, as Jakob Nielsen and his colleagues were among the first UX professionals to calculate the relationship between the number of UX issues uncovered and the number of participants involved. The mathematical model is derived from their years of experience conducting usability studies. Faulkner challenged Nielsen’s model in 2004 with a paper named “Beyond the five-user assumption: Benefits of increased sample sizes in usability testing.” She carefully designed and conducted a few studies with different sample sizes (5, 10, 20, 30, 40, 50, and 60 participants). What she learned from the follow-up data simulation and analysis is that 10 participants are enough to identify at least 82% of the usability issues, whereas a sample size of 15 can help to identify at least 90% of the issues. I even came across a sample size calculator on Jeff Sauro’s Measuring Usability site. Based on the binomial probability formula, it allows you to calculate, for instance, how many users are needed to discover 80% of the usability issues when all issues’ probability of occurrence is above 30%.
All of the above can be used as reference rationales to validate using a certain number of participants for a study. However, as specifically mentioned in Faulkner’s paper, having a highly representative user sample is crucial in uncovering the priority usability issues. Indeed, beyond all those statistical models, getting the right users is sometimes as important as (if not more important than) getting enough users.
Client Question 2: Are we telling the product team that 80% of our customers will fail to use this functionality because 8 out of 10 users failed in the usability study?
Well, the primary purpose of usability studies is to discover qualitative usability issues with an interface, as opposed to predicting the probability of those issues’ occurrence. However, the task completion rate is one of the key metrics we use to evaluate the usability of different UI features, and it is our responsibility to give clients and the product team a clear idea of how to interpret the completion rate.
The confidence level of the results is, again, closely related to the number of users included in the study. From a statistical standpoint, it’s not difficult to understand that the more users in the study, the more confident we can be in the results. However, with only 10 participants, how confident can we say we are in our results?
John Sorflaten has an interesting article discussing this topic. He put forward the limitation of using task success data to predict customer behavior on a larger scale. He recommended using the Adjusted Wald Interval calculator coded by Jeff Sauro to generate the lower and higher bounds of the task success data.
For instance, if 8 out of 10 participants succeed in a task, how could this data be used to predict 1,000 or 10,000 users’ behavior? By using a confidence level of 95% (if you run the same test 100 times, 95 of the times the results will fall within the acceptable +/- margin), Jeff’s calculator generates a lower bound of 48% success and a higher bound of 96% success based on the 80% task success rate from the usability study and accounting for the small sample size. And the same is true if 8 out of 10 participants fail in a task: The calculator predicts a chance of as few as 48% or as many as 96% of users failing the task when the UI is actually released and on the market.
In that sense, as opposed to using the 80% task success rate to predict broader user behavior, we as usability professionals can show the range between 48% and 96% as a reference range for the product manager or marketing team to make further interpretations or decisions.
Next time, when clients are debating between 8 or 10 participants, or the product manager is asking why the task completion rate does not match large-scale user data, these basic stats will help to answer the questions.
At 4 p.m. on a Friday afternoon in October, I am sitting in Beta Cafe with Xiaoyang and Chang, in the heart of Zhongguancun, Beijing.
Xiaoyang and Chang both look like college students to me, but they did sound pretty professional in the e-mail conversations we had before we met. I was introduced to them via Yang, a friend I knew almost 5 years ago through a college student technology competition. Before coming to work for Microsoft in Seattle, Yang used to work for Innovation Works, one of the biggest and earliest IT start-up incubators in Beijing. When I asked about the start-up community in Beijing, Yang shared his old contacts in Innovation Works with me, and they were Xiaoyang and Chang.
After graduating from a top engineering school in 2009, both of them came to Beijing with a passion for becoming IT entrepreneurs and started working for Innovation Works. “It’s not fun to work for a big company. You don’t see immediately how your work will have impact on other people’s life,” one of them proudly declares. In a society where people put stability, high salaries, and big company reputations in front of almost anything else, it’s refreshing and inspiring to see young people making choices like this.
Their paths in their not-too-long IT entrepreneur careers have been slightly different, though. Both of them started at Innovation Works, where they worked with small teams of engineers on original ideas. Chang and his team mainly worked on developing consumer mobile apps, whereas Xiaoyang’s team focused on creating virtual server and cloud computing solutions for overseas game companies. In the two years that have passed, Chang had a few setbacks and just started working on a new mobile app idea. Xiaoyang’s team, on the other hand, has received some funding and moved to a rented office space in Beijing. Neither of them is with the incubator anymore.
“Why is it more difficult to attract funding for consumer apps?” I ask, with an answer already in mind. Chang smiles. “The first question investors would ask if you are developing a consumer app is, ‘What if Tencent steals your idea and makes a similar (if not better) product?’” Tencent, an IT company that “controls” the majority of Chinese netizens via its predominant Instant Messenger QQ, along with the loose intellectual property policy in China, has become the biggest roadblock for B2C start-ups in China. “It’s way easier for Tencent to add a button that links to a new service on their QQ messenger or QQ space than for us to do user acquisition from scratch.” Neither Chang nor Xiaoyang seems optimistic about competing with Tencent. Indeed, if one out of 100 QQ users clicks on a button, that’s already one million users. Not great odds for start-ups — “unless you are invested by a big VC, so you’ll have enough money to do PR and marketing of your product before it gets beaten by Tencent,” one of them says.
“It seems that US users were more supportive of IT start-ups,” Xiaoyang says.
“How so?” I ask.
“It feels to me that US consumers would associate the app and services they use with their identity and values,” he says. “It’s like, it’s a cool thing to try out and use apps that come from start-ups. I don’t think Chinese consumers would be annoyed or even think about the consequences if all of their apps come from the same company.” Chang and Xiaoyang think there’s still a long way to go for Chinese users to become mature software app consumers.
With an inconvenient truth like this, B2B seems to be a safer route. That’s the main reason why Xiaoyang’s team survived and got funding. They are targeting overseas game companies as their consumers, who would pay for their services and are not part of Tencent or other big companies’ interests (at least for now).
“How do you guys think about UX?” I ask. I cannot wait any longer to throw out a question like that due to my UX background. According to Chang and Xiaoyang’s experience, Innovation Works used to have ‘in-house’ UX/UI designers (similar to a designer in residence, or DIR, for incubators here in the Bay Area). However, those designers were soon overwhelmed by the work load given the number of teams and ideas floating around in the incubator. The UX process hasn’t been streamlined very well. Over time, after a few good UI/UX designers got recruited by teams which eventually left Innovation works, the DIR policy was officially abandoned.
“Of course we think UX is important,” they tell me. “It might not have been a few years ago when the majority of Chinese people got started using the Internet. Back then, people thought the more information a website provided the better. But now, you need to make your product look sleek and actually usable. Successful B2C start-ups either have an outstanding designer or some engineers with outstanding design sense.”
“UX consulting is a cool idea — never heard of it before though. Most of the products and services can use some help in that aspect. However, you have to understand that start-ups usually start with very low budget, so you don’t expect to charge them a lot when they first start.”
“There are tons of things on Chinese entrepreneurs’ minds. Product development is something that they can control, but how to get past Tencent or similar big companies and how to protect their intellectual property are not.”
A lot of foreseeable difficulties in doing UX consulting for start-ups in China won’t be too different from the issues we ran into here in the States. Products might be different, user behavior and segmentation might be different, but the possibility of UX improvement with more extensive user research is still there, and the concern of cost is still there, and to what extent your research results could or should change or shape the founding team’s idea is still there.
I changed the topic. “How do you guys think about the VCs in the Chinese market now? Are they approachable? Are there too many of them, or too few?” This seems to be the question that gets the most optimistic answer today: “Angels are mostly coming from local capitals, whereas larger investors are from overseas. It’s not a problem at all to meet with investors in Beijing. There are networking events and lectures, or you could meet them in places like the cafe where we are sitting right now. For early stage start-up teams, they use cafes as their office.”
—
The latte I ordered was not as strong or as aromatic as the ones you could get in Peets or Philz in SF. Not sure where they get their coffee beans from, but I liked the way they drew a nice heart shape with steamed milk on the surface. What’s more, you have more than twenty different latte flavors to choose from, pretty much like those take-out menus you see in Chinese restaurants. The Chinese translation of ‘latte’ and ‘cappuccino’ doesn’t seem to bother young entrepreneurs in Beta cafe at all, as both Chang and Xiaoyang ordered their drinks very quickly. I am sure they’ve been here a lot.
Yes, they (we) were drinking coffee (not tea!), with the lovely wifi in the air, and the loud sound of coffee grinder in the background — not so much different from the scene you’ll see when you walk in a typical Silicon Valley cafe. As I see it, what has already happened and will continue to happen is the whole world adopting this IT start-up culture. The east, the west — we are both “brewing” this start-up culture and the ecosystem around it. It might take some time for the east to catch up (as we’ve been drinking tea for so long), but it won’t be long. It’s the information that we are transmitting and creating via the Internet, not the actual coffee beans, that needs to be physically transported.
And remember that button Tencent could promote to its QQ users. If just 1% of Chinese “tea drinkers” start drinking coffee, that’s already 1 million.
In my last post, I gave some background about startup pitches and spelled out some of the key ingredients in pitch presentations, and I ended with a little peek into one of those presentations by Spool. In this post, I’ll break that presentation down even more to explore why I found it compelling.
Spool is a very promising startup with a great pitch presentation/speech about their products. They don’t have fancy presentation slides, they don’t have very emotional, touching personal stories, and the speaker doesn’t really try to be humorous. However, when I saw him pitch, he managed to tell a logical, convincing, and well-organized story about his product in a genuine and succinct way, and that won my trust.
So here’s a look at how he did it. (I thought the Spool guys handled the Q&A pretty well, too.)
I did a bit of transcription so I could easily see the structure and order of the presentation. Spool had a very simple live product demo, so I put together some of the slides and images from it below. (Some of the images are from their website.) That’s one big thing for startups: If you have a very cool and working product, the live product demo itself is worth more than a thousand words. You need the slides more when you are showing a concept, but if you have a live demo, then just show the demo.
Also, Spool only took a minute or two to explain the technology behind the product. I feel a lot of startups are trying to sell the sophistication of the technology they use — even before going into their idea. The tech side of the product is indeed crucial; however, you need to let your audience buy into your idea first. Get them up to speed with your concept and product, then tell them just a little bit how hard you worked on the technology behind it. That’s when they’ll really be convinced and start to appreciate the technology.
