We’re just hours from Facebook’s IPO hitting the stock market, and whatever happens, it’s likely to make waves in Silicon Valley and across the United States. But Facebook’s 900 million users might not be aware of a key part of the business or marketplace side of Facebook: how the company profits from advertisers.
I thought of Facebook only as a social networking site until a few months ago, when I started trying to build a Facebook page for my blog. All of a sudden, as opposed to being a typical end user, I became a user who might be a business owner or an advertising or PR representative for a company or organization. To those users, Facebook can be a “marketplace” to attract and engage with more customers. With help from a friend who works on Facebook’s ads team, I was able to get some ads credit to try out the process of creating, modifying, and eventually evaluating my ad strategies.
And, of course, wherever there’s a user, there’s a user experience!
After spending one hour going through the process of building my first Facebook ads, here are some interesting thoughts from a UX perspective:
Instead of going through one unified portal for building an ad, users can start from multiple places: Facebook Ads, Facebook for Business, Facebook Pages, or of course your own page that you would like to use ads to promote. Each of these portals seems to have an independent information architecture. If the user is patient enough to navigate around, he or she should be able to find all of the information needed to get started, especially the button labeled “create an ad.” However, the whole experience is still very fragmented. It almost seems to me that multiple teams within Facebook are managing this process at the same time — and they probably don’t talk to each other too much.
Have a unified feel and flow if you are trying to get the same users to do the same thing — which, in this case, is to create an ad on Facebook.
The concept of Facebook Pages kept popping up when I was trying to read tutorials on Facebook ads. It took me a while to understand that having a Facebook page is the precondition to having an ad on Facebook. All ads have to be associated with a page. However, this idea got buried in articles describing how you should “target your user,” “group your business,” or “track your ads.” With a UX background, I’m always looking first for the user flow in any system — even a system as simple as an informational website. As a novice user, in this case, I had no idea how to start a Facebook ad from scratch. I read things like “build your presence with a page” or “talk to your consumers using your page,” but nowhere did I see something like: “In order to create an ad, you need a Facebook page for your business as a first step.”
Trying to educate your users about too much at one time usually overwhelms them. In this case, “how to create an ad” and “how to create an effective ad” are for users at different stages. Mixing basic flows with all the other information and strategies you would like your users to learn will likely confuse them. So assume your users know nothing about the platform. Hold their hands, step-by-step, for the basics before teaching them tricks. A PR person from Starbucks probably doesn’t need to review this basic knowledge, but the owner of the cafe around the corner who wants to advertise his little shop on Facebook would need the concrete steps.
This point is actually a positive one! Creating an ad is by nature more complicated than creating an event on Facebook, as there are more details and customizations involved. Instead of setting a goal for a service like, “Create a system so that users won’t make mistakes,” why not have a more realistic goal, such as “Design a system that has high ‘recoverability’ so users know what to do when they have questions or make mistakes”? Facebook’s neat “question mark” feature was a great help when I was working through the flow.
For systems with fewer details or shorter learning curves, using a click or hover-over effect to hide extra information when it’s not being actively requested is an unobtrusive way to have help ready whenever users need it.
You would expect Silicon Valley companies like Facebook to come up with products that are unified, user-friendly, and well-supported, but when it comes to business users, that might not be the case. Similar to some notorious enterprise software, Facebook still requires users who would like to post ads to download and sift through pages of start guide PDFs, and trying to figure out the step-by-step flow for building an ad for the first time can be a struggle. If Facebook ultimately wants to let every small shop owner quickly get the concept of Facebook ads and build ads for their business (without shelling out for an online ads consultant), there’s still room for Facebook to improve.
The EchoUser research team had quite a busy December. Our schedules were filled with recruiting users, drafting test plans, moderating usability sessions, writing reports, and, last but not least, arranging check-in meetings with clients throughout the project cycle.
Clients — regardless of their UX background — would raise questions and concerns about UX methodology in those meetings to make sure that their studies were on the right track and that they would get valuable and defensible data from the projects.
In the two usability projects I am on (both benchmark studies), I came across the following two interesting questions from our clients. Though the two questions seemed to have come from two different angles, they both point to one of the key issues in doing usability studies: how to interpret usability data with a small number of users. I thought I’d share the two client questions and hope to elicit some extended discussions here.
Client Question 1: How many participants is enough for a benchmark usability study? Eight, 10, or 12?
A lot of times, the question actually becomes, “Do we need a single-digit participant number or a double-digit one?” Clients want the usability study results to be defensible both from a statistical and a PR standpoint. When time and resources allow and it’s easy to recruit target participants, the question of “Should we get two more participants for the study?” has an easy solution: Let’s just do two more sessions. However, in a scenario in which qualified participants are very difficult to find or recruit (for instance, the study requires a highly specific user profile) or time and resources are limited, how many participants are needed? Is it worthwhile to spend two more weeks on the study just to make it to a total of 10 participants?
The bigger issue: What is the rationale we should use to validate the number of participants for a usability study?
If we go back to the classic model from Nielsen, five users are enough to uncover 85% of usability issues. That has been the UX industry standard’ for a long time, as Jakob Nielsen and his colleagues were among the first UX professionals to calculate the relationship between the number of UX issues uncovered and the number of participants involved. The mathematical model is derived from their years of experience conducting usability studies. Faulkner challenged Nielsen’s model in 2004 with a paper named “Beyond the five-user assumption: Benefits of increased sample sizes in usability testing.” She carefully designed and conducted a few studies with different sample sizes (5, 10, 20, 30, 40, 50, and 60 participants). What she learned from the follow-up data simulation and analysis is that 10 participants are enough to identify at least 82% of the usability issues, whereas a sample size of 15 can help to identify at least 90% of the issues. I even came across a sample size calculator on Jeff Sauro’s Measuring Usability site. Based on the binomial probability formula, it allows you to calculate, for instance, how many users are needed to discover 80% of the usability issues when all issues’ probability of occurrence is above 30%.
All of the above can be used as reference rationales to validate using a certain number of participants for a study. However, as specifically mentioned in Faulkner’s paper, having a highly representative user sample is crucial in uncovering the priority usability issues. Indeed, beyond all those statistical models, getting the right users is sometimes as important as (if not more important than) getting enough users.
Client Question 2: Are we telling the product team that 80% of our customers will fail to use this functionality because 8 out of 10 users failed in the usability study?
Well, the primary purpose of usability studies is to discover qualitative usability issues with an interface, as opposed to predicting the probability of those issues’ occurrence. However, the task completion rate is one of the key metrics we use to evaluate the usability of different UI features, and it is our responsibility to give clients and the product team a clear idea of how to interpret the completion rate.
The confidence level of the results is, again, closely related to the number of users included in the study. From a statistical standpoint, it’s not difficult to understand that the more users in the study, the more confident we can be in the results. However, with only 10 participants, how confident can we say we are in our results?
John Sorflaten has an interesting article discussing this topic. He put forward the limitation of using task success data to predict customer behavior on a larger scale. He recommended using the Adjusted Wald Interval calculator coded by Jeff Sauro to generate the lower and higher bounds of the task success data.
For instance, if 8 out of 10 participants succeed in a task, how could this data be used to predict 1,000 or 10,000 users’ behavior? By using a confidence level of 95% (if you run the same test 100 times, 95 of the times the results will fall within the acceptable +/- margin), Jeff’s calculator generates a lower bound of 48% success and a higher bound of 96% success based on the 80% task success rate from the usability study and accounting for the small sample size. And the same is true if 8 out of 10 participants fail in a task: The calculator predicts a chance of as few as 48% or as many as 96% of users failing the task when the UI is actually released and on the market.
In that sense, as opposed to using the 80% task success rate to predict broader user behavior, we as usability professionals can show the range between 48% and 96% as a reference range for the product manager or marketing team to make further interpretations or decisions.
Next time, when clients are debating between 8 or 10 participants, or the product manager is asking why the task completion rate does not match large-scale user data, these basic stats will help to answer the questions.
At 4 p.m. on a Friday afternoon in October, I am sitting in Beta Cafe with Xiaoyang and Chang, in the heart of Zhongguancun, Beijing.
Xiaoyang and Chang both look like college students to me, but they did sound pretty professional in the e-mail conversations we had before we met. I was introduced to them via Yang, a friend I knew almost 5 years ago through a college student technology competition. Before coming to work for Microsoft in Seattle, Yang used to work for Innovation Works, one of the biggest and earliest IT start-up incubators in Beijing. When I asked about the start-up community in Beijing, Yang shared his old contacts in Innovation Works with me, and they were Xiaoyang and Chang.
After graduating from a top engineering school in 2009, both of them came to Beijing with a passion for becoming IT entrepreneurs and started working for Innovation Works. “It’s not fun to work for a big company. You don’t see immediately how your work will have impact on other people’s life,” one of them proudly declares. In a society where people put stability, high salaries, and big company reputations in front of almost anything else, it’s refreshing and inspiring to see young people making choices like this.
Their paths in their not-too-long IT entrepreneur careers have been slightly different, though. Both of them started at Innovation Works, where they worked with small teams of engineers on original ideas. Chang and his team mainly worked on developing consumer mobile apps, whereas Xiaoyang’s team focused on creating virtual server and cloud computing solutions for overseas game companies. In the two years that have passed, Chang had a few setbacks and just started working on a new mobile app idea. Xiaoyang’s team, on the other hand, has received some funding and moved to a rented office space in Beijing. Neither of them is with the incubator anymore.
“Why is it more difficult to attract funding for consumer apps?” I ask, with an answer already in mind. Chang smiles. “The first question investors would ask if you are developing a consumer app is, ‘What if Tencent steals your idea and makes a similar (if not better) product?’” Tencent, an IT company that “controls” the majority of Chinese netizens via its predominant Instant Messenger QQ, along with the loose intellectual property policy in China, has become the biggest roadblock for B2C start-ups in China. “It’s way easier for Tencent to add a button that links to a new service on their QQ messenger or QQ space than for us to do user acquisition from scratch.” Neither Chang nor Xiaoyang seems optimistic about competing with Tencent. Indeed, if one out of 100 QQ users clicks on a button, that’s already one million users. Not great odds for start-ups — “unless you are invested by a big VC, so you’ll have enough money to do PR and marketing of your product before it gets beaten by Tencent,” one of them says.
“It seems that US users were more supportive of IT start-ups,” Xiaoyang says.
“How so?” I ask.
“It feels to me that US consumers would associate the app and services they use with their identity and values,” he says. “It’s like, it’s a cool thing to try out and use apps that come from start-ups. I don’t think Chinese consumers would be annoyed or even think about the consequences if all of their apps come from the same company.” Chang and Xiaoyang think there’s still a long way to go for Chinese users to become mature software app consumers.
With an inconvenient truth like this, B2B seems to be a safer route. That’s the main reason why Xiaoyang’s team survived and got funding. They are targeting overseas game companies as their consumers, who would pay for their services and are not part of Tencent or other big companies’ interests (at least for now).
“How do you guys think about UX?” I ask. I cannot wait any longer to throw out a question like that due to my UX background. According to Chang and Xiaoyang’s experience, Innovation Works used to have ‘in-house’ UX/UI designers (similar to a designer in residence, or DIR, for incubators here in the Bay Area). However, those designers were soon overwhelmed by the work load given the number of teams and ideas floating around in the incubator. The UX process hasn’t been streamlined very well. Over time, after a few good UI/UX designers got recruited by teams which eventually left Innovation works, the DIR policy was officially abandoned.
“Of course we think UX is important,” they tell me. “It might not have been a few years ago when the majority of Chinese people got started using the Internet. Back then, people thought the more information a website provided the better. But now, you need to make your product look sleek and actually usable. Successful B2C start-ups either have an outstanding designer or some engineers with outstanding design sense.”
“UX consulting is a cool idea — never heard of it before though. Most of the products and services can use some help in that aspect. However, you have to understand that start-ups usually start with very low budget, so you don’t expect to charge them a lot when they first start.”
“There are tons of things on Chinese entrepreneurs’ minds. Product development is something that they can control, but how to get past Tencent or similar big companies and how to protect their intellectual property are not.”
A lot of foreseeable difficulties in doing UX consulting for start-ups in China won’t be too different from the issues we ran into here in the States. Products might be different, user behavior and segmentation might be different, but the possibility of UX improvement with more extensive user research is still there, and the concern of cost is still there, and to what extent your research results could or should change or shape the founding team’s idea is still there.
I changed the topic. “How do you guys think about the VCs in the Chinese market now? Are they approachable? Are there too many of them, or too few?” This seems to be the question that gets the most optimistic answer today: “Angels are mostly coming from local capitals, whereas larger investors are from overseas. It’s not a problem at all to meet with investors in Beijing. There are networking events and lectures, or you could meet them in places like the cafe where we are sitting right now. For early stage start-up teams, they use cafes as their office.”
—
The latte I ordered was not as strong or as aromatic as the ones you could get in Peets or Philz in SF. Not sure where they get their coffee beans from, but I liked the way they drew a nice heart shape with steamed milk on the surface. What’s more, you have more than twenty different latte flavors to choose from, pretty much like those take-out menus you see in Chinese restaurants. The Chinese translation of ‘latte’ and ‘cappuccino’ doesn’t seem to bother young entrepreneurs in Beta cafe at all, as both Chang and Xiaoyang ordered their drinks very quickly. I am sure they’ve been here a lot.
Yes, they (we) were drinking coffee (not tea!), with the lovely wifi in the air, and the loud sound of coffee grinder in the background — not so much different from the scene you’ll see when you walk in a typical Silicon Valley cafe. As I see it, what has already happened and will continue to happen is the whole world adopting this IT start-up culture. The east, the west — we are both “brewing” this start-up culture and the ecosystem around it. It might take some time for the east to catch up (as we’ve been drinking tea for so long), but it won’t be long. It’s the information that we are transmitting and creating via the Internet, not the actual coffee beans, that needs to be physically transported.
And remember that button Tencent could promote to its QQ users. If just 1% of Chinese “tea drinkers” start drinking coffee, that’s already 1 million.
