What’s the Research Team up to?

The research team recently got together for a round table meeting to get our best ideas, best insights and best practices documented and into production.  Our main goal was to start defining processes and best practices for several topics that relate to our work- ultimately, we aimed to work smarter, not harder.

Some topics discussed last week:

• Kickoff meetings: Objectives with internal team vs. objectives with clients
• Recruiting: Better ways to approach the compensation process for usability test participants
• Metrics: What’s best to include for Client X vs. Client Y
• Client profile: How to keep track of our wide range of clients better

Some ways to approach our best practices documentation:

• Don’t write a book for deep exploration; instead, make bulleted lists or checklists
• Build on existing documents- these are living documents meant to change as we learn or discover new practices and new tools
• This will be in outline or ‘cheat sheet’ fashion- a quick, easy-to-glance and handy document to assist new people and remind veterans who are coming back to the activity

Collaboration is one of the research team’s greatest strengths and we’re dedicated to keep building on our processes and make great work even smarter.  Stay tuned for what’s coming next with our best practices project.

Thinking is synthesis, careful consideration, cost-benefit analysis, asking and researching.

Acting is speaking out, communicating, externally doing.

As designers it’s our job to find the right balance of thinking and acting, sometimes as a counterbalance to what’s going on around us (i.e. someone who speaks without thinking). It’s a tightrope walk, and every challenge involves a different mix of the two. For example, right now I’m in the midst of thinking and researching with very little acting – but I know that in a month’s time I’ll be mostly acting, with spurts of thinking in between.

What’s your ideal mix of thinking and acting?

Visual.ly dazzles the eyes with 20 great visualizations from 2011, from the serious to the outright silly.

Oh hi, pretty new Jet Blue redesign. It’s slick and a little Apple-y, no?

After its Qwikster stumble, Netflix is rebounding with a move into original programming. Fast Company’s Co.Create sat down with Netflix’s head of content to learn more about how that’s going to work.

We most certainly hope you’re not staring down a weekend full of bad dates … but if you are, Wot Went Wrong wants to help you give and get good feedback on why that coffee meeting didn’t turn into anything more. Wah-waaaahhhh.

If only we were in New York, we would have loved to see this talk on how “digital forensics” — aka, rescuing old drafts and archived versions of documents — uncovered some of the many changes to what ultimately became the musical RENT.

A 5-year-old analyzes logos. Spoiler alert: apparently a lot of companies are cheetahs.

Still trying to figure out what Pinterest is all about? We gawked at this infographic that spells out some key facts about the “social pinboard” in a pretty way.

On those slickly designed corporate “visions of the future” videos — why they capture our imagination with glitzy production, why they get so much traction, and what they’re missing (e.g., depth; informed projections of the future).

There are plenty of tools that let people track the nutrition values of specific foods, from a cup of cereal to a frozen dinner entree. But it’s tougher to find something that does what I want to do: enter all of the ingredients in my recipe, say how many servings it made, and find out what exactly I just ate.

Then a friend told me about MyPlate, an offering from Livestrong.com that lets you build your own recipes and spits out a customized label with all the standard nutrition facts. Cool! … if it worked, anyway. So I made an account, headed to the My Recipes tab, and started typing in this week’s dinners.

MyPlate did work, more or less — and better than I expected, anyway. I now have a little database of some of the meals I made this week, complete with estimates of their nutrition facts. But it felt a little weird to adapt my flexible (or, more accurately, flying-by-the-seat-of-my-pants) cooking process to the specificity of the information MyPlate needed.

Adding foods to a recipe is a bit annoying, but nothing I haven’t seen on similar tools before. In recording a stir-fry recipe, I had this list to choose from when adding broccoli:

Chopped broccoli vs. “stalk broccoli,” cups vs. grams — yikes. I’d just cut up whatever amount of broccoli had been sitting in my crisper, so I chose the most generic broccoli I could find in the list, took a guess on quantity, and moved on.

The funniest part came when I got to the last field:

Servings? Hmm, I cooked the stir-fry on Sunday and ate the leftovers for a few days. Counting up all the meals, I guessed it had made 7 servings. So far, so good. Yield, though? I have no idea! It made…a stir-fry? A big stir-fry? A lot of stir-fry? Absent any actual measurements, what was I supposed to type?

I went with, “a lot of stir-fry.” This field, it turns out, is what the nutrition label uses to tell you how big a serving is — “12 crackers,” or “1/4 box,” or what have you. So here’s the label I got back for this meal:

Serving size: “1/7 a lot of stir-fry.” Heh.

The experience of cooking can be a tricky one to capture. There’s a bit of push and pull here, I think — between wanting to get exact measurements and quantities and wanting to intuitively make food that tastes the way you want it to taste. I’m far from an expert cook, but I’m trying to get better at making up my own recipes — and sometimes that means I end up with a dash of this, a pinch of that, and absolutely no idea how much soy sauce I poured into the wok.

MyPlate is more flexible for cooks like me than anything I’ve come across before, as long as I’m OK with some guesstimating and some funny results (another one of my recipes has as its serving size “1/8 giant pot of tomato sauce”). But I do wonder: Are there some experiences even more free-form than cooking a dinner from scratch that are just immune to quantification?

A smattering of EchoUser office reactions to this plush desk nap pod:

“How do you breathe?

“It looks cozy … until you suffocate”

“It’s an ostrich-bag-thingy.”

“It’s a fuzzy deep-sea-diving helmet.”

This color-matching game is just as addictive (if not more so) than the kerning game from the same folks that we loved a while back — but it feels much more stressful!

Fine, fine, it’s not all fun and games around here. We learned a lot this week from this Uday Gajendar presentation on how to partner with a UI designer.

Clients — regardless of their UX background — would raise questions and concerns about UX methodology in those meetings to make sure that their studies were on the right track and that they would get valuable and defensible data from the projects.

In the two usability projects I am on (both benchmark studies), I came across the following two interesting questions from our clients. Though the two questions seemed to have come from two different angles, they both point to one of the key issues in doing usability studies: how to interpret usability data with a small number of users. I thought I’d share the two client questions and hope to elicit some extended discussions here.

Client Question 1: How many participants is enough for a benchmark usability study? Eight, 10, or 12?

A lot of times, the question actually becomes, “Do we need a single-digit participant number or a double-digit one?” Clients want the usability study results to be defensible both from a statistical and a PR standpoint. When time and resources allow and it’s easy to recruit target participants, the question of “Should we get two more participants for the study?” has an easy solution: Let’s just do two more sessions. However, in a scenario in which qualified participants are very difficult to find or recruit (for instance, the study requires a highly specific user profile) or time and resources are limited, how many participants are needed? Is it worthwhile to spend two more weeks on the study just to make it to a total of 10 participants?

The bigger issue: What is the rationale we should use to validate the number of participants for a usability study?

If we go back to the classic model from Nielsen, five users are enough to uncover 85% of usability issues. That has been the UX industry standard’ for a long time, as Jakob Nielsen and his colleagues were among the first UX professionals to calculate the relationship between the number of UX issues uncovered and the number of participants involved. The mathematical model is derived from their years of experience conducting usability studies. Faulkner challenged Nielsen’s model in 2004 with a paper named “Beyond the five-user assumption: Benefits of increased sample sizes in usability testing.” She carefully designed and conducted a few studies with different sample sizes (5, 10, 20, 30, 40, 50, and 60 participants). What she learned from the follow-up data simulation and analysis is that 10 participants are enough to identify at least 82% of the usability issues, whereas a sample size of 15 can help to identify at least 90% of the issues. I even came across a sample size calculator on Jeff Sauro’s Measuring Usability site. Based on the binomial probability formula, it allows you to calculate, for instance, how many users are needed to discover 80% of the usability issues when all issues’ probability of occurrence is above 30%.

All of the above can be used as reference rationales to validate using a certain number of participants for a study. However, as specifically mentioned in Faulkner’s paper, having a highly representative user sample is crucial in uncovering the priority usability issues. Indeed, beyond all those statistical models, getting the right users is sometimes as important as (if not more important than) getting enough users.

Client Question 2: Are we telling the product team that 80% of our customers will fail to use this functionality because 8 out of 10 users failed in the usability study?

Well, the primary purpose of usability studies is to discover qualitative usability issues with an interface, as opposed to predicting the probability of those issues’ occurrence. However, the task completion rate is one of the key metrics we use to evaluate the usability of different UI features, and it is our responsibility to give clients and the product team a clear idea of how to interpret the completion rate.

The confidence level of the results is, again, closely related to the number of users included in the study. From a statistical standpoint, it’s not difficult to understand that the more users in the study, the more confident we can be in the results. However, with only 10 participants, how confident can we say we are in our results?

John Sorflaten has an interesting article discussing this topic. He put forward the limitation of using task success data to predict customer behavior on a larger scale. He recommended using the Adjusted Wald Interval calculator coded by Jeff Sauro to generate the lower and higher bounds of the task success data.

For instance, if 8 out of 10 participants succeed in a task, how could this data be used to predict 1,000 or 10,000 users’ behavior? By using a confidence level of 95% (if you run the same test 100 times, 95 of the times the results will fall within the acceptable +/- margin), Jeff’s calculator generates a lower bound of 48% success and a higher bound of 96% success based on the 80% task success rate from the usability study and accounting for the small sample size. And the same is true if 8 out of 10 participants fail in a task: The calculator predicts a chance of as few as 48% or as many as 96% of users failing the task when the UI is actually released and on the market.

In that sense, as opposed to using the 80% task success rate to predict broader user behavior, we as usability professionals can show the range between 48% and 96% as a reference range for the product manager or marketing team to make further interpretations or decisions.

Next time, when clients are debating between 8 or 10 participants, or the product manager is asking why the task completion rate does not match large-scale user data, these basic stats will help to answer the questions.

Wondering what the folks at @echouser make of Jakob’s famous heuristics if applied to enterprise apps…

— ultan (@ultan) December 29, 2011

“Jakob’s famous heuristics” refers to the 10 rules of thumb that Jakob Nielsen has developed and promoted which commonly form the basis for the discount usability method of heuristic evaluation.

In my view, the short answer is the guidelines can be very helpful, each of them has something to offer when building or evaluating enterprise applications, BUT, and this is a big but, in the domain of enterprise software it can be harder to adhere to them compared to a consumer setting. I was talking with my colleagues about this, and @kimretta crystallized what I was trying to express nicely: in the enterprise setting, instead of just having to design for an interaction between a system and a user, the interaction is between a user, a system and a business.

All too often the system and the business get together first, and the end-user doesn’t get much time or attention. Currently, and I will overgeneralize a bit here, the development and deployment process of much enterprise software doesn’t include a person focused and empowered to champion user centered design. The developer is not the user, the purchaser is not the user, and the people who implement and support the IT infrastructure are not the user. Software gets developed to meet business needs, not user needs. And perhaps I’m revealing my social science perspective here but I think it is critical to realize that in some cases, user centered design changes in enterprise software require organizational change in the enterprise, something that is difficult in the best of times.

What we see in our practice however is that neglected user needs in the enterprise come back to haunt the business, through lost productivity and reduced worker satisfaction.

What’s up with Enterprise software?

It is easy to find critiques of enterprise software (I enjoyed this entry on the topic).  Common refrains for why enterprise software typically offers a substandard user experience include: the disconnect between those who purchase and those who use, legacy lock-in, a ‘more-is-better’ mind-set about adding features, and the way enterprise software is first and foremost aligned to business rules, at times with a seemingly blatant disregard for the humans who need to interact with them.

For each heuristic that Jakob calls out,  I could tell a story about a piece of software where I’ve personally seen the negative effects on users caused by disregarding that heuristic. Sometimes the negative effects are small, mitigated by ‘software calluses’ users develop as they learn the idiosyncrasies of a particular system. But sometimes the effects are large, resulting in users looking for any way possible to avoid using the software, reducing productivity, efficiency and morale. Some of the 10 may not apply as much as others in particular contexts, and we definitely see niche- and expert-user audiences able to adapt to systems that have less polish and fewer affordances for new users in spite of things we might otherwise consider problematic.

For example, take a heuristic like “Match between system and the real world” which I frequently see enterprise software struggle with. Remember the three parts I mentioned earlier, system, business and user? Which real world are we to now design for? Inevitably, employees understanding of the ‘real world’ of the business (perhaps incomplete, based on role) is different than the employees’ ‘real world’ outside of work (which may be influencing their expectations about how interfaces should behave). Also the ‘real world’ of each business unit may be different in non-trivial ways, even for people using the same software at the same company, not to mention between companies. Building and customizing software in this environment is a different challenge from crafting a more singular consumer offering.

Browser UX

One trend that is relevant here is the move to browser-based interfaces.  This is not new of course, but over the last decade or two the software frame that one uses to access enterprise tools is increasingly the same one used for personal activities. As a result I have seen expectations about the enterprise tools overlap higher-quality consumer level offerings. If someone sees a particularly helpful feature while using other sites on the web, they will ask about their enterprise app: “Why can’t my work program do that?”

Enterprise = just plain tough?

The current state of enterprise user experience seems to be improving, however there is a long way to go, and Jakob’s heuristics continue to be relegated to the “nice-to-have” category, to end-users detriment.

This is not to say that Jakob’s heuristics are a universal or final answer, because they are not. As enterprises pursue native app development on mobile devices, tablets, or on our Minority Report platforms of the future, there will always be the need to look at a specific context  to see what makes sense. But just because integrating general, non-enterprise-specific heuristics is hard in the enterprise environment doesn’t mean it should be disregarded. Making enterprise software more usable by consumer standards will create benefits in the longer term, with more efficient workplaces and less time accommodating systems that offer a poor fit to the work that needs to get done.

Yeah, we love our smartphones, but…under water? These iPhone goggles made a — wait for it — splash at CES. Can you still get e-mail 20,000 leagues below the sea?

Every bad presentation we’ve ever had to attend feels exactly like this: Every Presentation Ever

Pretty food packaging design projects, packaged in what is itself a pretty food packaging design project — is it lunchtime yet?

And finally: Lionel Richie’s “Hello” in movie dialogue. Well played, internet.

I did a quick round of visits to local stores.

I found a 32” Sony I liked at a good price and a professional salesperson.

The following Sunday I went back to make my purchase.

The salesperson was not there.

While waiting for another, I overhear the manager trying to convince a caller to buy last year’s Sony 57” model for the same price I was paying for the 32”.

I approach him and commit to buying it. He finds the remote for it, which is quite large.

It does not turn on the TV. “But that is easy to solve,” he says. I’ll get the manual.

The only manual for the model is in Spanish.

“That’s easy to solve.” Take the remote with you and take the batteries out. That will set it back to default values. When the TV is delivered, put them back in.

The TV arrives a week later. My wife is miffed about rearranging the furniture to fit it in.

I put the batteries in. The remote does not turn on the TV. I call the manager. “That’s easy to solve. Come to the store and I will give you one of our new programmable remotes, no charge.”

I program the new remote and turn on the TV. Looks great. I press the PIP (picture-in-picture) button. A small PIP window appears, showing the same channel as the big picture because I have cable TV. Also a second press gets a split-screen PIP; a third gets 16 images of the same channel. A fourth press goes back to the screen with the small PIP window. I can’t turn off PIP.

The old remote and TV had an extra function, a key labeled “PIP OFF.” The new TV does not have or recognize that function.

The manager is confused, offers to come to my house (he doesn’t really believe me).

Manager comes and can’t turn PIP off. Suggests I call their customer service, available to all customers for the first 30 days.

Call customer service and explain the problem. “That’s easy to solve. I will write the issue on the work order and send one of my people. But can’t schedule you for at least 10 days.”

Technician arrives, looks about 18. But enthusiastic despite having no description of the issue on the work order.

“I have never seen that new remote before. Do you have the manual?” Spends 20 minutes looking through it. Can’t turn off PIP.

“This is easy to solve. We have a guy at the shop who used to work for Sony. I will call him.” Goes out to his truck.

20 minutes later, comes back. “He showed me how to set the TV back to the default settings.” He proceeds to press and hold several buttons. PIP disappears.

“It’s easy to solve. All you have to do is use this sequence to get the defaults back.”

“But I also lost all of my personalizations, like which channels are active, which are my favorites, etc. I am not going to go through that every time someone accidentally presses the PIP button.”

“Humm”

I get an insight. “How about if we program the PIP key to do something else, like channel up. That way there will be no harm if someone pressed it by accident. The PIP function is useless anyway with cable.”

We program the button. Success. “That was easy to solve.”