What is user testing?
User testing is a way of quickly determining whether a product is intuitive and usable for the average person in your target demographic. Discount usability engineering, as pioneered by Jakob Nielsen, is a "quick and dirty" way of testing, so that time and money are saved while being reasonably accurate.
While many marketing departments currently do something similar with focus groups and polling, personal opinions are highly subjective and based on unreliable memories. User testing focuses on observing and questioning the user in their natural environment, or as close as possible, given the constraints of the task. Time-sensitive tasks, such as surgery, compromise by using observation and waiting until afterwards to question users.
Rather than asking the user what they think they might need, user testing observes the user at work or play and evaluates how the product is used in their lives. The stereotypical image of user testing is in a lab with a one-way mirror, but this doesn't reveal the many factors competing for the user's attention in real life. Always try to be as close to the user's natural environment as possible.
What are the benefits?
User testing will make you money, both in terms of saving on technical support costs and in gaining more business through customer referrals. Unfortunately, user testing is one of the first things cut to meet project budgets or deadlines. If present at all, it's usually done at the end of a project as a form of quality assurance. Unfortunately, this is also when it's too late and expensive to make major changes.
Don Norman wrote that most of this mentality derives from a product-focused market, where the common business model in the industry is to churn out products that become obsolete quickly and give rise to newer products. Many of the functions added for a perceived value clutter and weaken the products rather than improve them. User testing seeks to change the market so that it focuses on the user. This will result in better tools for the user's tasks, and allow the user to focus on the job.
User testing is best used in an iterative design phase, as in an evolutionary software development plan. Rapid prototyping is done early in the project to identify the actual needs of the user, and can be done with software or even printouts. The closer the prototype is to the final product, the more accurately user testing will guide design. However, finalized prototypes have the disadvantage that the more a design looks like a finished product, the more users get hung up on aesthetic details such as color or font. With a properly presented rough design, users focus more on interaction, which is what user testing is best suited for observing.
More benefits of user testing can be found at:
http://www.asktog.com/columns/037TestOrElse.html
The user
The first thing to know about user testing is that engineers and designers are not the user. They know the product intimately, and hopefully will eventually also know the task intimately. While they may try to put themselves in the user's shoes they can't unlearn what they already know, and it's impossible to change one's perspective of things. They will miss some details when creating prototypes, but that's what user testing is for. Remember that you're not designing for yourself; you're designing for the user. One usability professor at Carnegie Mellon University said he always hired undergraduate rather than graduate students, because they were more likely to remember what it was like to not understand the course material, and would be better at explaining it.
Next determine who your primary user is. This can be culled from marketing data and varies with the product. Some things such as kitchen appliances also take unexpected users into consideration so that a weekend guest could figure out how to use it at 2am. Others such as an AutoCAD tool for engineers are more specialized. Even in expert-oriented interfaces, an interface designed for novices is beneficial. Studies have found that in times of crisis, experts revert to novice mentality and learn in the moment. Think of jet fighters and nuclear power plants-anything outside of the normal routine or drills are going to be unfamiliar to them, and drills can't possibly cover every scenario anyway.
Also read:
Set Phasers on Stun, And Other True Tales of Design, Technology, and Human Error, by Steven Casey
http://members.aol.com/AegeanPC/AegeanHP.html
The testing environment
The three different settings for user testing are in the lab, in the field, and onsite. Roughly defined, labs are either in-house or at an outside firm, and bring users into an artificial environment. This can have one-way mirrors or cameras, or even try to replicate the user's environment by way of furniture. Onsite means going to the user's environment and is the opposite of lab testing, which is inherently artificial. Working on the street is somewhere in between, involving approaching people and getting rejected, until you get good.
Each approach has benefits and drawbacks. Working in the lab is a controlled environment and allows you to pre-select users based on age or demographic, as appropriate for your target user. However, users are in an artificial situation, not reflecting the complexity of their normal environment. Working onsite is more ideal because it's possible to observe all tasks related to the user's work. However, this takes up more of the user's time and potentially requires more clearance on their end, as well as travel costs on the usability team's side. Doing user testing on the street is free and removes the overhead of scheduling users at the cost/flexibility of selecting them on the fly.
In any case you have selection bias, where the users tested are not representative of the user population or the average user. There's a fair amount of self-selection in the lab, where the users that you or an outside agency recruit are motivated by either the monetary compensation or altruism. In the former case you're paying more than they would make otherwise, so this can skew your data towards users with lower incomes. In the latter case your user is giving their free time to a stranger, which is definitely abnormal. Similarly, testing on the street selects for users loitering with nothing better to do, and testing onsite selects for users with expendable time. Also, attractive people get more slack. There's no real way to design around this; just be aware of possible biases in your research.
The rest of this article will be based on street user testing, which is what I have the most experience in.
Preparation
Based on your market data, go to where your users will spend their free time. I've gone to truck stops at 3am to research thermostats; we were interested in families where members worked irregular or conflicting hours, and how that affected usage. When searching for potential users, it's best to "case out" people. See whether they're doing something involved or if they're in a good mood. This will take longer but will increase the likelihood that someone will agree to help you out. Introduce yourself and tell them you're working on a project and would like their help. Give an honest estimate of how long the test will take, and stick to it. A good way to convince people is to say that you're trying to break the interface, empathizing with the taught helplessness that most people feel about technology. Users are motivated by an opportunity to improve things; everyone's a critic.
Make it very clear that you are doing a user test, not an experiment. You're there to observe their actions and are ultimately testing the interface, not them. Tell the user that if he or she wishes to end the test at any time, they may do so. One critical point is that you tell the user that the interface was designed by another person or group, even if it wasn't. This white lie allows the user to more openly criticize the interface and hopefully get past the bounds of politeness. It also allows you to bond with the user against the design and enhances the impression of working together toward improving the interface.
You should have a list of predefined tasks for the user. These are critical tasks that you're trying to ensure are easy. Asking a user to just mess around with an interface won't be very instructive; they need a clear goal. It's best to provide a pre-made sheet* of the user's tasks. This is to help standardize tests, so that users get the same introduction, and also to give you time to set up. When possible, use artifacts rather than describing them. For example, when testing a copier you might give the user four single-sided sheets of paper and two double-sided stapled sheets of paper, and tell them to make the latter from the former. This keeps you from using leading words and influencing user decisions. A user will zero in on the phrase "two-sided" if it's in the interface and you mentioned it earlier.
*It's been argued that a task list can make users frustrated if they don't complete all of the tasks on the list, but I haven't found this to be the case.
Your interface can be on paper or on screen. Paper is often done before any code is made, and basically users point to objects and you shuffle paper to reflect what the system status should be. Although this is removed from what the actual interface will be like, it's very fast and frees you from the limitations of technology. Also, rough-looking designs are often easier to test with. Users feel more open to criticizing them, and don't get caught up in the minor details of a more polished design, such as font or colors. The major trick here is getting people to "push" buttons on a paper as if their finger were a mouse cursor.
Running with it
The most valuable thing you get from user testing is the user's thought process. You want to find out what they're expecting the interface to do, and thus determine what mental (conceptual) models they have of its inner workings. Whenever their expectations and the actual result don't match up, you have a serious problem. A good way to explain this is to tell the user that you wish you could read minds, but because you can't, you need them to think out loud the entire time they're working. It's common for users to forget to do this, but prod them along if necessary. It may be useful to give an example, so bring a different sort of interface so as not to bias them, and demonstrate. For a digital watch it might go:
Ok, so my task is to set the time an hour ahead. I'm looking for a "set button"...hm, that's the light. This one is the mode, so I need to cycle back to the time mode...ugh. Ah! there it is...now I have to get to the hour position...oh, that's the light again. Oh, there it is. Now, get out of set mode...and I'm done!
As you can see, it's mostly stream-of-consciousness. Take copious notes, because there will be a lot. I tend to scribble on blank paper, so I can use notation and draw arrows between themes.
During the test, be sure to give constant feedback, but kindly decline to help users get through a task. Remember, you're trying to learn from them. A common question is "What does this do?" It's best to just turn the question around and ask, "What do you think it will do?" This gets back to understanding their thoughts and expectations of the product. If they're finding many problems and getting frustrated, be supportive with comments such as "Yeah, lots of users have been getting stuck on that." This helps them to not feel stupid, which is very important because you want them to have fun. If they get critically stuck on a part for a long time, you can let them go and skip to the next part.
When you're done, you should immediately flesh out your notes and add details about physical reactions and anything else you noticed. After about a day the memories won't be as fresh. Compile the major findings across the different users (about 5 is enough) and prepare design changes based on your work.
In special cases you sometimes have the opportunity to work with two users. If they have a healthy relationship (ie, aren't going through a divorce or otherwise hate each other) and can work together, essentially all you have to do is eavesdrop while they build a shared conceptual model.
In summary
Learning the basic forms of conducting user testing can take a matter of weeks. However, perfecting the art of working with people and building rapport without leading them can take months. Like most things it just takes practice and the right mindset. It's a bit like being a salesman when approaching users, but then like being an objective spectator during the user test.
Questions or corrections? Email byeung@alumni.cmu.edu
|