Ways to test usability

The kind of test that you use to measure the usability of something depends on what that thing is and where it is in the development cycle. For example, you would perform a different kind of usability test (a paper prototype test) for a product that was still in the “design” stage, compared to the usability test (formal test) for a product that was already complete.

Dumas and Redish list a structured approach to usability testing in their book, A Practical Guide to Usability Testing (1994) but I like the list presented by Alice Preston in a January, 2004, article in the Society for Technical Communication (STC) about the types of usability tests you can do. Her list (from Vol. 10, No. 3 of STC) is no longer online, but I would summarize it this way:

Preston suggested a range of different tests and techniques to evaluate usability, from user interviews to focus groups to formal usability tests.

Questionnaires: The questionnaire is a very basic kind of usability test that surveys users about a product or design. The questionnaire method is quite flexible and can be applied to products in various stages of development, from the “design” phase to mature products.

Interviews and observations: Using this method, you can interview individual users about how they use a product to accomplish real tasks. Later, you can observe these users while they use the product and document how they actually interact with the product. This two-step approach provides additional insight about what users think they do and what they really do.

Heuristic review: A heuristic review is a formal term that describes an expert’s opinion of a product’s usability. The heuristic evaluation might be based on simple review, or on criteria such as Affordances.

Focus group: Interview a group of users about a product. The focus group provides feedback about how users might use a product, but well before the product actually exists. The focus group can also indicate features that might be useful in a final version of the product.

Group reviews and walkthroughs: In this method, you can present a possible design to a group of users, who then comment on it. This is essentially a focus group where users can provide feedback about what things they like about the design and what they don’t like, as well as describing how such a product might be useful to do real work.

Walk-around reviews: This method is similar to the group review. Post several versions or iterations of a design concept in a room and ask a group of real users to walk around the room and comment on each design.

Do-it-yourself walkthrough: Similar to the walk-around review, you might simulate using the design by walking through a prototype or design concept, pretending to accomplish real tasks. This is essentially a paper prototype test that doesn’t involve any users.

Paper prototype test: Print all the user interface mockups, including menus and screens that a user might encounter when using the system. Ask the tester to simulate real tasks by prompting them with scenario tasks.

Prototype tests: If you have an early version of the product, ask real users to simulate doing work with it. This is a prototype test instead of a formal usability test because such a product may not be in a working state, but only the mockup is functioning.

Formal usability test: Invite a group of users to test a product. This might be the actual product that is undergoing a change to the design, or a new product that is in a working or mostly-working state. Present each tester with a series of scenario tasks, one at a time, and ask the tester to complete those real tasks with the system. Observe what they do and what they say, and how long it took them to complete each task (and if they were able to complete it).

The formal usability test is what most people think of when they hear “usability test.”

Controlled experiment: Invite a group of people to test two versions of a product or two versions of a design, using realistic scenarios and tasks. These tests usually provide controls and statistical balancing to generate a detailed comparison of the two products. This is also called an “A-B” test, because it involves testing and comparing two variations of a design.

Fit the method to the project

Usability testing is not a “one size fits all” idea. In any development product, for example, the product stakeholders must sponsor usability testing throughout the process. Leverage the different methods to evaluate usability testing, and apply the method that is most likely to generate useful results. In the early phases of a product’s development, focus groups and prototype tests may provide the best results. Later in the product life cycle, formal usability tests and controlled A-B tests might be more appropriate.