programmer Usability testing for developers

Learn the basics of usability with this open source presentation

I first got started with usability testing during my Master’s program in technical communication, many years ago. In studying usability, I was fortunate to work on an independent study where I designed, performed, analyzed, and reported on a usability test for an outside client. Because I also have a long history in open source software, I connected with open source projects, and partnered with the GNOME Project to perform a series of usability tests on the GNOME desktop.

After earning my Master’s degree, I continued working in usability testing of open source software. In addition to my own work, I often mentored others in usability testing, including with the Outreachy internship program, which provides paid internship opportunities to women and others who are underrepresented in open source.

I also wrote articles about usability in open source software; inspired from the central theme of my Master’s capstone. And whenever I had the opportunity, I presented at conferences.

In 2021, I spoke at GUADEC (formerly the GNOME Users And Developers European Conference) about how to make your apps easy to use: Usability testing the simple way. GUADEC shared the video from the conference via YouTube; you can find my presentation at about 3:33:00 in the video:

I start with the question “what is usability testing?” You can find more formal definitions about usability, but my personal “shorthand” is can real people do real tasks in a reasonable amount of time. Usability is subtly different from user experience, although the two are related. UX is more about a user’s emotional response to using a product; usability measures how easily the user can accomplish tasks. Typically, a product with strong usability also has positive UX, and systems with poor usability usually have a negative UX, but that’s not always the case.

The rest of the presentation focuses on how to do usability testing in several steps: writing personas and use scenarios, crafting scenario tasks, performing the test, and analyzing the results. I give several examples of personas, and actual scenario tasks that we used in previous usability tests with GNOME. For example:

[Calendar] A client has asked to meet with you at their office next week. Add an event called Meeting with client to the calendar for next week on Thursday at 10:00.

Note that each scenario task sets a brief context then asks the tester to do something specific. But the scenario task needs to find a balance so that it provides enough specificity so the tester can do the task, but not too much that the task effectively becomes a to-do list.

I recommend at least five testers in a usability test, with the caveat that usability testing is not “one and done.” You should do several iterations of usability testing, where five testers give you good enough data that you can make informed changes to the design. After a few iterations of testing and improvements, most developers can arrive at a design that works well for everyone.

I also share how to represent the results in a heat map grid. The heat map grid is a visual way to show the test results: scenario tasks are in rows, and testers are in columns. Each cell in the grid represents how easy or hard each task was for that tester, using a color to indicate the level of difficulty: green for easy, yellow and orange for more difficult, red for very difficult, and black for the tester couldn’t do the task (it was too difficult). After generating a heat map, you can look for “hot rows” to see which tasks were the most difficult for your testers: don’t worry about “cool rows” with a lot of green and yellow, but focus on the “hot rows” with more black and red.