Posts Tagged ‘usability’

Remember that, the end-goal of conducting usability testing doesn’t stop at timing how long it takes users to accomplish tasks. Ultimately, we need to identify usability issues: aspects of the site’s design, organization, functionality that presented problems to users. Here is where your observations and the interviews provide useful data.

Make sure that, in addition to detailed and clear presentation of usability metrics, as discussed in my previous posts, you identify and explain usability issues. Your report should make it clear to the reader what aspects of the website presented problems.

You can identify major issues for each task, and have a separate section where you list the usability issues, and make recommendations for fixing them. Remember to be very specific about what the issue is and what your recommendations are.

When presenting this information on slides, I recommend placing each issue and recommendation on a separate slide. The slide could look something like this:

Screen shots are helpful here. If a screen shot refers to a specific URL, make sure you write the URL at the bottom of the image, so it’s visible and maybe even clickable. If you use screen shots in the slides, then you will need 2 slides per issue (one with the screenshot), and another one like the one above.

How do you decide what is the best way to present information? When should you use a table, a bar graph, or a pie chart? It takes a bit of thinking about the nature of the information and the message you want to get across – then, Excel can do the rest.

Here is a quick resource for you that explains a bit about making these decisions (link opens pdf).

I believe tables and bar graphs will be most useful to you. So, how do you decide whether to use a table or a bar graph?

  • Tables are great for presenting individual values, but if you cram too much information into one table, it becomes overwhelming. Tables do not facilitate comparisons. In a table, the reader has to search for comparisons among data and compute them mentally, then remember them for later. This is a lot of hard work!
  • Bar graphs, on the other hand, are great for rankings and comparisons.

One thing to be careful about are the sides of bar graphs: What do the axes represent? Are they accurately titled? If you display scaled along the axes, are they correct? Check what types of units you are displaying (absolute numbers, percentages, etc.).  When you work with 5 participants, one participant represents 20%. While marking 20% on an axis is technically accurate, it is a bit misleading. When working with such small numbers, I suggest being very cautious about using percentages, if using them at all.

Let’s look below at three ways of presenting the same information: Level of agreement on a 5-point SA-SD scale, where 5 is SA, and 1 is SD. Imagine the agreement is that the task was easy to complete.

The table presents all the individual values. Look at it for 3 seconds or less and answer: Which task was easiest to accomplish?

Here is a different view of the values each participant gave. It doesn’t show the averages, but, can a quick look at the colors give you an overall idea about which task was the easiest?

(Stacked bar graphs can be very effective. We saw a great example in one presentation yesterday of a stacked bar chart showing task completion rates).

Below is a simple column graph representing only the means for each task. You lose the detailed information about each participant’s responses, but you gain even more clarity about what task was easiest to complete:

The examples in this post are all useful for the first part of the results section, where you present combined results. Things should be much easier in the second part, where you present the data for each task. See my previous post for an overview of the three parts of the results section.

I talked about this in class, but I want to provide a written explanation of how to structure the Results section of your reports.

The results section is the most important one. You spent a lot of time and effort collecting data, and now is the time to analyze and present it. The results section shows off your work. Use and present all the data you collected, don’t keep it secret!

The results section should progress from broad to more and more specific: The first part should present results across tasks, and the second part, results for each task. Then, include in the Appendix the data for each user. So we move from an aggregate of data to individual data points.

Overall Results Across Tasks

This sub-section presents data that enables comparisons across tasks. Compare the tasks on each metric, and show averages across tasks. This is broad-level data that sets expectations for what’s to come: Which was the easiest task? Which was the most difficult? How does expected difficulty compare to actual difficulty across tasks? And so on. Comparisons are best illustrated with bar graphs.

Results for Each Task

This sub-section presents the metrics for each one of the tasks, and enables comparisons across individual users. This is where we begin to have access to individual-level data. Which participant completed the task fastest? Which one took the most time?

Within the sub-sub-section for each task, present the quantitative and qualitative data for all the metrics you collected, and discuss anything you know from observations that might explain the results. Include quotations that illustrate the main points you extracted from the qualitative data.

This blog post has some charts that show metrics per task, and then overall metrics at the bottom. Take a look and note the difference.


The Appendix presents the results at an even more granular level. Present the results for each participant. So, your appendix will have 5 sections, one for each participant. Start with the demographics (but withdraw information that may compromise the participant’s anonymity), then move on and present the participant’s results for the pre-session questionnaire, metrics for each task (the qualitative data can be a summary with 1-2 quotes, not a full transcription), and the post-session questionnaire.

If you follow this progression, you give your reader different views of the data, starting with a broader picture and moving on to individual data points.

The usability report templates I pointed you to, chapter 8 in the Tullis & Albert book, and the class presentations give you options for presenting this information. See also sample reports by Tullis that give you more ideas about what how to present information.

Bonus link: Nielsen’s famous article Why You Only Need to Test with Five Users. You can cite it to back up the number of participants you tested with.

Click image to enlarge

If I’m not really paying attention, I often end up clicking the “Add Another Folder” button when I really want to hit “Save.”

Then, I waste time by backtracking – I have to cancel and start all over again. Reason 5,437 why I hate Blackboard.

What is the usability issue at the root of this user error?

Resources for Web Usability Principles class, Sept  9:

Practice applying these usability principles. Take screenshots of very good and very bad examples, and post them on your blog. Explain in the blog post what the problem is, what principle was violated, and how the problem could be fixed.