Posts Tagged ‘Resources’

Research resources on Bb

Posted: September 16, 2011 in Resources

I have just finished added a folder to your Readings on Blackboard. It is called Research Resources and it contains useful book chapter about:

  • how to write research questions
  • how to organize literature and write a literature review
  • sampling in qualitative research

Please review, use, and cite as you prepare your own research project.

I just uploaded on Bb, as promised in class, one of my older papers, so you can see yet another example of a literature review. I’ve made comments on the side and highlighted text to draw your attention to the organization of the literature review, and how I used it to build an argument for the study.

Note that although the paper is about corporate presence on Facebook, there’s almost no research I review on that particular topic. Instead, I review research on closely related ideas. This is one common mistake students make: If they can’t find research on their particular topic (which is actually a very good thing!), they don’t look one level away, to topics that are related, but not quite identical.

A good literature review funnels ideas, concepts, methods, findings from several areas of research into this one thing, which becomes your research question. Speaking of research questions, if you don’t know what they look like, take a look at the slides below to see some examples. Ideally, your literature review should end in a specific research question.

I prepared these slides for a presentation about scholarly research I gave to an Intro to Grad Studies class (CGT 501), but I’d like all of you in TECH 621 to review it.

In addition to helping students understand the characteristics of scholarly research, I mentioned a few helpful blog post from my other blog, PR Connections. They are:

I know that one of the most difficult challenges of your final report and presentation is figuring out the most effective ways to communicate data. It takes scientific precision, artistic creativity, and great communication skills. It should be a fun challenge for graduate students – but, at this time of the semester, it gets quite painful, I know…

Take a break, watch the video below. It will remind you that there’s power and joy in data visualization:


Oh, and… never mind Power Point. Here’ the new requirement for your final presentation! /badjoke.

Remember that, the end-goal of conducting usability testing doesn’t stop at timing how long it takes users to accomplish tasks. Ultimately, we need to identify usability issues: aspects of the site’s design, organization, functionality that presented problems to users. Here is where your observations and the interviews provide useful data.

Make sure that, in addition to detailed and clear presentation of usability metrics, as discussed in my previous posts, you identify and explain usability issues. Your report should make it clear to the reader what aspects of the website presented problems.

You can identify major issues for each task, and have a separate section where you list the usability issues, and make recommendations for fixing them. Remember to be very specific about what the issue is and what your recommendations are.

When presenting this information on slides, I recommend placing each issue and recommendation on a separate slide. The slide could look something like this:

Screen shots are helpful here. If a screen shot refers to a specific URL, make sure you write the URL at the bottom of the image, so it’s visible and maybe even clickable. If you use screen shots in the slides, then you will need 2 slides per issue (one with the screenshot), and another one like the one above.

How do you decide what is the best way to present information? When should you use a table, a bar graph, or a pie chart? It takes a bit of thinking about the nature of the information and the message you want to get across – then, Excel can do the rest.

Here is a quick resource for you that explains a bit about making these decisions (link opens pdf).

I believe tables and bar graphs will be most useful to you. So, how do you decide whether to use a table or a bar graph?

  • Tables are great for presenting individual values, but if you cram too much information into one table, it becomes overwhelming. Tables do not facilitate comparisons. In a table, the reader has to search for comparisons among data and compute them mentally, then remember them for later. This is a lot of hard work!
  • Bar graphs, on the other hand, are great for rankings and comparisons.

One thing to be careful about are the sides of bar graphs: What do the axes represent? Are they accurately titled? If you display scaled along the axes, are they correct? Check what types of units you are displaying (absolute numbers, percentages, etc.).  When you work with 5 participants, one participant represents 20%. While marking 20% on an axis is technically accurate, it is a bit misleading. When working with such small numbers, I suggest being very cautious about using percentages, if using them at all.

Let’s look below at three ways of presenting the same information: Level of agreement on a 5-point SA-SD scale, where 5 is SA, and 1 is SD. Imagine the agreement is that the task was easy to complete.

The table presents all the individual values. Look at it for 3 seconds or less and answer: Which task was easiest to accomplish?

Here is a different view of the values each participant gave. It doesn’t show the averages, but, can a quick look at the colors give you an overall idea about which task was the easiest?

(Stacked bar graphs can be very effective. We saw a great example in one presentation yesterday of a stacked bar chart showing task completion rates).

Below is a simple column graph representing only the means for each task. You lose the detailed information about each participant’s responses, but you gain even more clarity about what task was easiest to complete:

The examples in this post are all useful for the first part of the results section, where you present combined results. Things should be much easier in the second part, where you present the data for each task. See my previous post for an overview of the three parts of the results section.

I talked about this in class, but I want to provide a written explanation of how to structure the Results section of your reports.

The results section is the most important one. You spent a lot of time and effort collecting data, and now is the time to analyze and present it. The results section shows off your work. Use and present all the data you collected, don’t keep it secret!

The results section should progress from broad to more and more specific: The first part should present results across tasks, and the second part, results for each task. Then, include in the Appendix the data for each user. So we move from an aggregate of data to individual data points.

Overall Results Across Tasks

This sub-section presents data that enables comparisons across tasks. Compare the tasks on each metric, and show averages across tasks. This is broad-level data that sets expectations for what’s to come: Which was the easiest task? Which was the most difficult? How does expected difficulty compare to actual difficulty across tasks? And so on. Comparisons are best illustrated with bar graphs.

Results for Each Task

This sub-section presents the metrics for each one of the tasks, and enables comparisons across individual users. This is where we begin to have access to individual-level data. Which participant completed the task fastest? Which one took the most time?

Within the sub-sub-section for each task, present the quantitative and qualitative data for all the metrics you collected, and discuss anything you know from observations that might explain the results. Include quotations that illustrate the main points you extracted from the qualitative data.

This blog post has some charts that show metrics per task, and then overall metrics at the bottom. Take a look and note the difference.


The Appendix presents the results at an even more granular level. Present the results for each participant. So, your appendix will have 5 sections, one for each participant. Start with the demographics (but withdraw information that may compromise the participant’s anonymity), then move on and present the participant’s results for the pre-session questionnaire, metrics for each task (the qualitative data can be a summary with 1-2 quotes, not a full transcription), and the post-session questionnaire.

If you follow this progression, you give your reader different views of the data, starting with a broader picture and moving on to individual data points.

The usability report templates I pointed you to, chapter 8 in the Tullis & Albert book, and the class presentations give you options for presenting this information. See also sample reports by Tullis that give you more ideas about what how to present information.

Bonus link: Nielsen’s famous article Why You Only Need to Test with Five Users. You can cite it to back up the number of participants you tested with.