CGT 512 Fall 2010: Pilot Test

Posted: November 2, 2010 in Assignments
Tags: ,

The purpose of the pilot test is to test your procedures and instruments. How did your pilot test go? Did all your forms and instruments serve you well, or did you come across things you wanted to change? If so, what did you change, and how?

  1. MOania says:

    it went alright. the camera is tricky to set up, do to the low zoom levels. actually set up the camera on the desk. we also might set up another one to get more face detail or screen detail. i have been assigned on running the entire thing, since i guess my english is the best, and i am an american citizen. haha. we all worked well as a team, and just practice will make it flow better. i also need to write a better script.

    • Mihaela says:

      thank you for the feedback! I understand the issues of working with a small, cheap camera… As to you being the only one assigned to run the sessions, that’s not fair! Everyone in your group needs the experience of working with research participants, and is certainly able to do so – I am an international, too. If I can do it, so can they 🙂

  2. korash says:

    The pilot test with Marcus was interesting. Marcus is still trying to figure out how to upload the video file compressing that file. We have found out couple of things that we need to change or add on the test instruments.

    1. Gift Card Survey link on Qualtrics has to open new window. The previous setting was that Gift Card Survey Page was updated on the current Qualtrics window. So we lost the Usability Test Questionnaire after clicking the Gift Card Survey link in the pilot test.

    2. Added text box for researcher name on Coding Form and Interview Form for the record. We also deleted the Start and End Time text box on Coding Form because time and duration for each task will be recorded separately by time-taker. Observers who take notes on Coding Form and Interview Form do not need to worry about the time and duration.

    3. Determined the order of Participant ID #:
    Faculty Participants: F-1, F-2, F-3, etc..
    Graduate Students: G-1, G-2, G-3, etc..

    4. Created time measurement metrics for recording time and duration for each task.
    Decided the cut-off time for each task.

    5. PDF Survey Form is created to compile the Coding Form and Interview Form which are manually written during the usability test.

    6. Added Qualtrics URL, nanoHUB ID, and password on the first page of testing material and Qualtrics where we enter each participant’s ID # for the convenience.

    We have updated these things on our testing instruments and we will send you the final version tomorrow at around noon for your review again.

    We have also realized that we need to decide the roles other than 1-Administrator, 3-Observers, 1-Time Taker for each test. Such as:

    1. Download video file from cam and Upload video to SharePoint.

    2. Collecting Coding Form & Interview Form and compile, then export those data to excel spreadsheet.

    3. Time Measurement Data Updating on spreadsheet.

    These works could easily get in a hassle but these are also important to analyze the collected data and write the report.

  3. Mihaela says:

    It sounds like you paid attention to a lot of details and are thinking through the changes. Very good. Re #1, if I understand correctly, the answers don’t get submitted to the databases when the gift card form is linked as you had it before? That’s important to know. Let me know when everything is ready to go, so I can give you access to the research participants.

    • korash says:

      After the gift card survey, we re-visited the usability test qualtrics using the link and we were able to get back shortly to the page which has gift card survey link and where we lost. Then we clicked next button from there to complete and finalize the test to submit the answers to the database.

      • Mihaela says:

        This is what you need to do: In Survey Options > Survey Termination, check the box “redirect to URL” and paste there the URL of the gift card form. This will ensure that all data gets submitted to the database before participants move on to the gift card survey.

  4. Duskie says:

    Our pilot test went pretty well. We found a couple of things that we needed to change. One of the changes we were thinking about making involved using a webcam in order to record our user’s facial expressions. However, we were not sure if this was allowed? We also found that we needed to change our script/survey to tell the users to go to the nanoHUB home screen after completing each task.

    • Mihaela says:

      Re: video recording – ideally, you’ll set up the camera so you can record both the screen and a side-view of the participant. This way, you will catch the nonverbals, and you will also know what exactly on the screen they refer to. If you record facial expressions separately, it will be hard to associate them with specific usability issues on the site you’re testing.