User Interface Usability Evaluation with Web-Based Questionnaires

Last updated: by Gary Perlman (garyperlman.com | quest@garyperlman.com)

Abstract
Keywords
Introduction
Standard Forms of Questionnaires
Combining Options
Options Reference and Design Form
Data Format and Analysis

Abstract

quest.cgi is a customizable Web-based perl CGI script to administer and collect data according to a few "standard" user interface evaluation questionnaire forms (QUIS, PUEU, CSUQ, and PUTQ). The script can be customized and run from a remote server, garyperlman.com/quest/quest.cgi, from which data are emailed in a simple format. The script accepts options as part of its form data:

general customizations such as system name, administrator email, ...
customizations to the rating scale such as number of points, labels, ...
number of open-ended positive/negative comments requested

The script uses these options to present the questionnaire on the Web, gather ratings, general comments, and question-specific comments, and email the results to the designated survey administrator. This document is a user manual that allows the online reader to design and create their own questionnaire. It is available at: garyperlman.com/quest/. The script is freely available, AS-IS, in a shell archive at: quest.sh. No permission is needed to use the questionnaire software, although permission from the survey questionnaire copyright holders may be required.

Keywords

Subjective ratings, Usability evaluation, User satisfaction, User acceptance, User interface questionnaire, Online survey administration, Remote evaluation, Measurement, World-Wide Web, CGI, form, Human-computer interaction, Human factors

Introduction

Questionnaires have long been used to evaluate user interfaces (Root & Draper, 1983). Questionnaires have also long been used in electronic form (Perlman, 1985). For a handful of questionnaires specifically designed to assess aspects of usability, the validity and/or reliability have been established, including some in the following table, some of which are discussed in Chapter 6 of Tullis & Albert, 2008.

Acronym	Instrument	Reference	Institution	Example
QUIS	Questionnaire for User Interface Satisfaction	Chin et al, 1988	Maryland	27 questions
PUEU	Perceived Usefulness and Ease of Use	Davis, 1989	IBM	12 questions
NAU	Nielsen's Attributes of Usability	Nielsen, 1993	Bellcore	5 attributes
NHE	Nielsen's Heuristic Evaluation	Nielsen, 1993	Bellcore	10 heuristics
CSUQ	Computer System Usability Questionnaire	Lewis, 1995	IBM	19 questions
ASQ	After Scenario Questionnaire	Lewis, 1995	IBM	3 questions
PHUE	Practical Heuristics for Usability Evaluation	Perlman, 1997	OSU	13 heuristics
PUTQ	Purdue Usability Testing Questionnaire	Lin et al, 1997	Purdue	100 questions
USE	USE Questionnaire	Lund, 2001	Sapient	30 questions

There are other questionnaires, including instruments from the HFRG (Human Factors Research Group):

SUMI: Software Usability Measurement Inventory
MUMMS: Measurement of Usability of Multi Media Software
WAMMI: Website Analysis and Measurement Inventory
FAQ: Frequently Asked Questions about Questionnaires in Usability Engineering (compiled by Jurek Kirakowski)

Also, see LaLomia & Sidowski, 1990, 1991, 1993 for reviews of measurements of satisfaction, literacy, aptitude, attitude, and anxiety.

NetRaker (netraker.com) once provided online usability evaluation, with a free trial version. Now there are many online survey services, most offering free simple surveys or trial periods:

Note: Although these instruments are online, many are copyrighted and some require a fee for non-educational use. See the references at the end of each form for more information.

These questionnaires are similar in format, so a Web-based system to present them has been developed (as seen in the Example links above). The system, quest.cgi, is a perl-based (version 4 or 5) CGI script that presents a form for a questionnaire and emails data to a specifiable address. Many options for the format of the questionnaire are read from the incoming form data, so a surveyor can adapt an existing questionnaire (although in ways that may change the validity or reliability of the standard forms of the questionnaires). The rest of this document describes how to use the standard forms of the surveys and how to adapt them.

Standard Forms of Questionnaires

Three options to quest.cgi specify all you need to evaluate a system using a Web form.

By default, the CSUQ is used, but you can supply a form= option:
```
form=QUIS
form=CSUQ
form=PUEU
form=PUTQ
```
You can specify a system name (e.g., My Web Site, WordProcessorII) that will be used in the form and reported in the data.
```
system=HCI_Bibliography_Search
system=VR_Applet
```
You can specify an email address to which the data will be sent when the rater submits their data.
```
email=perlman@acm.org
```

Combining Options

Options can be appended to a URL or supplied as form data (either hidden or visible).

To append options to the quest.cgi URL, the list of options begins with a ?, and an & is placed between options. Also, many characters must the hex-quoted (e.g., space is shown as %20). So, to evaluate a system called My HomePage Maker with QUIS, and send the results to marketing@myhomepage.com, you would append ?system=My%20HomePage%20Maker&form=QUIS&email=marketing@myhomepage.com.
Evaluate My HomePage Maker
To pass the options in via a form, you could create a form with quest.cgi as the action and with input elements for system, form, and email, which can be hidden.

Options Reference and Design Form

The following options can be supplied as form data or on the URL. The default values of the options depend on which form is in use. The default or modified values are kept in the HTML as hidden values; they can be viewed by looking at the HTML page source.

GENERAL OPTIONS
OPTION	EXAMPLE	DESCRIPTION
system		The name of the system to be evaluated. This is used as part of the Subject: line in the emailed data.
object		The type of object to be evaluated (by default "System") This is used as part of the form and in some questions.
the		Definite article to precede the object at the start of a sentence. Used for translation to languages with gender (e.g., French Le, La).
this		Pronoun to precede the object at the start of a sentence. Used for translation to languages with gender (e.g., French Cet, Cette).
email		The email address to send the data to.
form		The form of the questionnaire. Allowed values are: QUIS: Questionnaire for User Interface Satisfaction PUEU: Perceived Usefulness and Ease of Use NAU: Nielsen's Attributes of Usability NHE: Nielsen's Heuristic Evaluation CSUQ: Computer System Usability Questionnaire (and the 3-item After-Scenario Questionnaire, ASQ) PHUE: Practical Heuristics for Usability Evaluation PUTQ: Purdue Usability Testing Questionnaire USE: USE Questionnaire If unspecified, the CSUQ is presented. If any other value is provided, it causes a server error.
title		The title at the top of the form. This is used as part of the Subject: line in the emailed data.
sender		This is used as the Sender: of the data.
help		This overrides the paragraph of help at the top of the form. Supplying this requires a fair amount of expertise with HTML forms.
RATING SCALE OPTIONS
I like to use seven-point rating scales because they allow three levels of either positive or negative ratings; two levels seems too few. I prefer to range from 1-7 (bad-good) instead of the more techno -3...+3.
OPTION	EXAMPLE	DESCRIPTION
lowval		The value at the low end of the rating scale. This value can be negative.
highval		The value at the high end of the rating scale.
lowlab		The label at the low end of the rating scale (for questions not assigned a special label).
highlab		The label at the high end of the rating scale (for questions not assigned a special label).
NA		The label to indicate Not Applicable.
NAval		The value to send when NA is chosen.
commlines		The number of lines in the comment area. If 0, then no comments can be added to questions. If 1, then a single input field is provided. If 2 or more, a textarea is provided.
NEGATIVE/POSITIVE POINTS
I like to ask users about the N most negative / positive points about a system (first negative, then positive). I usually get N/2 answers, so 3 or 5 is a good number to ask for.
OPTION	EXAMPLE	DESCRIPTION
negative		Number of negative points about system for rater to supply.
positive		Number of positive points about system for rater to supply.

Data Format and Analysis

Data is sent to the email address in a name=value format, with one name and value per line. Missing comments on specific questions and missing negative/positive points are omitted. You are on your own for the analysis. Obviously, you must coordinate with the questionnaire form.

Here is an example mail message with questionnaire data.

From garyperlman.com Mon Oct  5 12:47:00 1998
Date: Mon, 5 Oct 1998 12:46:59 -0400
Subject: quest.cgi: Questionnaire System Evaluation
Sender: perlman@acm.org
From: perlman@acm.org (quest.cgi)
Apparently-To: perlman@acm.org

system=quest.cgi
date=98-10-05 12:46:58
comment=I like the generalization of this system.<br>Thanks.
q1=1
q2=0
q3=2
q4=1
q5=1
q6=0
q7=-1
c7=I found the concept novel & difficult to grok.
q8=0
q9=NA
q10=0
q11=-1
q12=-1
q13=0
q14=0
q15=1
q16=1
q17=1
q18=2
c18=It has more than expected, and yet, not enough.  Sigh.
q19=2
negative1=no analysis tools
negative2=hard to completely customize
negative3=questionnaire terminology does not closely match my domain
positive1=nice looking forms
positive2=lots of options
positive3=based on established questionnaires
positive4=lets me automate data collection