StEP: Standardized (Usability) Evaluation Plan

Table of Contents


Full Paper Reference

Grissom, Scott B. & Perlman, Gary (1995) StEP(3D): A Standardized Evaluation Plan for Three-Dimensional Interaction Techniques. International Journal of Human-Computer Studies, 43:1, 15-41.

Abstract

Usability evaluation is a critical component of software development. However, skills necessary to develop a valid and reliable evaluation plan may deter some organizations from performing usability evaluations. These organizations would benefit by having an evaluation plan available to them that was already designed for their needs. A standardized evaluation plan (StEP) is designed to evaluate or compare a wide variety of systems that share certain capabilities. StEPs are developed for a specific domain by usability specialists. These plans can then be used by evaluators with limited experience or facilities because the skills necessary to use a StEP are not as demanding as the skills needed to develop a StEP.

Techniques have been proposed to make three-dimensional interfaces more flexible and responsive to the user but the usability of these techniques have generally not been evaluated empirically. StEP(3D), a standardized evaluation plan for the usability of three-dimensional interaction techniques, combines performance-based evaluation with a user satisfaction questionnaire. It is designed to be portable and simple enough that evaluators can make comparisons of three-dimensional interaction techniques without special equipment or experience. It evaluates the usability of interaction techniques for performing quick and unconstrained three-dimensional manipulations. Two empirical experiments are reported that demonstrate the reliability and validity of StEP(3D). Experiment 1 shows StEP(3D) is appropriate for comparing techniques on different hardware platforms during summative evaluations. Experiment 2 shows StEP(3D) is sensitive enough to detect subtle changes in an interface during formative design.

We make recommendations for developing StEPs based on data we collected and on our experiences with the development of StEP(3D). However, the recommendations are not limited to three-dimensional interaction techniques. Most of the recommendations apply to the development of StEPs in any domain and address issues such as portability, participant selection, experiment protocol and procedures, and usability measures. A collection of StEPs designed for particular domains and purposes would provide a library of reusable evaluation plans. This reusable approach to usability evaluation should reduce the cost of evaluations because organizations are able to take advantage of previously designed plans. At the same time, this approach should improve the quality of usability evaluations because StEPs are developed and validated by usability specialists.


Abbreviated Recommendations for Testing StEPs

The following abbreviated recommendations are based on data collected and on experiences with the development of StEP(3D). However, the recommendations are not limited to three-dimensional interaction techniques. Most recommendations should apply to the evaluation of any interaction technique. See the full paper in IJHCS for full explanations and background data.

TASK ANALYSIS

ASSIGNMENT OF PARTICIPANTS DESIGN OF TEST MATERIALS USABILITY MEASURES PROCEDURE