CO-OPERATIVE
EVALUATION
Primary Reference Sources
Monk, A., Wright, P., Haber, J., and Davenport, L. (1993) Improving your
human-computer interface: A practical technique. Prentice Hall International
(UK) Ltd.
Summary description
This is a cost-effective technique for identifying usability problems in
prototype products and processes. The technique encourages design teams
and users to collaborate in order to identify usability issues and their
solutions. Users work with a prototype as they carry out tasks set by the
design team. During this procedure users explain what they are doing by
talking or ‘thinking-aloud’. An observer records unexpected user behaviour
and the user’s comments regarding the system. The observer also actively
questions the user with respect to their intentions and expectations. This
provides qualitative information concerning any difficulties that the users
experience and the features or interface elements which give rise to these
problems. The method is applicable in the early phases of the design cycle
once a prototype is available for evaluation or at a later stage when an
existing product is to receive further development.
Typical Application Areas
This method can be employed to evaluate any working prototype or implemented
system. It provides a valuable means of verifying and eliciting user requirements.
Benefits
Usability problems can be detected early in the design process. Information
on the user’s thought processes as well as their actions can be obtained.
Communication between designers and users is promoted. The method can be
utilised by those with little or no human factors training.
Limitations
The technique depends upon the use of appropriate tasks and is therefore
unsuitable in the very early stages of design when the task domain has
not been clearly defined. The close involvement of designers in this evaluation
technique makes it unsuitable in circumstances which require an independent
assessment, such as quality assurance The recordings can contain large
amounts of information which may be time consuming to analyse.
Cost of use
This method requires a functional prototype to evaluate and one or more
users to take part (five is the recommended maximum). A location for the
evaluation and realistic tasks to occupy the users will also be needed.
At least one person will be required to act as observer and prompt the
user with questions. This person does not have to have expertise in human
factors as the method is well documented. Some means of recording the user’s
comments and actions will also be needed (eg. audio or video recording,
system logs, notebooks). The number of days required will depend on the
number of users taking part, the duration of each user session and the
time required to analyse the information and produce the report. Assuming
the participation of five users (for two hours each) a co-operative evaluation
could be completed within 5-10 days.
Costs of Acquisition
Compulsory training is not specified.
Suitability for Telematics Engineering
This method may prove valuable as an adjunct to prototyping activities
and prescribes a means for eliciting user feedback.
How to get it
The method is very fully documented (see the primary reference source).
Main steps of the method
The general steps for this method are shown below. This is followed by
an indication of the kind of output produced by this method.
-
Recruit users. Identify the target user population, i.e. those who will
use the system in the future and select users directly from or akin to
this target population.
-
Prepare tasks. Select realistic and appropriate tasks which test those
features of the system that people will actually use in their work, and
those features implemented in the prototype. Write task instructions for
the users. Ensure that tasks are clearly expressed, perhaps by conducting
a pilot session and working through the tasks and instructions. This will
also indicate how much time is required from each user.
-
Prepare any pre-set questions to be asked and clear instructions for those
running the evaluation.
-
Ensure prototype is ready to support the tasks
-
Ensure recording facilities are available and functioning
-
Conduct each session, by observing the users as they work through the tasks
and recording what they say, and by exploring their impressions and intentions
through relevant questions.
-
Make notes of unexpected events and user comments but keep up a dialogue
so that the user is encouraged to explain their actions and expectations
of the system
-
Conduct post-session interviews with the users, drawing upon pre-set questions
and issues raised during the evaluation
-
Debrief the users and thank them for their co-operation
-
Analyse information obtained, summarise unexpected behaviour and user comments.
Consider the themes and severity of the problems identified.
-
Summarise design implications and recommendations for improvements and
feed back to system/process design team. The tape recordings can be used
to demonstrate particular themes.
The primary deliverables from this method consist of a summary of the users’
comments and evaluations and their observed experiences whilst using the
system. In addition the method results in a report which lists the identified
problems with associated severity ratings and makes recommendations for
improvements to the product.