Laboratory-Based Observation

---

Primary Reference Sources

Rubin, J. (1994) Handbook of Usability Testing. John Wiley, NY.

Nielsen, J (1993) Guerilla HCI: Using discount usability engineering . In R Bias and D Mayhew (Eds) Cost Justifying Usability. Academic Press, Boston.

Summary description

This is an approach to studying user behaviour in the laboratory, and may be used at practically any stage in the development process when there is a representation of the software that users can interact with. The book by Rubin is cited on account of its clarity of exposition, but this approach is documented in many sources. J Nielsen advocates an approach he calls ‘discount usability engineering’.

Typical Application Areas

These approaches may be used at any stage in the process, although ‘discount usability engineering’ assumes the existence of a prototype that can stand on its own.

Benefits

The method can be seen as an alternative to more extensive trials, which may take place later, prior to a release of a product.

Limitations

If planning and preparation work is underestimated, results may be of little value.

Cost of use

Basic needs for this approach are paper forms for the observers, and equipment to show the interface on a screen or wall. This equipment could include an overhead projector, a computer screen, a barco, or a video based system with monitor. Optional materials needed would be a video camera to record the test session, log software and a computer with a simple data logging program.

Costs of Acquisition

This approach is public domain. The work of Rubin is particularly useful as a starting point.

Suitability for requirements engineering in Telematics:

This method deserves consideration as it directly involves end users. While it can be applied early in the design with a paper and pencil prototype, it also integrates well with Performance Measurement and other metrics oriented methods that can be applied at a later stage of the design.

Detailed description of method

For this approach to be successful it is crucial that thorough pilot testing be conducted by the investigator before introducing users into the lab. This pilot testing has a number of steps:

1. The description of the tasks
2. A simple test procedure with written instructions
3. A predefined format to identify problems
4. A debriefing interview guide
5. A procedure to rank problems
6. An estimation of the proportion of problems identified (optional step)

If pilot testing is underestimated then results may be of little value. This planning will identify usability goals. Roles may then be distributed within the design team, and planning for the test can be administered. The number of subjects needed is then estimated, and a written procedure is made, with at least one pilot test being performed. Useful results can be obtained with three or four users; some data gathering techniques may need larger user sample sizes in order to minimise bias.

Using the test material, realistic scenarios will be recorded, along with demonstrations and instructions. After the session, data may then be collected from the subjects using interviews and questionnaires. Once trials are run, data is analysed, metrics (if any) are calculated, and problem severity is prioritised in an implications report.