Frequently Asked Questions about User Validation: Performance Measurement

Return to BASELINE Home Page

Return to FAQ Overview Page

1. How can you measure user performance ?(a1)

2. How can you measure the effectiveness of an information application ? (a2)

1. How can you measure user performance ?

   1. By measuring the time it takes a user to complete a certain task or set of tasks.
   2. By measuring the time it takes to learn how to operate a system.
   3. By counting and classifying system-defined events such as number of errors, number of accesses of help information etc.
   4. By assessing the quality of the user's interaction or the result of the interaction.

   Performance measures can be simple counts of events (event frequency) or they can be density of events (number of events per unit time), or they can be probability statements (eg number of correct actions in a sequence of actions). Density and probability statements are preferable as they are dimensionless metrics. Do take care when summarising time measures: they tend not to be normally distributed, hence the arithmetic mean is a poor estimator of the central tendency of a set of time data.

   An experienced evaluator can do user performance measurement with a stopwatch and a clipboard. However, the danger is that there is no 'audit trail' left for re-checking and further analysis if needed. Thus, video recording is usually associated with user performance measurement. Noldus (http://www.noldus.com) supply a useful set of hardware and software components that greatly ease the task of video analysis. There are many suppliers of 'usability laboratories' in the USA that offer these kinds of facility to a greater or lesser extent, including some 'portable usability laboratories'.

---




2. How can you measure the effectiveness of an information application ?

   Effectiveness is the ability of a user to produce a specified kind of result, whether this be the correct hits in a search, or the number of true errors in a spreadsheet, or the appropriate actions carried out in response to an error condition. Effectiveness is to some extent influenced by the usability of the application to be evaluated! A useful effectiveness measurement scheme would be created by making up a set of rules whereby one may judge the user result in an objective way, giving 'points' for desirable features in the result so that critical features attract more points than do trivial features. It is always useful to have a second opinion both for the set of rules, and to check that the rules are being applied consistently when user results are being measured. It would be very useful to be able to cite percentage or probability figures: for instance: 'user x got a score of 96% for finding specified matches in the database' (where we know, because we set up the user trials, what would constitute a score of 100%).

   Another, less valid, way of measuring effectiveness is to ask the user how effective they feel with the application. Some care should be taken in interpreting this latter measure, because how a user feels about their performance may turn out to be quite unrelated to how they actually perform!