Company A is a large organisation geographically distributed over the British Isles, whose headquarters are in London. They have a small usability laboratory, situated outside London, but the work reported here was conducted by a human factors consultant local to head office.

In company A, a process improvement programme was being produced. The processes were implemented in LOTUS NOTES. Two usability evaluations were carried out at intervals of 12 months in order to allow changes to the system to be incorporated into the application and to also allow the planned upgrade of the version of LOTUS NOTES.

Evaluations were planned using the Context of Use analysis results. In order to conduct the Context of Use analysis, contact was made with the major stakeholders in the process as well as some of the end users involved, and each was interviewed separately.

Two teams from Company A were closely involved from the very beginning: the development team, and the quality process improvement team.

The evaluations involved the use of the SUMI questionnaire, time on task, and critical incidents reports. For each evaluation, a group of approximately 20 end users was selected and invited to participate. Nobody declined.

1. The first evaluation demonstrated some basic problems with the application as delivered and suggested some recommendations for improvement. Some of the recommendations were easy to implement and were acted upon immediately while the others were planned and implemented in time for the second evaluation to measure them. The development team manager said the results of the critical incidents were 'helpful.'
2. The second evaluation was performed within the same database in LOTUS NOTES. The first evaluation found generic problems with the LOTUS NOTES application, which were corrected during the interim period. The second evaluation found new problems more to do with the content of the applications rather than the way they were presented. These problems, some of which had already been identified informally, were addressed at the next upgrade of the NOTES based databases

These field trials demonstrated a consistent way of evaluating the usability of the trial applications that would be applicable to all the development projects within the group. The processes can be used with relatively little training resulting in large benefits from a relatively small investment.

In this case study, the outside consultant approached the development team directly and involved them in the process from the start. There were a mass of small, irritating problems that the development team had heard about, but it wasn't till they received the 'critical incidents' report that the manager of the design team could see which were the most loudly-complained-about problems. The second evaluation results enabled the design team to show that they were effective in responding to user problems and enabled them to prioritise the content usability aspects.

There was some resistance at first within the organisation to devote time to the studies, so the human factors consultant went round from group to group individually and got buy-in. The fact that senior management strongly supported the project, but didn't 'police' it, was very important. By the time of the second evaluation, however, company personnel were much more committed to giving time to the project. The quantitative results from the SUMI questionnaire, when the results of the two evaluations were presented together, were a very effective measurement of the success of the project within the organisation, and this graph was used by senior management to demonstrate this success to other centres in the organisation.

The development team was in-house, but there was nevertheless an effective developer-client separation. Basically, the clients wanted to know whether their in-house developers were doing enough for them, or should they sub-contract the work to an outside party. The usability evaluations enabled personnel from the client to voice their views, sometimes strongly held, about the progress and direction of the project without getting into a confrontational situation. Once the clients had seen that the evaluations were not a 'show job' they were much more willing to participate in the second evaluation, 12 months later. Confidence in the development team developed between the evaluations.

This could have been a potentially explosive situation, with the HF consultant getting caught in a three-way firefight between the usability laboratory, developers, and the organisation's end users (here known as the 'clients'.) The key strategy employed was do carry out a 'light' evaluation which was seen as being parallel to the laboratory work, and to make individual, personal contacts between personnel in the developer's team, the quality management team, and the client. Views from all of these sources were listened to and incorporated into the evaluation plans so that the evaluation itself was seen as a communal effort, not something that had been imposed on from outside.

Copyright EMMUS 1999.
Last updated: September 27, 1999.