|
Frequently Asked Questions about User Validation:
|
The first step in any process that needs to be controlled in terms of budget, timing and other constraints is to develop a plan, showing how available resources will be deployed, when, and to what result. User validation is no different. A user validation plan should cover the following topics:
When should user validation planning start ?
The best time to start user validation planning is at the definition stage of a development project. Develop the user validation plan in consultation with the customers and end users of the application (at least with representatives of both).
Who should perform user validation ?
User validation is a process whereby real users are given access to parts of a developing system. The opportunities for influencing the users' comments is great. Therefore performing user validation should always be in the hands of persons independent from the design team. However, it may be of value for members from the development team to observe users working with the developing system. This can be achieved by either videorecording user behaviour and replaying it to the development team or by implementing an observation procedure where the development team does not disturbe the test user.
How do you determine the 'focus' of a user validation ?
The use of an electronic information application with specific properties and quality aspects is always a function of all system components, i.e. the interaction of the user and the application in a specific context. Validation methods can however focus on measurement of partial aspects of the overall system. The main components which can be the focus of user validation in terms of quality of use are:
How do you set quality goals and evaluation criteria ?
The criteria for evaluation are the starting point for the selection of appropriate - i.e. valid and efficient - validation methods. The main concern within your project, critical success factors will not be measurable as such and must be defined in terms of operationalised measures which are assumed to be indicators for quality factors such as efficiency and effectiveness of use, robustness, learning effort, user satisfaction.
General critical success factors must be broken down into more detailed and specific quality factors to be validated, which then are finally defined in terms of operationalised measurement procedures. An entirely empirical procedure would suggest that the definition of factors emanates from operational measurement procedures by a comprehensive empirical validation process, and would result in empirically identified factors by factor analysis. Such a procedure is not always possible due to the absence of validation criteria, and also of limited use in the context of the development of innovative electronic information applications because innovative features and qualities cannot be assessed by reference to criteria derived from the analysis of existing traditional applications. Quality factors must be defined in terms of properties of the new applications to be achieved by future development efforts.
Assessment and evaluation is by definition always comparative. Comparison may be made on the basis of
Where can you find information regarding the use of evaluation criteria and metrics ?
A good description including examples of how to define evaluation criteria and how to select metrics and methods is provided in the chapter on Usability Specification Techniques in Hix and Hartson (1996, p 221 ff.). Specific background information about the selection of measures for human factors research is provided by Kantowitz (1992) and Muckler & Seven (1992).
References:
Hix, D., & Hartson, H.R. (1996). Developing User Interfaces. Ensuring Usability Through Product & Process. New York: John Wiley & Sons, Inc.
Kantowitz, B.H. (1992). Selecting Measures for Human Factors Research. Human Factors, 34(4), 387-398.
Muckler, F.A., & Seven, S.A. (1992). Selecting Performance Measures: "Objective" versus "Subjective" Measurement. Human Factors, 34(4), 441-455.
These and other more general references can be found on our references page.
Is there an outline for a user validation report available ?
The ELPUB 105 Usability Study: Handbook for practical Usability Engineering in Information Engineering Projects suggests a structure for a user validation report, and the BASELINE User Validation Assistant form can be used as a basis for writing the user validation report. These outlines must be considered suggestions only. The precise structure for a specific user validation project must be agreed with the users of the validation results (e.g. management, the development team, user groups, etc.)
How can you cope with changing project managers in the course of a project ?
Produce a project management plan, including milestones, roles and responsibilities of project team members. Add a description of the most relevant activities and decisions carried out within the course of the project. From the usability engineering point of view it is essential to have a written user validation plan which is updated with modifications and user validation results while the project is proceeding.
Is there a process model for user validation ?
Two process models are relevant to mention:
Setting milestones and coordinating work seems to be difficult in EU funded projects? Do standardised procedures exist ?
Commercial project planning tools exist but to our knowledge are considered too complex and therefore not used too often. The UPI project has created checklists allocating roles and responsibilities to project team members and for coordinating work between team members.
What is the importance of a usability champion ?
To add the role of a usability champion to project teams in Information Engineering has been suggested frequently. A usability champion is a person, possibly the project or quality manager, who has the overall responsibility for usability engineering in a development project. For large development projects with many organisations and companies located in different countries the existence of a usability champion seems to be reasonable. The usability champion plans and organises the usability work. Usability testing is then done by other people in the project. This procedure helps to assure that evaluations are carried out by different project partners in a way that the results are comparable.