|
Frequently Asked Questions about User Validation:
|
Where can one find information on how to use questionnaires in user validation ?(a1)
How can one measure the attitudes of users towards a system ? (a2)
Where can we get help on designing our own questionnaires ?(a3)
What is the difference between a survey and a tailor-made questionnaire ? (a4)
Our tailor-made questionnaire for the elicitation of user tasks and requirements is too long. What can we do ?(a5)
Can we use a question like "How many years experience do you have with the software" in our tailor-made questionnaire for user requirements analysis ? (a6)
What are the differences between SUMI with QUIS ? (a7)
Is it possible to receive some general information about SUMI and what is the cost of SUMI training ? (a8)
Is SUMI no longer appropriate / preferable (in comparison to MUMMS) or might it still be considered as a superset of which MUMMS is a subset ? (a9)
Does MUMMS address multimedia in the traditional sense (embedded text, sound, video in typically a single application produced by Icon Author or Director) or does it address multimedia in the all encompassing European sense (just about "any piece of software")? Could MUMMS be used for usability evaluation of multiple choice quizzes ? (a10)
Is MUMMS available in several languages as is SUMMI ? (a11)
Does MUMMS come with scoring program ? (a12)
How and to what extent does the content of MUMMS differ (if the difference is significant) from SUMI ? (a13)
Can SUMI be used in a multimedia context of use (as is the intention of MUMMS) even though SUMI was not specifically developed with multimedia in mind ? (a14)
Where can one find information on how to use questionnaires in user validation ?
HFRG provides a useful FAQ on this topic.
How can one measure the attitudes of users towards a system ?
Different methods for user acceptance testing are available and are useful in different phases of the development process:
Early in the development process when the design concept, system specification, mock-ups or prototypes are available the focus group technique is an adequate mean of testing the users' point of view. The emphasis of focus group interviews can be either on a small subset of usability issues, or on the whole concept of a new application. Advantageous is the user acceptance testing of alternative solutions during one focus group session. Focus group studies require only a few groups of about 6 potential users to identify major usability issues which could much later, when the application is on the market, result in lack of user acceptance.
When the product is finished, or when it is already in use, then the following questionnaires can be applied:
When administering either of these questionnaires it should be taken into account that the results are only reliable to the extent that users are tested who have had experience of carrying out representative tasks with the software under evaluation.
These questionnaires can be grouped into three categories:
References:
Brooke, J. (1996). SUS: A Quick and Dirty Usability Scale. In: P.W. Jordan, B. Thomas, B.A. Weerdmeester & I.L. McClelland (Eds.), Usability Evaluation in Industry. London: Taylor & Francis.
Lewis, J.R. (1995). IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction, 7(1), 57-78.
Ben Shneiderman (1992). Designing the User Interface: Strategies for Effective Human-Computer Interaction. 2nd Edition. Reading, MA: Addison-Wesley Publishing CO.
Kirakowski, J. (1996). The Software Usability Measurement Inventory, Background and Usage. In: P.W. Jordan, B. Thomas, B.A. Weerdmeester & I.L. McClelland (Eds.), Usability Evaluation in Industry. London: Taylor & Francis.
Kirakowski, J, and M Corbett (1994). SUMI: the Software Usability Measurement Inventory. Brit J Ed Technol, 23.
Kirakowski, J. (1997) Background notes on the SUMI questionnaire. http://www.ucc.ie/hfrg/questionnaires/sumi/sumipapp.htm
These and other more general readings can be found on our references page.
Where can we get help on designing our own questionnaires ?
The HFRG FAQ on questionnaires gives a list of useful reference sources for questionnaire design and also the advice that this is not something for the faint hearted to venture into! While do-it-yourself factual questionnaires are used productively in activities such as requirements gathering or market analysis, do-it-yourself opinion or attitude questionnaires are usually extremely unsuccessful, simply because of the amount of iteration needed to develop a really useful attitude or opinion questionnaire.
What is the difference between a survey and a tailor-made questionnaire ?
It is worth pointing out the difference between a survey and a tailor-made questionnaire. Survey questionnaires usually limit themselves to factual issues that are 'publically verifiable'. Thus the main bias in responding to survey questionnaires is due to the inability of a respondent to report accurately (because the respondent colours the information either deliberately or unconsciously). Tailor-made questionnaires on the other hand usually ask at least some statements of opinion and subjective perception. Here, in addition to the biasses discussed above for survey questionnaires, the additional biasses of misleading or misunderstood questions will arise. If tailor-made questionnaires are used for quantitative purposes, then additional biases due to scaling methods being improperly applied will also be a risk. The sum of these potential biases makes tailor-made questionnaires a thoroughly dubious business.
Present the questionnaire as is to a subset of users and ask them to rate the relevance of each question for their tasks. Check the questionnaire for redundant questions. On the basis of the information you gather remove questions which were rated "not relevant" by the users and remove "redundant" questions.
The number of years experience with a system does not tell very much. The frequency of use during this time is important. Did the user occasionally or regularly use the software? If he used the software only a few times during the year he may not be very experienced and look up the relevant commands in a manual whenever he is using the language. If he uses the software daily or several times per week it is more likely that he is experienced.
What are the differences between SUMI with QUIS ?
SUMI scales are discovered by empirical factor analysis studies and have been replicated. It refers to a standardisation base which yields expected values. It has known reliability and validity statistics. It has been used extensively in Europe and the USA. It exists in standardised foreign language translations. It is a fully fledged commercial product. None of the above are true about QUIS.
Please see the Official SUMI Web Pages. Individuals not familiar with usability engineering will need about one day's training to learn how to apply SUMI.
The developers of SUMI and MUMMS consider both questionnaires at the same level, but they are applicable to different domains. One may even wish to use them both to evaluate the same system: for instance, one might use SUMI to evaluate the quality of interactions of "information providers" with a learning system, and MUMMS to evaluate the multi-media effectiveness of the resulting applications as perceived by "learners".
MUMMS addresses quality of use issues of both views of multimedia. But a piece of software that claims to be multi-media and really doesn't do more than display a few pictures, some canned noises, and lots of menus & text would receive a poor score on MUMMS because the end users would be asked about its "genuine multi-media properties".
Is MUMMS available in several languages as is SUMMI ?
MUMMS is available in German, Dutch and English. Italian users are requesting an Italian version, which will be available soon.
Does MUMMS come with scoring program ?
Not yet, it is envisaged that MUMMS will come with a screen-based administration option which will integrate with a scoring program probably called MUMMSCO. MUMMSCO will take advantage of all the development effort put into making SUMISCO (for SUMI) easy to use.
On the contrary to using the very stable SUMI questionnaire (applied by many high-profile companies in the IT business) MUMMS is more experimental at present. Although it is already quite high in terms of reliability estimates the whole MUMMS scoring procedure at present is done by HFRG using a spreadsheet. So the turn-around will take just that little longer, and the report is not as standardised as the SUMI report.
MUMMS is explicitly targetted at multi-media software frequently found in education, edutainment, information points, and applications with a lot of visual and graphic content. By comparison, SUMI is targetted at desktop software frequently found in an office or home environment or other kinds of work environments. To some extent these two domains overlap, but also, there are extreme points: one would not use SUMI to evaluate an information point touchscreen; nor would one use MUMMS to evaluate a hospital ward data entry system.
It depends on the applications. For instance, SUMI has been used with success to evaluate pedagogical software from the learners' point of view, which included a fair amount of animated graphics and sound. Did that make it a multimedia application? To most experts' eyes, yes. Was SUMI inapplicable? No, all the SUMI questions were answerable, and indeed learners explicitly compared the system being evaluated to other more office-like products they had experience of.