Berlin States Museum Trial Evaluation Summary

The value of the artistic heritage under the care of the museums and art galleries of Europe in the information society steadily increases. Various multi-national moves took place in 1995 - 96 to acquire the electronic distribution rights to this heritage most of which would imply that control of the information relating to the artefacts would pass away from the curators. This would have been an insupportable situation and most of the great collections of Europe refused. However, it does indicate that substantial multi-national corporate pressure exists and therefore any exploitation of the heritage that the keepers themselves wish to undertake must meet the prevailing market standards for multi-media software, if not exceed them.

The Staatliche Museen zu Berlin does not itself intend to produce the applications or the software but to enter into contractual agreements with local companies with speciality in the area of multi media informatics. However, the Educational Service wishes to set up an evaluation and prototyping centre which can co-ordinate user testing, set standards, and ensure that commissioned software reaches a level of usability that is comparable to that available through multi-nationals. Thus user and market-orientated testing is essential for the continued development of this strategy in the Museums.

Two multimedia presentations have so far been prepared for the Staatliche Museen zu Berlin. One is for a temporary exhibition on the voyages of Columbus dealing with historic background and details of the passages. This presentation includes text, audio, and images both static and moving. The museum at this point has to know what the user response to this kind of exhibition is, in order to know how to deploy resources most effectively in the future, and in order to present a viable competitor to more market-orientated products.

The other is a prototype information centre about the so-called museums island of Berlin, a complex of five buildings which house nine different collections in locations near the city centre. The objective of the information centre is to provide the visitor with information about the history of the five buildings, including information about the various collections and their exhibits presented there.

CompArt GMBH is a local company which has been identified as a leading-edge organisation with whom the Staatliche Museen zu Berlin is able to set up a contractual relationship. CompArt is a producer and the Educational Service became involved with the MAPI project in order to promote the development of the evaluation and prototyping centre. CompArt is expected to participate, albeit in a minor role in the evaluations and certainly as a service support provider for technical aspects of the trials.

As a result of discussions with the Human Factors Research Group in University College Cork, Ireland, the kinds of evaluation methods most likely to be useful for the Educational Service were identified. These were as follows:

1. Context of Use analysis, as an essential pre-requisite to allow the correct targeting of user samples, and to keep a record of the evaluation circumstances;
2. A version of the end-user questionnaire SUMI revised for use with multi-media products and standardised and translated into the German language;
3. A free-form 'critical incidents' type of questionnaire to allow end users to express detailed remarks about the products under evaluation in the absence of the possibility of conducting many face-to-face interviews;
4. A simple demographic questionnaire to ensure that the end user samples observed stay within the parameters of the user populations identified in the Context of Use analysis.

Discussions regarding the use of performance measurement (PMM) came to the conclusion that for the duration of the MAPI project this would not be a viable option because of

• resource difficulties in setting up a dedicated laboratory which would simulate the Museum environment in sufficient detail to enable realistic testing scenarios following the Context of Use analysis;
• difficulties arising from privacy issues in setting up cameras within the museum itself for recording the actual experiences of end users who wish to avail themselves of the facilities installed.

The 'cognitive stress' questionnaire measures were also discussed but deemed to be inappropriate for software which is used at an extremely discretionary level.

Evaluations therefore were planned of the controlled naturalistic environment type.

Preparation of Evaluation I

For the current evaluation, bearing in mind the production schedules of CompArt, it was decided to start with an evaluation of the Kolumbus Reise application, situated next to the North American section of the Ethnographic Museum. This is a change from the sequence originally proposed which was to start with the Museums' Guide.

With the assistance of M Kelly (MConsultancy Ltd.), CompArt, and the Human Factors Research Group (HFRG), an initial assessment of the Context of Use of the Kolumbus Reise application was made, and an evaluation plan was drawn up, stating required sample sizes, sample composition, and time frame.

HFRG produced the first standardised version (version 1.0) of their second-generation questionnaire Measurement of Usability of Multi-Media Systems (MUMMS), based on their previous experiences with the SUMI questionnaire. MUMMS is a 50-item questionnaire with the following scale characteristics for version 1:

Scale	Reliability
Efficiency	0.676
Affect	0.723
Helpfulness	0.676
Control	0.696
Learnability	0.786
Global	0.911

The scale factor structure sufficiently replicates the SUMI factor structure to indicate that the same scale constructs are operative in MUMMS. Reliabilities are sufficiently high to warrant the use of this instrument for the evaluation. The current standardisation base is founded on a number of commercial multi-media products that have been evaluated with MUMMS while the current evaluation was planned.

The SUMISCO scoring program was adapted to take account of the population characteristics of the MUMMS scale and the outputs of the program were modified to reflect the MUMMS questionnaire. MUMMSCO is therefore a fully-fledged scoring and reporting utility which is designed to evaluate results from MUMMS.

MUMMS itself was translated into German with the assistance of CompArt and the critical incidents questionnaire and the user characteristics questionnaire were incorporated within the MUMMS questionnaire itself. HFRG conducted a training course in the Museum on how to collect and compile the information, and inspected the evaluation setup on site. A member of the HFRG was available for part of the time during the evaluation in order to respond to any last-minute queries and emergencies, but in fact the evaluation passed smoothly with no major hitches. It was important for the Museum that HFRG maintained, as much as possible, a 'hands off' policy during the evaluations in order for the Museum staff to really make use of the opportunity to gain experience and know-how.

During the evaluation, the Museum staff stood a little way off from the display-mounted computer, and approached users who had obviously taken a serious interest in the application, rather than just taking a cursory look. Users were invited to carry on with the application, and to fill out an anonymous questionnaire about their experiences when they had completed their interaction. There were no refusals.

Results of Evaluation I

Detailed results are shown in Annexe 2, which is in part an annotated listing of the MUMMSCO program output.

The age range and experience range were as expected from the Context of Use analysis, that is, the sample was not biased in favour of any particular age range and the users who responded to the evaluation had a good mix of users with no experience of multi media products and users with at least some prior experience. There were no statistically significant effects on the quantitative scales of either age or level of experience.

MUMMS evaluation

The target graph shows the median score obtained on MUMMS as well as the upper and lower 95% confidence limits for the median. The five scales of MUMMS are so scored that the population (expected) mean is 50, with a standard deviation of 10. It will be seen that the Kolumbus Reise application is scored well the mean, indicating that the end users perceive that it has a high user satisfaction.

The meaning of the five subscales are as follows:

• Efficiency: This is the lowest scale, and it indicates that sometimes the users perceive that the product gets in the way of their use of it. In an ideal scenario, the product would be effectively 'transparent' to the user. However, the median for Efficiency is not at a level considered to be critical.
• Affect: The general emotional response to the product. As will be seen below, the users were generally enthusiastic about the Kolumbus Reise in the Museum.
• Helpfulness: The degree to which the product is perceived by the user as helping them use it, not only by help information and other explicit instructions, but also with reference to the way the product is structured. The Kolumbus Reise is admittedly a fairly simple product (although there are some reservations about its functionality -- see below) and therefore helpfulness comes out with a high satisfaction rating.
• Control: The degree to which users feel in control of the product, rather than being passively made to sit through it, as for instance in a traditional type of tape-slide presentation. It is clear that Kolumbus Reise is able to catch the engagement of the users and give them a feeling of being 'in the driving seat'.
• Learnability: The degree to which the users feel that they are able to start using the product without a preliminary training period or orientation, and that they feel confident about the product to be able to explore features within it that are not immediately obvious. Again, there is a high score on this subscale, due in part perhaps to the simplicity of the interface, but also because the product is itself well organised with the casual end user in mind. Of note is the feature that approximately 10% of the sample indicated severe problems with learnability. This group of users were all inexperienced in multi-media, but there were no trends due to age apparent within the group.

As may be expected, the Global satisfaction score is also high.

Of the critical incidents type comments, content analysis was carried out separately on the positive and negative comments. The most frequently occurring positive comments are: ease of use, quality of the pictures and graphics, and the users having a feeling of 'personal agency' when using the product. The most frequently occurring negative comments are that the speed of the presentation is slow (this is balanced by users who said the speed was 'fast enough'), and that there is a lack of choice in the application because not enough additional information can be found from it. Of note is the fact that the substantial majority of negative comments are constructively critical, which is not always the case in this kind of anonymised evaluation by the general public.

The critical incidents comments therefore reinforce the analysis using MUMMS and clarify some of the important issues to the users. They give in addition a general approval for the concept of introducing multi-media services within the Museum.

Future Plans under MAPI

The next evaluation will take place in the autumn of 1996, and this will be of the information centre prototype. The Education Service of the Museum aims to develop its own in-house standardisation base, with the assistance of the Human Factors Research Group, and by the end of the project to have gained sufficient experience and training to be able to conduct and analyse such evaluations by itself.

By the end of the summer, it is expected that the analysis software will be in place, and staff will be sufficiently trained to conduct routine analysis by themselves. Updating the standardisation base will remain a technical activity that the Human Factors Research Group will carry out, most probably under a data providers' agreement with the Museum.

Future Plans beyond MAPI

Multimedia projects of the Staatliche Museen zu Berlin in the near future:

Forthcoming multimedia projects planned by the SMB will be of two different types:

1. One kind of multimedia application will give museums visitors orientation and guidance in a complex museum building and/or will give an overlook over the different locations of various museums and their collections belonging to the same organisation. The Staatliche Museen zu Berlin, for example includes 17 different collections located in five different districts of the city of Berlin.

One can generally distinguish between museums visitors who want to visit a certain collection or a special work of art - they want to know how to get to the location they desire - and another group of visitors who just want to go into a museum without having a special idea what they want to see - this group will need some information on the collections and their objects.

Multimedia applications of this kind will have the function of a pilot system, making information accessible to museums visitors which help them to plan their visits and to see the things they want to see. This also includes information on opening hours, entrance fees, special events like guided tours or film presentations and on public transport to the chosen location.

This pilot system will be installed in POIs (points of information) not only in entrance halls of museums, but also at airports, railway stations, hotels etc. Since the depth and with of information necessary varies with the place where the POI is installed, it is planned to produce these multimedia applications in a kind of modular system so that the modules relevant for the place of installation of a POI can be chosen accordingly.

The first modules of this pilot system will be installed by mid 1997 in the entrance hall of the newly built museum complex (picture gallery, collection of prints and drawings, museum of decorated art and library of arts) for European Art at the Kemperplatz in Berlin.

The installation of the pilot system will improve the present situation considerably, because not all museums have information desks in their entrance halls and at many places visitors are left alone with information leaflets and wall displays, sometimes even out of date.

2. II. The other type of multimedia application is planned as an information system to be used by visitors of a museum giving background information to certain objects, to parts of a collection or other specific topics of interest. In addition to the traditional slide shows in a museum the multimedia applications have the merit of interactivity, thus giving visitors the chance to navigate on his own through a complex of information offered to him. It is therefore extremely important that these information systems are easy to use and that they stimulate an atmosphere of interest and engagement. These systems should by no means compete with the original objects of art in the museum, they will therefore be presented in separate rooms of the museum, adjacent to the exhibition.

Two different projects are being planned to be installed in 1997:

• With the opening of the new picture gallery in the museums complex for European Art the multimedia information system will be available to visitors. The general objective of that system is to give users the idea that the picture presented are in one way unique piece of art but on the other hand stand in an continuity of historical, social and political developments which are important for the understanding of artistic production in general, not only painting but also music, architecture, literature etc.
• Made possible by the rapid development of 3-D scanning system it is planned to produce a multimedia information system on one of the world famous antique bronze statues, "The Praying Boy", including a 3-D presentation. This statue is at the same time a well documented antique object regarding provenance and status of restoration. Beyond that it had its influence on sculptures in the 18th and 19th century. All these information will be enclosed in the multimedia application made available to visitors of the Antikensammlung Berlin by the end of 1997.

On the basis of the presently collected database, the Museum is setting up its own standardisation base for MUMMS and intends to pilot test all the forthcoming applications as part of a quality management procedure with its suppliers.

For more information about introducing multi-media in Berlin States Museum contact Mr. Juergen Mrosek.

[MUMMS Homepage] [top of page]