The combination of different types media such as video, audio, text, and graphics provide the opportunity to share the potential of each. In general if people are exposed to more than one medium at the same time, this leads to an increase in the useršs understanding of information. However care is needed to integrate media together in an effective way.

If combining audio with visual information such as text or an image, it is important that the two sources of information are clearly linked together. For example a system may present the user with a piece of text with an icon of an audio speaker for the user to click on to select speech. This gives a clear indication that the two are related. If the sound comes on automatically, then a symbol should also appear to draw the users attention to the text being spoken. It is normally best for the system to speak the words exactly as they appear on screen, unless the aim is provide a summary of the text. It may be useful, within the text, to highlight the current element being spoken. Word-for-word highlighting is appropriate for a program to teach reading or foreign language skills. For speech output used to simply emphasise the text, it is preferable to simply show the current line being spoken. It is important to allow the user to interrupt the spoken output at any time. This interruption may be automatic if the user moves on to another activity.

Be careful about different forms of speech being presented at the same time. Users may experience difficulty attending to a passage of speech, for instance, if other sounds or music are also present. When speech output commences over background sound, suddenly muting the background may seem unnatural break. Thus it may be preferable to turn down the volume of the background rather than turning it off altogether.

When the system presents a new subject, information about substantially different subjects or topics should be removed from the visual display and auditory output. This will ensure that the user realises that a new subject has started.

Synchronising media elements

Audio, video and animation are presented over a certain period of time, so synchronisation between the corresponding sequences becomes important. It should be possible to schedule audio, video and animation sequences to take place in specific relation to each another to create a sensible information flow which does not overload the user. Constraints between multimedia objects like these can be specified in three different ways: time-based (pure synchronisation), object-based (pure sequencing) or a combination of both.

If presenting audio and video of someone speaking, then make sure that the lip movements synchronise well with the sounds. Poor synchronisation is particularly noticeable if the sound arrives before the vision.

Copyright EMMUS 1999.
Last updated: September 27, 1999.