Learning Impact in Virtual Museums

USABILITY HEURISTICS | USER INTERVIEWS | USABILITY TESTING | THEMATIC ANALYSIS

M.Sc HCI Thesis - Learning Outcomes of Technologies used in Virtual Museums

Introduction

Visiting a museum is a very popular pastime and has been for centuries. With e-learning becoming more and more integrated into today’s educational landscape, virtual initiatives by museums have become increasingly invested in enhancement of visitor enjoyment, immersion and usability in addition to knowledge gain.

Additionally, museology’s recognition of learners as active participants in the construction of their own educational insights has replaced traditional learning models that saw learners as passive actors concerned only with acquisition and retention of large quantities of information.

The choice of technology utilised to build a learning environment has substantial influence over its learning outcome. Recent studies have shown that the presence of some qualitative elements in VMs can be linked to stronger educational impacts than others. Our study digs deeper to understand the learning outcomes of VMs in relation to their underlying technology and usability score.

Research Question

How might the elements of digital technologies used in virtual museums produce an impact on learning outcomes?

Framing the Research Question

We began our study with multiple open-ended research questions:

• How do virtual museums define learning?

• How do they measure learning that takes place in physical museum spaces?

• How can we duplicate the complex and unpredictable character of learning?

• How can the nature of technology used in museums create varied experiences?

After reviewing existing museology literature and surveying trends in virtual museums, we refined our research question.

Our study is aimed at comparatively analysing how the impact on learning, as defined by Hooper-Greenhill’s Generic Learning Outcomes, differs when digital technologies characterised by a) interactivity and b) spatiality are used in virtual museums.

Literature Review

What is Learning?

• Learning includes emotions, attitudes, inspiration as well as acquisition of facts.

• Learning is both intentional and unintentional.

• Learning in museums is inspired by and arises from collections but is not necessarily about the collections.

• A focus on the ‘outcomes and impact of learning’ reveals the dimensions of learning in museums.

  The Changing Definition of Learning

• There is almost universal agreement that “learning is an active process that requires engagement”.

• The combined research of the past century belives that this process is “significantly modulated by the learner’s previous experience, culture and the learning environment”.

• Such agreement highlights learning as a phenomenon of such complexity that it can no longer be “sufficiently generalised” for all users.

To accommodate such learning, museums now need to focus on “supporting an individual’s ability to exercise choice and control over their intellectual interests and curiosities” by reconciling the multiple perspectives that visitors bring to the museum with the experiences they have in the museum.

Generic Learning Outcomes

One of the frameworks that have been used to effectively conduct research on learning outcomes in the context of museums is Hooper-Greenhill’s 5 Generic Learning Outcomes (‘GLOs’). It includes:

1.  An increase in knowledge and understanding – includes for example learning facts or information and the deepening of understanding, for instance how things relate to each other; 

2. An increase in skills – includes know-how and the development of skills in order to be able to do new things. It encompasses intellectual skills, social skills, communication skills, physical skills and information management skills;

3. A change in attitudes or values – includes feelings, perceptions, opinions about ourselves or attitudes towards other people or an organisation, increased capacity for tolerance and empathy, increased motivation, and also attitudes towards an organisation and about an experience; 

4. Enjoyment, inspiration and creativity – includes having fun, feeling happy, being surprised, inspired, innovative or creative and exploring, experimenting and making things; and 

5. Action, behaviour, progression – includes what people do, intend to do, or have done and changes in people’s behaviour as a result of the learning experience.

The Virtual Experience

There are many types of virtual museums and multiple methods to sort and distinguish them. In our study, we used a method that distinguishes museums based on their quality dimensions as defined by Sylaiou et al. (2017).

This method of distinction is not clear-cut and so one type of virtual museum can work along multiple dimensions but will be defined by its most prominent one.

1. Imageability - Perceptual quality of an online museum that makes it memorable. E.g. panoramic representations of real-world museum spaces. 

2. Image scalability – The functionality that makes a user equipped to closely examine the exhibition or artefact. E.g. scalable images or text 

3. Navigability – The degree to which a user’s ability to search for information and communicate is perceived from structural elements. In the context of online exhibitions or databases, navigability could be referred to as interactivity.

4. Virtual spatiality - The extension of physical museum space and the metaphors of architecture to virtual space. E.g. 3D exhibit. 

5. Narration – The inclusion of storytelling elements to engage the user. E.g. Video material. 

Usability Heuristics 

Since the main purpose of any museum is education, usability in this case is highly focused on how conducive a system is to user learning.

An environment has to be created in which the user feels motivated and able to learn. One highly popular method for determining system usability in museum studies is the application of the systems usability scale (SUS), which was originally developed by John Brooke. This scale uses factors such as authenticity, interactivity, navigation and learning to evaluate the appropriateness of a virtual museum environment to learning outcomes.

Methodology

Our study lasted approximately 60 minutes per participant and included three parts: 

1. A qualitative within-participants usability test of two pre-selected virtual museums (15 minutes each); 

2. A self-reporting questionnaire focussed on usability outcomes following each of the usability tests (5 minutes each); and 

3. A post-interaction, semi-structured user interview to assess learning outcomes (25 minutes).

Our methodology for evaluation was based on the premise that the distinct characteristic qualities of each digital technology and visualisation method would facilitate different learning outcomes. We considered learning as a dynamic process that transpired uniquely for every visitor as a result of their brand of active engagement, behavioural disposition, knowledge baseline and skill formation.

Participants

We recruited 10 (2 male and 8 female) English-speaking post-graduate students from the School of Information and Communication Studies at University College Dublin as participants.

For this study, the post-interaction interview was considered the primary source of data collection on learning outcomes while the self-reporting questionnaire was the primary source of data collection on usability outcomes. 

Selected Cases of Museums 

Our approach to select case studies of virtual museums for the study began with creating a list of virtual museums available online at the time and cataloguing their characteristics including their underlying quality dimensions. We limited the scope of our desktop research by three filters:

1. VMs that were in the English language. 

2. VMs whose physical counterparts were located in either Dublin or London (based on our geographical location). 

3. VMs that were among the three most visited/ popular museums in each city. 

This gave us a list of 6 potential museums for our study. We further filtered to only those that prominently reflected one visualisation method such that they could act as a representative for their respective quality dimension.

The two virtual museums that were selected for our study were: On the left, the EPIC Irish Emigration Museum representing 3D architectural walkthrough environments (virtual spatiality) and on the right, The British Museum, representing database visualisations (interactivity).

Research Process

• We conducted ten 25-30 minute user interviews in addition to twenty 15 minute usability testing sessions.

• While in a session, we recorded the voice and screen of participants and wrote out detailed notes in order to make sure we were asking the right questions. 

• The participants were given time to fill out their usability focussed questionnaires after each usability test.

• We later transcribed interviews using voice-to-text dictation software and coded everything using MAXQDA into thematic groups, with codes for each participant so their identity was anonymous while their demographic information was preserved.

We analysed the data for qualitative questions, such as, “Which virtual museum did participants enjoy or recall information from or prefer? Why?” (Short answer: both and neither depending on what they expected from a museum experience). 

Most of our time was devoted to reading and re-reading each participants’ narrative to make meaning from their words. What were they most concerned about? What did they think was the difference between the two museums? How did they feel after virtually visiting both museums? How much effort were they willing to make to look for information? What was the value they placed on fun in the context of a museum? What did they notice first during their museum visit? What were their levels of (dis)satisfaction with the quantity of information found? 

Findings

Systems Usability Scale 

• The EPIC museum achieved a usability score of 53.89 out of 100 which translated as an ‘OK’ rating signifying its usability was below acceptable. 

• The score for the British museum was 61.5. This translated as a ‘good’ rating on the scale but is considered below average in the current digital ecosphere.  

• Therefore, both SUS scores reflected below average (under 68) usability performance in the aspects of effectiveness, efficiency, and overall ease of use which was interpreted as a strong signal for the need of significant design updates to both experiences. 

Usability 

• As for EPIC Museum, the lack of readable information and low resolution imagery led participants to feel stifled in their usage and exploration of the museum. Data pointed towards a clear need for more interactive elements within the 3D walkthrough and most participants expressed their inability to position themselves around the museum, especially to read any of the textual elements. 

• In the case of British Museums, participants experienced a variety of navigation issues from moving along the timeline to selecting items on the timeline. While some participants found the abundance of information and different modes like audio and maps very useful, others did not think that the information was presented in an engaging way. However, regardless of their opinion on the way it was displayed, participants felt that the British museum offered deeper information which was also more readily available to them.  

Generic Learning Outcomes

• Participants enjoyed the 3D experience offered by EPIC Museum. They stated that it felt immersive even with large parts of the information being inaccessible. This virtual museum was successful in evoking happiness, wonder and fun amongst other positive feelings amongst participants who expressed increased motivation to visit the physical museum and explore more virtual museums. Participants did not feel that they acquired any new information from the EPIC Museum experience. The level of their perceived learning was unanimously quite low.

• In general, participants preferred the British museum over EPIC for learning in terms of knowledge and understanding. The environment created by the British museum, in combination with the abundance of information offered, was familiar and sparked curiosity and trust.

Discussion

• Defining the museum experience: The results of the data analysis illustrated that every participant’s personal definition of a ‘museum experience’ was a key factor that guided their judgement and perception throughout the study. There appeared to be a split in the participants, where some placed a premium on knowledge building during a museum experience whereas others equated the experience with particular environmental and aesthetic cues and the feelings they evoked. 

• Knowledge & Understanding: It was observed that the simple text and image affordances of data visualisation, highly structured information architecture and common graphics were particularly preferred by participants for traditional educational outcomes or rote learning.

• Enjoyment, Inspiration, Creativity: Virtual spatiality associated with 3D architectural walk-throughs was perceived as more immersive and created higher levels of immediate curiosity, fascination and playful exploration in participants. By contrast, the British museum’s use of interactivity, characterised by structure, user control, familiarity and cohesion lacked surprise but facilitated comfort.

• Usability: An inextricable interconnectedness of usability and technological affordances for learning outcomes was displayed in the data collected in this study.

• Interactivity v Virtual Spatiality: Our findings revealed that while 3D interactivity was found more enjoyable it did not necessarily guarantee better learning outcomes. We found that new environments boosted excitement in participants but the cognitive overload prevented them from efficient learning. Familiarity of interface, on the other hand, helped participants to extend their attention towards understanding and recalling content in the museum.

Conclusion

We examined both virtual museums and found that neither of them was above average in terms of usability, which had a significant effect on user satisfaction and the learning process. We hence concluded that usability was a crucial factor when studying learning outcomes of digital technologies. Further qualitative research on technologies used in virtual museums, keeping usability levels constant or comparable, will be needed in the future for more generalised conclusions about usability, learning and their inter-relation to be drawn. 

On investigation of interactivity and virtual spatiality, our results suggest that users expect value in terms of both novel excitement and traditional knowledge building from a museum experience. A well-balanced combination of the two technologies would hence be ideal to engage users, irrespective of their definition of a museum experience. 

Read full-text study: Learning Outcomes of Technologies used in Virtual Museums (PDF, 758KB)