Mixed Reality in Cultural Heritage Symposium
Summary of discussions
On 03/06/20 I organised a free online symposium with the aim to scope current challenges facing Mixed Reality (MR) Technology to identify opportunities in developing Cultural Heritage immersive experiences. The Symposium brought together professionals across cultural heritage, museums and galleries, digital media design and immersive technologies, along with scholars working with immersive technologies. The Symposium started with four panelists who asked challenging questions/provocations through the presentation of related projects. Below is a summary of the challenges that were posed by both heritage organisations and technology developers with additional points from the discussions that followed.
Note: For the purpose of the summary and during the Symposium I use Milgram's definition of Mixed Reality continuum to encompass all experiences from Augmented (mobile and smartglass AR) to Virtual Reality (VR).
Challenge #1: Use and Affordability
Based on statistics only a tiny percentage of visitors use MR technologies such as mobile AR and VR installations. Is there a need for MR interactions in heritage spaces (historic sites and museums)? Are the audiences ready? What prevents mass adoption? In addition, as different sites have different number of visitors is an experience that is expensive to produce justified for smaller venues with small number of visitors?
::: Points raised :::
There is a complex set of socio-economic reasons behind the small number of visitors using immersive technologies so it is impossible to know for sure how audiences interact or will interact with new technologies. A Mixed Reality experience aims to add content and value to the visiting experience and enrich it (for AR, the equivalent example of watching opera with and without subtitles was mentioned). Many early technological experiences felt short of the engagement factor either because of hardware (e.g. obtrusive, heavy, expensive) or of the experience design (e.g. replicating rather than enriching). However, with both these factors improving considerably, and with audiences being more used to immersive technologies and to 'consuming' high quality audiovisual content there is an opportunity to change the heritage experience.
Another important point that was mentioned in relation to this was the value of the shared social experience in the historic site. Sharing is very powerful, and if MR can be designed in such context, then the organisation can begin to market and gain revenue for it. This could also help with offsetting the cost of an expensive device as affordability is an issue (e.g. Microsoft's Hololens AR headset costs nearly £4K at the moment). Other, non mutually exclusive, ways to offset the cost is to look into where such devices could be used in places where the cost would be considerably higher. An example that was mentioned was the cost of live interpretation as opposed to the cost of recorded interpretation to be used as virtual material.
Challenge #2: Disrupting the Visitor Flow
Immersive installations change the atmosphere for the rest of the museum visitors. VR and smartglass AR requires one person at a time to try it while the rest await for them to finish their experience resulting in a gathering of a group of people in one place. This gathering disrupts the visitor flow. Waiting times may also put some visitors off. In addition, with smartglass AR there is the issue of having other visitors walk in front of the glasses and break the experience.
:::: Points raised :::
This is an important point, particularly with VR installations at the moment. Mobile AR is used in the visitor's phone so in terms of traversing the space it works very similar to the audio guides. With AR, particularly smartglass AR, it could be a significant disruption for viewers when people accidentally stand or pass in front of them. This will benefit from a broader discussion about shared heritage spaces, as challenges with multiple people in a space are also replicated in non-digital heritage settings. For example, some visitors are not interested in live performances (live reenactments) but also don't respect the space of those who are attending by walking in front of them during the performance. Is there value in creating spaces in the historic sites that are only for these different experiences? Can the heritage organisations afford to do so? Is there extra value created for their visitors in this way?
Challenge #3: Practical issues and Inclusivity
A lot of venues/sites don't have any rooms which is a challenge for deploying VR and AR technology. In addition there are usually WiFi issues with speed, no 4G coverage (and not even 5G as it will be difficult to e.g. put a 5G tower in the middle of Stonehenge).
::: Points raised :::
These challenges are real! Sometimes there isn't space to store even audio guides. In addition, AR suffers in difficult light conditions such as very low lighting and bright sunlight. Some partial solutions are to not use it in large outside spaces, to create blended experiences (e.g. with use of binaural audio to enhance experience in difficult settings). If the experience includes sharing content remotely then WiFi issues need to be resolved. However, for non-shared experiences, both in AR and VR, Wifi connectivity won't be an issue as applications can be stored in the local device storage.
On an interesting note, due to the covid-19 situation visiting patterns may change for some time turning to pre-booking and timed visits. This may solve partially some issues with storing and use of immersive devices. The notion that visitors will be bringing their own could solve the space problem but again the site has to ensure an inclusive experience and thus provide devices for those who won't be able to afford them no matter how much affordable they become.
Challenge #4: Embodiment and Interactivity
VR and smartglass AR can significantly change the viewer's embodied sense and their kinaesthetic understanding. What is lost, gained, confused and transformed in this change and in relation to how the viewers experience the physical space? Should designers strive to make seamless interactions trying to make the technology invisible or use the fact that the technology changes the user's embodiment as a design pillar? Can the viewer interact with the scene that takes place in front of them in a meaningful way and for learning purposes (e.g. watch a minuet virtual footsteps and try to learn to dance it)?
There are different responses here with regards to what technology is used since the embodied sense in VR is slightly different than that of AR. From R&D projects it is known that audiences do not want very complex interactions as they have to deal with the complexity of the changed embodiment which is already a massive disruption. From the perspective of MR designers, the aim is to make the experience as seamless as possible, which may mean controlled interactivity, and careful consideration and curation of the virtual content. It is worth noting that context is paramount. A specialised gamer audience will expect more from a VR experience than the average heritage visitor will. Things may change with the '21st century audiences', who may undergo this change much easier due to them being more exposed and used to digital immersive experiences.
Creating meaningful experiences is at the heart of the design process and these may or may not include parts where viewers are required to interact. In Sutton house Stories, we created the possibility of interaction but it was never required. Research (and the user tests during Sutton house Stories production) has shown that viewers do not always want to interact. This is also supported by the fact that embodied experience is happening no matter what one does. Even if they are just watching or hearing somebody dance they are having a physical response. One does not need to be dancing in order to physically engage with the dancing that occurs in front of them. If, however, the aim is to teach to the viewer the minuet, this is possible, it will just require different design and creative decisions.
Challenge #5: Sanitisation and Safety Measures
How is hygiene ensured for visitors using shared MR devices (especially in light of the covid-19 situation)?
::: Points raised :::
There are already sanitisation procedures for VR and AR headsets. Particularly AR headsets can be reliably and cheaply sterilised using UV baths. The distancing that will be required in the wake of the covid-19 situation can be accommodated with AR glasses as you can interact at distance with other people effectively. VR already requires an exclusive space for one person for safety so social distancing is easier to implement.
Challenge #6: Collaborative Research and Development Work
In many cases the collaboration between cultural heritage organisations, designers and developers happen as Research and Development projects, often with the participation of academic scholars. Is there a necessity and a way, if so, to make these collaborations easier to happen? Can they even be institutionalised? Is the role of the curator being extended or a new role emerging with the skillset of bringing digital technology, design, and heritage education together?
::: Points raised :::
Digital interpretation is becoming increasingly the focus and the heritage staff's skillset is being updated extensively. Many heritage organisations are eager to work with academics in an R&D capacity. Such interdisciplinary collaborations require a lot of agility which is easier found in small museums. However, with the recent crisis, interpretation became more agile and the sector became more creative and more risk taking with its digital strand. This can support a more institutionalised approach, for example by integrating such collaborations as part of the investment programme on behalf of the cultural heritage organisations. Financial barriers will be the biggest challenge in terms of resources and government funding. However, partnering with the university sector opens avenues to UKRI funding.