Imagine being in the middle of the audience at a concert of your favourite artist. You have a drink in your hand and are enjoying a great performance. At a certain point you find out that an extra, virtual layer is offered at the concert: via augmented reality, or AR.
People around you grab their phones and hold the screens above their heads. The wildest visuals appear in space through the screens. You are curious, but also reserved. You want to enjoy the moment and experience the concert as it is. On the other hand, it's something that the artist has added to the show, so it must be worth it.
So you give it a chance. You grab your phone and scan the QR code that is projected on the screens on the covered side of the stage. As long as it works on your not-definitely-new phone. And it's not too much of a hassle to install it. By the way, your data bundle is almost empty. Can this be achieved at all? From Wi-Fi?
There are many technical questions at the heart of a well-functioning AR experience. In this article, we provide insight into our development process and the various technical steps we go through when developing the Open Culture Tech AR experiences.
1. The problem
To make an augmented reality (AR) experience during a concert accessible to a large audience, a number of technical challenges must be overcome. These challenges include the diversity of mobile phones and operating systems, varying performance and hardware capabilities, the availability of mobile internet or WiFi and the bandwidth of these signals. In addition, it is important that the user experience, the UX, of the entire process is as simple as possible and contains as few steps as possible.
2. The state of the art
To provide an AR experience on a mobile phone, there are generally two approaches: directly in the browser (on the web) or through a specific app that can be downloaded from the App Store (iOS) or Google Play (Android).
AR on the Web
With AR on the web, the AR experience opens immediately after opening a link (e.g. after scanning a QR code) in the browser (e.g. Chrome or Safari) on the phone. The advantage of this is that the AR experience is accessible because the users do not need to download an app. The downside is that the AR experience on iPhones is severely limited because iOS does not have native support for webXR.
AR through an App
To offer AR via an app, the visitor must first download this app on their phone (for example after scanning a QR code). The advantage of this is that the app can be developed for both iOS and Android in one codebase (for example, by building the app in Unity). This allows us to create a native experience that is optimally tailored to the capabilities of both iOS and Android. The downside is that visitors must first download an app. This is a high threshold; experience shows that few people are willing to take this step.
3. Our Mindset
We want the best of both worlds: to offer the richest user experience without creating a barrier to actually using the experience. For Open Cultuur Tech we are working on a solution that uses App Clips on iOS. This is a technique that allows a small portion of an app to be served in the browser without downloading the entire app. An example of this is scanning a QR code with which you can immediately buy a cinema ticket, without having to use the app.
One limitation of an App Clip is the maximum file size. Apple does not allow this to be larger than 15 MB so that it does not take too long before the content of the App Clip is downloaded. Now we are investigating how we can stream assets in an App Clip so that we can offer more than 15 MB of content. In this way we can offer material from different artists via one app. Based on the scanned QR code, we load the right experience.
We want the best of both worlds: to offer the richest user experience without creating a barrier to actually using the experience. For Open Cultuur Tech we are working on a solution that uses App Clips on iOS. This is a technique that allows a small portion of an app to be served in the browser without downloading the entire app. An example of this is scanning a QR code with which you can immediately buy a cinema ticket, without having to use the app.
In practice, this means that the user can scan a QR code or click on a link. This natively shows an introduction screen with a button to start the experience. The user then immediately enters the native AR experience.
Android offers a similar system: Google Play Instant. If this setting is active, an 'Instant' option will appear on the Google Play page of the app, allowing the app to be opened without installing it. The downside to this is that the user experience isn't as good as on iOS. For example, this feature is not active by default, but the user must have activated a setting himself. In our experience, this 'instant option' replaces one friction with another and is not an accessible solution.
To still offer a frictionless experience on Android, we take advantage of the fact that webXR is excellently supported here. Because we can use ARcore on the web, this experience can feel native. The user experience then looks like this: the visitor scans a QR code or clicks on a link, after which the AR experience opens directly in the browser.
A disadvantage of the above solutions is that we have to develop for two different platforms: in Swift for iOS, and in javascript (using the webXR API) for Android. Because the AR apps themselves only need to display a 3D object and will therefore not be too complex, we do not expect too much 'double' work.
4. Conclusion
Offering an accessible AR experience during a concert in front of a large audience requires careful technical considerations. The choice between a web-based approach and a specific app has advantages and disadvantages that differ per platform. Our proposed approach for iOS users with App Clips allows us to deliver a seamless and rich AR experience without the barrier of downloading a full app. We provide high-quality web-based AR to Android users.
These technical solutions open up new possibilities for artists and event organisers to surprise their audience and involve them in an interactive concert experience, without the audience dropping out due to a too high threshold.