Smitty is a rapper from Amsterdam who talks to different versions of himself in his music. He talks to his youngest self and older self about different aspects of life. In his live shows he wants enhance this concept through new technology. Together with Open Culture Tech, Smitty has developed a live show that uses our immersive Avatar technology and mobile Augmented Reality to make this happen.
The evening was composed of three parts. The first part consisted of the audience reception where we used mobile AR to introduce the audience to Smitty's lyrics. The second part consisted of the live show in which we projected various 3D avatars of Smitty on the white walls of the hall. The third part consisted of a Q&A between the audience, Smitty and members of Open Culture Tech.
The entrance
Upon entry, large QR codes were projected on the screen to access the experience. To highlight the lyrics of Smitty's music, we created an AR experience with the Open Cultuur Tech AR app. The idea behind this was that we created a virtual world where Smitty's lyrics floated through space. In the AR experience, 5 different texts from Smitty were spread throughout the room. Visitors could walk through the white empty space of @droog and see the different texts, in the same way as you would at an art exhibition. The AR experience was a warm-up before the show.
The show
In order to make the 3D avatars as prominent as possible, we wanted to create the illusion of an LED wall in the @droog. An LED wall is a wall of LED screens on which you can play visuals. Such a wall is very expensive and therefore unfeasible for most smaller stages. In addition, LED requires some distance between the audience and the screens to provide a clear image. This is also difficult in many smaller stages.
We solved this by installing two projectors that were of good enough quality to project onto the walls. The projections had to run from the ceiling to the floor because otherwise it still looks like you are looking at a normal projection. The projectors were aligned in such a way that they projected onto the walls on either side of the stage. This resulted in minimal shadows from the band on the projections.
Various atmospheric images were projected on this back wall to support the show. These atmospheric images were a combination of free videos from, for example, Pexels and your own video recordings. After the second issue, Smitty's first 3D avatar was introduced on screen. This animated 3D avatar was a younger version of Smitty who slowly turned towards the audience. An older version of Smitty was then shown and these avatars were edited together. The different avatars, in different animations, built up to an eclectic mix that worked towards a climax.
Because we did not want to show the avatars for the entire show, but also wanted to show other atmospheric images, we created a simple VJ setup via TouchDesigner, a software tool with which we could build our own video player. This way we could control the visuals on the projections with a midi controller. Using an HDMI splitter we could control both projectors with 1 laptop.
An important condition for using projectors is that there cannot be too much light in the room because the projections will then become less visible. In Smitty's case, the projections provided enough light to illuminate the room. With two small RGB spots and a white spot on Smitty himself, it was sufficient to properly illuminate the stage.
The Q&A
In addition to music lovers, the audience also included many musicians and fellow rappers of Smitty. For this group, LED walls, animated avatars and augmented reality are not within reach. From the conversations with the audience it became clear that they found the show, which lasted approximately 45 minutes, impressive. The visuals added a valuable layer and supported Smitty's story from the content. This confirmation is important for the progress of Open Culture Tech to validate that our technology is usable for the target group. Follow-up agreements have been made with various fellow rappers to investigate how the Open Culture Tech toolkit can be used more broadly within the Amsterdam hip-hop community. To be continued.