Jan Modaal is a punk “smartlappen” singer from Amsterdam. He is an expressive artist who highlights social themes in his music and lyrics. On August 24, he will play live at Cinetol in Amsterdam. Below, we will give you a sneak peek of the conceptual and technical content of this evening.
As a stage performer, Jan Modaal is the frontman and guitar player in various punk bands. Apart from playing the guitar and singing, Jan has limited opportunities to express himself physically and visually on stage. For this reason, Jan and Open Culture Tech are exploring how we can augment his face and body movement – and translate this into real-time visuals using our own Avatar creator tool.
With the latest version of our Avatar creator tool, it is possible to upload yourself as a 3D character (with Polycam) and animate this avatar in real-time using a webcam that copies your movement. In our first live show with OATS, we did this by using a motion capture suit, but this was too expensive and tech-heavy. So for Jan we developed a webcam feature. In addition, it is possible to place the Avatar in a 3D space and adjust the background environment. This 3D creation can be live streamed and projected on a large screen.
From this technical basis, a concept has been developed for the live show of Jan Modaal, called “Jandroid Modaal”. During this show, large curtains are hung at the front of the stage. Jan will stand behind these curtains and perform his show while his movements are recorded by the webcam – which turns his movements into 3D motion. A livestream of this animated Avatar is then projected onto the curtain on stage.
The background environment is decorated with videos that Jan Modaal makes with Runway. This is a popular text-to-video AI tool that allows you to easily generate short videos based on text prompts. In these videos, various “future scenarios” are generated from different “punk” angles. Cyberpunk, solarpunk, steampunk, biopunk, etc.
In our Jandroid Modaal show, the fusion between digital and physical reality is central, while Jan explores the tension between these two worlds. During the live performance, Jan will play with the world in front of the screen and the world behind the screen. We are not revealing exactly how this will be, this can be experienced live on August 24 at 8:00 PM in Cinetol Amsterdam.
The use case with Jan Modaal has resulted in a series of new functionalities for our own Avatar creator tool that we will soon release on the website of Open Cultuur Tech.