Jan Modaal is a punk “smartlappen” artist. Jan is energetic and expressive on stage, but during his performance, he needs to focus on making his music. That's why we looked for innovative ways to use new technology to enhance Jan's physical expression – and his connection with the audience. This challenge led to the creation of the Jandroid Modaal concept.
In Jandroid Modaal, the physical merges with the digital in a future-punk performance centered around avatar technology. The technical foundation for this performance is our Open Culture Tech “Avatar Tool”, which we also used for OATS and Smitty. To enhance Jan’s physical expression, we focused on his facial expressions and body language. To improve our prototype, we built a new feature into our Avatar Tool that allows live motion tracking based on camera input—instead of an expensive motion capture suit.
In preparation for this show, Jan explored new ways to design himself as an avatar, resulting in six different virtual alter egos, each with corresponding background images. These backgrounds consist of AI-generated videos created in Runway ML, a generative AI video tool. These six avatars then formed the basis for a new musical repertoire, with electronic, synthetic music as the foundation.
To present his six avatars on stage, Jan decided to hang a curtain in front of the stage and stand behind it, while a projector displayed the avatars on the stage. Behind the screen, Jan was filmed by a webcam that tracked his movements and transmitted them to the Avatar Tool, which animated real-time 3D characters. A second camera captured his facial expressions so that his mouth and eye movements could be copied in real time to the avatar.
The setup with motion tracking and performing behind the projection screen posed a challenge for the lighting. Some lighting was needed behind the stage to track Jan’s movements, as the cameras couldn’t register him in complete darkness. At the same time, it had to be as dark as possible behind the stage because we projected onto the screen from the front, and backlighting made the projections less visible.
For this reason, we placed a small light on stage. However, it broke halfway through the show and was immediately replaced with a backup light, which turned out to be much brighter. As a result, the audience could see Jan’s silhouette through the screen. In hindsight, many people in the audience only then realized that the avatars were responding to Jan’s movements in real-time. This unexpectedly became a great addition to the show.
Halfway through the show, Jan began tearing the screen apart, making himself visible to the audience. At first, he cut out a square, standing in a kind of frame, with projections still surrounding him. As the show progressed, the entire digital world came down with the screen, and Jan finished the show in his physical form.
This show took place in a small music venue at Cinetol, Amsterdam, and is currently being further developed. If you're curious about more, keep an eye on our schedule for updates.