Timeframe: January 2022 – August 2022

Background

The nature of 2D cutout animation is the recording of analog fragments frame-by-frame. This technique is an established way to create moving pictures. In ExperiMotion 1, cutout animation was combined with virtual reality. Here, the animation appears at a micro-level (i.e., animated details of objects) and at a macro-level (i.e., animated environments). 

Project Content

ExperiMotion 1 took place in cooperation with guest artist Max Hattler, whose current research focuses on synesthetic experience and visual music and the narrative potential of abstract animation. Fellow invited artist Sune Petersen's practice mainly evolves around generative video performance and playful performative installations. The animation style of ExperiMotion 1 is based on the film Shift (2012) by Max Hattler, and the animated objects were borrowed from the collection of Hans „Nick“ Roericht, a design professor at the Berlin University of the Arts. Another source of inspiration was the Hattlerizer developed by Petersen and Hattler. For the auditory part of the project, Sune Petersen developed an algorithm to sonify the stop-motion loops based on the principles of optical sound.

Goals

Since the cutout animation technique is inherently flat, working with it in a three-dimensional virtual space was a particularly interesting challenge. Therefore, the following research questions were developed:

• How can we define space in a virtual environment? 
• How can we create spatiality in a virtual environment with an animation technique (cutout) that is flat in and of itself? 
• How do we transfer the haptic properties of the cutout animation to the virtual world?
• What are the possible functions of a stop-motion loop in an animated virtual live performance?
• How can we create spatial sound in a virtual environment?

Methods and Outcomes

Artistic research was an essential part of the working process. For this purpose, logs, video recordings, and voice recordings, as well as transcriptions and screengrabs of essential milestones, discussions, and meetings, were made to understand how exactly the decision-making process took place during the span of ExperiMotion 1.

At the beginning of ExperiMotion 1, the VRinMotion team built two prototypes in TouchDesigner and Unity, working with video feeds and capturing the stop-motion animation right out of the VR. After discussions with the invited artists, the team decided to proceed with Unity and import cutout animation loops via an internal network into the VR. For the Unity prototype, the team implemented the Mixed Reality Toolkit (MRTK) by Microsoft. With this toolkit, artists, as well as participants, can place the cutout loops freely in VR space and change size and direction. Because the backface-culling in the shader that displays the animation as texture is turned off, the cutout loops are also visible from the backside, and it is possible to build 2 ½ dimensional objects. Haptic aspects of the stop-motion objects are also translated very well into the virtual environment because of the good resolution of the Varjo XR-3 Headset used for the project.

In addition to manual placement, spatial pre-arrangement of the loops was done in After Effects and vvvv and imported into the virtual environment for further manipulation and arrangement. In addition to placing animation loops in the foreground, the cutout animation can be imported into a background sphere, which can be irregularly distorted by a UV-shifting algorithm implemented in the shader.

Participants

Max Hattler, Sune Petersen, Christoph Schmid;
Franziska Bruckner, Clemens Gürtler, Matthias Husinsky, Mario Zeller, Arian Jalaeefar, Christian Munk, Julian Salhofer, Peter Schoiswohl;
Ramon Brullo, Patrick Horvath

Talk by Franziska Bruckner: Between the Frames: Expanding Stop-Motion for Virtual and Hybrid Environments at Expanded Animation. Mapping an Unlimited Landscape, Ars Electronica Center, Linz. (2022, September 9)