Go to Sonarworks main page
Immersive Audio
We have all experienced surround sound in one way or another; movies, theme park rides, and video games all use surround audio to heighten the sense of action and emotion. Unfortunately, surround sound typically requires a certain amount of equipment and proper setup to provide a consistent experience to the listeners. Very few homes have proper 5.1, 7.1, or Atmos listening rooms, so content producers have been slow to adopt surround mixes to their content. Those who have been paying attention to the world of VR, gaming, and 360° videos, recognize that Ambisonics has replaced traditional surround sound implementation and this audio works well with headphones.
Traditional surround playback requires several speakers to be placed around the room so that sounds can be panned to various physical, but discrete, locations. Ambisonics, on the other hand, produces a seamless 360° sphere of sound. Ambisonic audio can be recorded with Ambisonic microphone arrays or created with multiple busses during mixing. The Ambisonic signal can then be encoded as a binaural signal, which plays back properly on headphones. So traditional surround requires 6 or more speakers in a room, while Ambisonics works with headphones.
Sound mixers and producers, like Martin Rieger, have embraced Ambisonics as a way to embed immersive, 3-dimensional soundtracks into multimedia content that every consumer can enjoy. This 3-D audio provides a more visceral feel to 360° and VR video, where the audio helps guide the audience through the experience. You don’t simply watch a story—you are in the story.
“Sounds of Germany”
To mark Beethoven’s 250th birthday, the German National Tourist Board hired Martin Rieger to produce an immersive video series that showcases Germany and its cities Berlin, Stuttgart, Dusseldorf, and Leipzig. While ostensibly promotional material to promote travel to these German cities, the videos highlight the history, culture, and modern life of these vibrant locales. Rieger, a veteran recordist and re-recording mixer boasts strong backgrounds in audio production, composition, video production, and programming, making him the ideal candidate to produce this series of videos that incorporate point-of-view video production with immersive 3D audio content.
Sonarworks, which knows a thing or two about headphone listening, was excited to meet with Martin to discuss his approach to this project and his thoughts on Ambisonic audio and headphone listening. We discussed his projects, focusing on the “Sounds of Germany” immersive 3D audio online marketing campaign.
Meet Martin Rieger
Sonarworks: What about your background suits you to produce not only the audio portion but the video portion of immersive content such as “Sounds of Germany?”
Martin: I studied media engineering and production, which included music, programming, and 3D engines for VR, like Unity. I became interested in VR around 2016 and I specialize in multimedia for headphones. For “Sounds of Germany” and other productions, we use 3D video and Ambisonics, or immersive audio. 360° media is different than VR. With VR, the participant controls the [often computer generated] environment, while in 360° environments, the viewer can look around [a real live environment] from a fixed perspective and the producer controls the sights and sounds that they are guided through. For instance, in “Sounds of Germany” we wanted to have the viewer walk up to a fountain and turn around while they hear the sounds of the fountain and ambience all around them moving with the image.
Sonarworks: You have created over 40 VR experiences. How do you see 3D audio and VR growing?
Martin: I feel like sound engineers will not be the ones to instigate changes in audio. Engineers need to look to the audience to see what works for them. An example of this is the emergence of 8D music mixes—an audio engineer would never think about mixing music this way. Music is not produced to be panned in a circle around the listener’s head. But look at how many streams these 3D versions of songs get! Sometimes traditional stereo sound is appropriate, but sometimes the VR sound works well with picture and adds value. Sound with vision is really important to listeners. We use our eyes to tell our ears where sounds should be coming from. The video content drives us to create sound that compliments the image.
Sonarworks: Before we get into the actual production of the “Sounds of Germany” videos, explain why producing for headphones is the goal.
Martin: They [the German National Tourist Board] want to reach as wide an audience as possible and everybody has some kind of headphones. I would recommend open-back over-the-ear headphones, but really whatever pair of headphones the listener has will still present the experience well. This project will be streamed over the internet, and everybody has some sort of headphones or earbuds.
Sonarworks: For this project, you not only produced the audio as the recordist, sound designer, composer, and mixer, but you also produced and edited the video. Is that typical for you?
Martin: (Laughs) No! This is not the ideal workflow for most projects or clients, but in this case, planning the shots and editing the video allowed me to integrate the audio that I imagined using and also to create a tempo with sounds for the videos that I had in my head. The videos of each city are conceived to flow into each other as a part of a larger composition, and I had an overview of the entire flow.
Sonarworks: Martin used music and sound design that was based on the locations and images of the video to provide a sense of atmosphere for the viewer. The sense of tempo was often produced by using sounds that symbolized the locations, like a piece of machinery, a train, or a bell ringing which then became part of the sound design or even the musical score. How did you capture the audio on location?
Martin: For the location recording, the backbone was the Neumann KU-100 dummy head. I like to get creative and use the dummy head how a usual sound engineer would not. A dummy head is usually a binaural reference of a certain place or performance, but I want to create a new reality where the sound tells a story that is more than just realistic—it’s larger than life. I also used an ambisonic microphone mounted to my rig to capture the ambience. The dummy head gets one perspective and with the ambisonic mic, you can rotate the sound in 360° to put sounds in the right places.
Sonarworks: So you used the dummy head as a mobile microphone as you moved around, for instance, the string quartet in Stuttgart?
Martin: We filmed everything in the first-person view and captured images for the best sound experience to push the hyper-realistic effect to the limit. For the live musicians, we moved the Neumann [KU-100] into the room with us so that the audience felt that we were actually moving around the players. The music that was performed live [on camera] was recorded with the dummy head, including moving the dummy head around the performance. During mixing, I had the audio from the dummy head and also mono sound sources that went along with the images. All the music tracks [including the score] were mixed in stereo. The sounds that created the musical pulse were kept as separate tracks, too. So with the Ambisonic mic’s audio, we had three layers to work with during mixing. Everything, even the Ambisonic tracks were eventually mixed down to [binaural] stereo. I also wanted to take the enthusiasm about 8D audio and rotate certain [mono] sound sources around the audience to support the video images.
Sonarworks: Which DAW do you work in for this type of production?
Martin: I work almost exclusively in Pro Tools for its speed, but I see advantages for more complex surround routings and the general flexibility in REAPER and Nuendo. For instance, the IEM plugins support very high-order Ambisonics—up to maybe 64 tracks. This works very well in REAPER. With my headphones, I use the Sonarworks [SoundID Reference] plugin as the last plugin on my master chain, and I use the systemwide app for other audio playing through my computer. My headphones are Sennheiser HD650s that have a custom calibration curve from Sonarworks.
Martin: This project was not easy and in the meantime, it has brought me to my limits. But that’s exactly why I do these kinds of projects: only outside the comfort zone can growth arise.
More Information
Beethoven’s 250th birthday was celebrated in 2020, but due to Covid-19, the shooting took place over the summer of 2021 with Martin Rieger and cameraman Mathias Wallot traveling around Germany for about one month. Along with the German National Tourism Board, Sennheiser and Dear Reality helped support the production. Our readers can learn more about Martin and 3D audio on his website and “Sounds of Germany” can be enjoyed here.
