Philips Presents Obsessed With Sound: The Unheard Heroes
Posted by Philip Bulley | Filed under Actionscript 3, Portfolio
My first project at Stinkdigital was also the most technically challenging I’ve ever been briefed on to date. Essentially, the brief was to create a full-screen audio/video mixer featuring approximately 50 audio channels and two HD quality video streams, all of which must be synchronised. If that wasn’t enough, each audio channel should also simultaneously display its current pitch and when clicked on should display it’s individual musical score in it’s entirity. Oh, and to avoid the use of a streaming server if possible!
Visit interactive experience website
I came out of that briefing simply thinking, “wow!”. The complexity that the project offered was certainly a little daunting. “Can Flash Player step up to the task?”, “Is something like this even possible at this scale?” were a couple of the thoughts that went through my mind. Without truly knowing the answer to those questions yet, my confidence levels increased upon hearing that the production team had already scheduled at least 2 weeks for me to create a prototype (prototype! awesome! this can be done! you guys rock!).
Multi-track audio synchronization
Immediately it was obvious that Flash Player had some shortcomings when it came to synchronizing multiple sound channels with beat-perfect accuracy. The “workaround” was nominating a master track and evaluating it’s time against the time of other/slave channels, if the sync went out by a specified threshold, I’d stop the slave channel and restart it at the master time. Not great, but it did the job.
I wanted to make switching between the tracks of the individual musicians as smooth and as quick as possible. Without resorting to the use of a streaming server, the idea was to cut up each track into 8 separate MP3 files. The first MP3 file would contain the full duration of the musician’s solo performance, the second would begin at one eighth into the performance and so on until the last MP3 which would contain the final eighth. That way, when the user decided to switch to another track, we’d look at the current master duration then figure out which MP3 to load. This solution is great, as it utilizes progressive download and is CDN cache friendly. The only downside is having to create and manage the 440 MP3 files!
Multi-track video synchronization
Video would of course, suffer from the same sync issues as audio. But unfortunately due to the way keyframes work, the same solution was not an option as you cannot seek to individual frames (unless you were to encode the video with a keyframe on every frame, inflating the file size to ridiculous levels). Instead the best solution did at first seem a little crazy, but it worked a treat; combining both camera angles in one video stream would ensure that synchronization was 100% perfect.
Example of 2-up video stream running in VLC Player
Encoding the video would be a challenge, squeezing that much video down the pipe at any one time would need a considerably high bitrate if it were to look half decent. We managed to get good results with 2.4Mbps for 2x480p and 3.4Mbps for 2x720p using the Sorenson H.264 encoder (the Main Concept H.264 encoder had tighter restrictions on video dimensions). Such a bitrate would mean that we’d have to buffer a large amount of video before starting playback – a great excuse to use some of the wonderful time lapse photography of the Metropole Orchestra setting up their instruments that had been supplied to us.
Metropole Orchestra timelapse photography sequence as used in video buffering preloader
Real-time pitch display and musical score
With a little bit of help from Melodyne and this online MIDI to XML converter, creative technologist Dan Beattie did a great job of figuring out a workflow to turn solo MP3 tracks of each artist into something that could be used within Actionscript. Fellow developer Jasper Stocker then did a sterling job of parsing that XML to power the musical score display.
A labour of love
That’s exactly what this project became, thanks to Stinkdigital allowing myself and Flash Dev Neil Nand the time to really add the finesse in terms of animation and functionality. It turns out that it was all worth it too, so far it’s scooped awards at Cannes, Clio, D&AD, SXSW, One Show, plus others (they went and listed them, cute little icons included, on the Stinkdigital site).
(Via Stinkdigital London and Tribal DDB Amsterdam)
Tags: as3, music, robotlegs, Stinkdigital, tribal ddb
One Response to “Philips Presents Obsessed With Sound: The Unheard Heroes”
-
Aaaaw – thanks for all the hard work Bulley, and glad you’ve got such fond memories of it!
