Tag: Microsoft Research

Article – From the horse’s mouth

Microsoft Research

Abstract

Detailed article – PDF

My Comments

Microsoft is working on a way to create better recordings from many smartphones and audio recorders recording the same event

Microsoft has completed some research on how to amalgamate audio recordings of a meeting that were captured by different recording devices to turn out a higher-grade recording that captures the whole of a meeting. It is seen as being the audio equivalent of experiments and projects that aggregate multiple camera views of the same object, or could be seen as a way to create a “Claytons microphone array” using multiple recording devices with their own microphones.

The technique involves the creation of audio fingerprints of each of the recordings in a similar vein to what Shazam and its allies do to “name that song”. But these fingerprints are used to match the timing of each of the recordings to identify what was commonly recorded, allowing for the fact that one could start or stop a recording device earlier or later than another person.

This can lead to TV-grade multi-camera video recordings from a combination of DSLRs, high-end cameras like this one…

The technology that is assumed to be used in this context are standalone file-based digital notetaker recorders or the audio-recording function incorporated in many a smartphone or tablet typically by virtue of an app. Typically these recorders are recording the same event with integrated microphones and implementing automatic gain control and, in some cases, picking up their “own” background noise.

But you could extend this concept to integrating audio recordings made on legacy media like audio tape using standalone devices, or the soundtracks of video recordings recorded during the same event but are subsequently “dubbed” to audio files to be used in the recording. A good example could be someone who uses a “shoebox” or handheld cassette recorder to make a reliable recording of the meeting using something they are familiar with; or someone videoing the meeting using that trusty old camcorder.

… and camcorders like this one of special events.

There are plans to create further research in to this topic to cater for recording music such as when the same concert performance or religious service is recorded by two or more people with equipment of different capabilities.

A good question to raise from the research is how to “time-align” or synchronise a combination of audio and video recordings of the same event that were recorded at the same time with equipment that has different recording capabilities. This is without the need to record synchronisation data on each recording device during production, and allowing for the use of equipment commonly used by consumers, hobbyists / prosumers and small organisations.

The reality that can surface is someone records the event using top-shelf gear yielding excellent audio while others film from different angles using camcorders, digital cameras and smartphones that record not-so-good sound thanks to automatic gain control and average integrated mics, while the good digital cameras and camcorders still implement their excellent optics and sensors to capture good-quality vision.

Once this is worked out, it could then allow a small-time video producer or a business’s or church’s inhouse video team to move towards “big-time” quality by using top-shelf audio gear to catch sound and the use of one or two camcorders operated by different operators to create “TV-studio-grade” multi-camera video.

Who knows whether the idea of post-production audio-level synchronising and “blending” for both conference recordings and small-time video producers.

These LG 4K OLED TVs could be showing a tasteful wallpaper image while music plays

Increasingly, there is an interest in creating subtle wallpaper-type images and displays that can work effectively in the background whether with or without music.

Most home AV systems, especially those kept in a lounge area include a television or similar display device but if you intend to play audio-only content like music, you end up with a device that becomes useless in these environments. You typically end up with a blank screen which could be put to better use.

But you may want something that is effectively serving as a visual wallpaper that can exist in the background without destroying the flow of the conversation. This is more so if you are intending to play background music and something like a karaoke-style lyrics display or a music video may simply look out of place and end up more as a distraction.

Cafes like this could benefit from advanced video wallpaper displays rather than ordinary television

Businesses will also see this appealing as a merchandising tool for promoting their products and services and community organisations may want to exploit these concepts as a way to “set the mood” prior to running an event or promoting their efforts. Even cafes and bars would value the idea of creating a dynamic visual wallpaper rather than showing ordinary television content to set the ambience.

It is infact something that theatres and cinemas have practiced, typically before the programme session starts or during an interval where they show still images, typically advertising while background music plays.

What is being achieved with existing technologies

The concept has been tackled in various different manners such as DVD / Blu-Ray players, network media players and the like showing a screensaver-style image while audio-only content is being played.

A good example that stays in my mind is a Sony DVD/VHS combo that was used to play music CDs through the TV’s speakers and this unit showed a screen which changed between different colours while the music played. Similarly, some hotel TV systems may show a photograph and a text description of the radio station, or the radio station’s logo if the TV is being used to play a radio station, with better implementations showing the current song’s title or other dynamic information that is obtained from tha station’s RDS or similar datastream. Most regular-computer platforms have the ability to run a screensaver which typically is a pattern or animation that comes in to play when the computer is not being used. Or you could set up some network media players to pick photos up from your DLNA media server and run them as a slideshow while you have music playing from that same media server or another media server.

Static image from Christian playlist video used at church gathering that I cited

Sometimes, when people put up music on YouTube, they set up a visual wallpaper that accompanies that music. This may typically be a static image that never changes, or an image that may change at some point. A good example of this that I saw during a church get-together hosted at someone’s house was this YouTube “static-screen” video with the top 50 of the newer Christian contemporary praise-and-worship songs as its audio content while a static slide having a picture of a flying dove and “Because Your lovingkindness is better than life” appearing as a text quote at the bottom that picture. This is similar to how some people have created “continuous-music” DVDs which show one or more static images but have music playing. These and similar examples were based on a practice that WPIX-TV in New York City undertook since the 1950s where they showed a Yule Log burning in a fireplace (YouTube) with some classic Christmas carols and songs playing through the TV’s speaker.

How could this be taken further?

Live Wallpaper

There can be better ways to go about this. One would be to take a leaf out of Android’s book and exploit the “Live Wallpapers” concept which can allow you to create customised wallpaper images that can show real-time information or update the display in a dynamic manner. For that matter, Google could extend this concept to the Android TV platform which Sony is supporting for their 4K Smart TVs and Free have based their latest Freebox on. This could be in the form of showing selected pictures from one or more sources such as your photo collection or a stock-photo library, which could apply well to pictures of a city or country or some general pictures.

Cinemagraphs and short video loops

A beach image could be animated with the waves splashing in

A trend that is starting to become real is the concept of “cinemagraphs” which are video loops that appear like an ordinary photo but have subtle movement taking place. An example of this is a picture of someone sitting by the fireplace with the fire’s flames flickering, a Christmas tree with the twinkling lights or a beachside image with the waves rushing in. Even a close-up of someone may have some accidental movements like twitching or lips puckering.

Apple placed a revision in to the iOS 9 camera app which implements “Live Photos” that allows you to grab a short moment of animation just by taking that photo. Facebook and Tumblr even has integrated support for these “Live Photos” to be contributed to a user’s Facebook or Tumblr presence using the latest versions of the social networks’ iOS native-client apps. Third parties have opened up some desktop software and Websites which turn Live Photos in to animated GIFs which can be uploaded to Websites.

Microsoft’s research division is taking a step in the right direction by completing final research on “cliplets” where a short video can be broken in to “layers” with the ability to create separately-programmable loops for different objects in the video.

Enabling programmatic control of advanced wallpaper

Once a particular file type is defined for these kind of photo-video hybrid images to permit the creation of separately addressed video loops, it could be feasible to have these images driven by external properties like the time of day. This can range from one or mroe different text messages appearing under programmatic control to a graphic that is altereted depending on certain conditions.

An example of this could be a hybrid picture of London’s Big Ben showing the current time of day or a beachside image showing a dusk view when it is dusk. This could also allow for one or more text layers to be created to show text like quotes from a “quote library”, current song metadata, current time or weather information and the like. A media player that supports this file type could then show these for a user-defined time period then show another regular image or hybrid image for that time period.

Conclusion

This will become a constant issue as people want to use visual wallpaper on their display screens especially as these screens become increasingly large thanks to 4K UHDTV technology. Here, people may want to look at ways to keep these screens showing something appropriate during social events but not distracting their guests or patrons; or a business wants to keep their clients aware of their special offers.