The Magic Behind VideoSurf’s Computer Vision Video Search Engine Nov 6, 20087:01 PST

Lots of folks have been asking us about our visual recognition feature, here at VideoSurf. To understand our technology it’s helpful to think about the face recognition tasks we all perform daily, without a second thought. Try this exercise for instance: Whimsical eyes framed by black bushy eyebrows, a neatly trimmed mustache and a wild mop of white hair? It turns out that if you show this caricature to most folks, old or young, chances are they’ll fire back within seconds: Albert Einstein.

Albert Einstein

How we all do this is one of the key strategies used here at VideoSurf. What makes us unique is a search engine based on visual identification, rather than text alone. Where other companies primarily use assigned text tags to catalogue videos, we’ve taught computers to “look” inside the video, to identify specific images, such as famous faces, and index them scene-by-scene. Our goal? To radically change the way users search, discover, and watch online videos. To become your one-stop search engine.

To do so, our VideoSurf team has stitched together a 3-pronged approach of advanced search technologies, fast computation methods, and computer vision. It keeps us superbusy but not out of trouble - some of our spouses have begun to suspect that it’s us and not the computers that are seeing other people!

Just like a police sketch artist can quickly extract and then enhance a con man’s prominent features to add his profile to the local wanted list, VideoSurf’s computer vision strategy applies a model-based approach to identify faces and other leading features. Our algorithm basically spends its days in a virtual living room, watching millions of videos to learn how to detect and follow relevant objects, much like we learn to navigate our world by watching, listening, touching and tasting from infancy on.

From building blocks to recognition

Having downloaded and visually processed millions of videos, we’ve put together a bank of building blocks (eyes, brows, lips, etc). Taking facial features into consideration, such as color, shape, a trademark hairstyle like that sported by former VP hopeful Sarah Palin, or spatial relationships, like closely set eyebrows - we can then use these building blocks to generate and detect faces. It’s like a recipe that helps us predict how a given person will look, even in different situations, like lighting (a sun-streaked or overcast day) or mood (a frown or ear-to-ear grin).

We also use context (meta data) - assuming in sport clips, for instance, that the mountain of skin-colored muscle surrounded by water blue is far more likely to be Michael Phelps than Alan Greenspan. Indeed, it’s unlikely these guys will appear together in the same video(or group of videos) and we use this as an additional cue, so that even if two folks look alike, we rarely mix them up.

Searching for Michael Phelps at VideoSurf

Our model is also helped along by a teacher that feeds in our own experience: a mouth can’t be placed above a nose, for instance, as well as applying text-based tagging to catalogue faces, scene transitions, and other relevant features. To do all this requires heavy computational horsepower, which is where our multigrid processing comes in. The main idea is that of hierarchical clustering of data, similar to the famous 20 questions game: the huge sample of faces is partitioned into groups (blue, brown-eyed etc.). And this is followed by sequentially-dependant questions (such as European or Asian look) that let us quickly zero in on the person in question.

We already have over 12 billion (and rapidly growing) visual moments indexed from videos found across the web, allowing consumers to visually navigate through their results to easily find the specific scenes, people or moments they most want to see. You can now spend less time searching and more time being entertained.