Channels ▼


Gesture-Based Computing for the Desktop

In addition to C++, the SDK also provides libraries for C# as well as the Processing language. Speaking of Processing, this was source of beta issue I encountered. I initially attempted to use the latest 64-bit Processing 2.0 for Windows beta with the libraries supplied with the SDK, but discovered that the SDK libraries only work with the 32-bit stable release of Processing 1.5. Intel is aware of these issues and should have them ironed out in time for the final 1.0 release later this year. To try out the Processing framework, copy the entire contents of the framework\Processing\libraries directory to your sketch directory. Then use the SDK's sample sketch that imports the SDK library, initializes the PXCUPipline, and displays the images captured from the gesture recognizer:

import intel.pcsdk.*;
private PImage display;
private PXCUPipeline pp;
void setup() {
       pp=new PXCUPipeline(this);
       int[] lm_size=pp.QueryLabelMapSize();

void draw() {

Compared to the C++ counterpart, running this sample Processing code is substantially smaller. And since Processing was built with graphics in mind, the number of image manipulation actions, deformation transforms, and visualization techniques that can be applied to the data captured is quite extensive.

Figure 5: Using the Processing language for motion detection and image-capturing purposes.

And yet, even though the addition of Processing is a great help for rapid prototyping and testing, I was dismayed to discover that Intel currently has no plans to offer language libraries for Java, Python, or Ruby. Considering just how many graphics and image-processing libraries there are for a language such as Python, this is a real bummer. It also limits the portability of app prototypes and the like to Processing alone, should Intel ever support any other operating system besides Windows.

Overall Impressions

The SDK samples provide a good basis for learning the key essentials of interacting with the camera, though none of the demo showcase applications demonstrated must-have, mind-blowing possibilities. In its current state, the technology isn't nearly as precise when compared with less-expensive console game controllers like the Microsoft Xbox Kinect, Sony Playstation Move, and Nintendo Wiimote. Granted, the Kinect is considerably larger in size and has benefited from being in developer/students/home brew hacker hands much longer. As such, it has the lead on consumer-oriented hands-free gesture control. The Move and Wiimote benefit from the fact that they are purpose-built devices that must be held and maneuvered in space to work. It's great if you have one of these charged devices nearby and supported by the platform you're interacting with, but they are like operating a stylus versus your own finger for Smartphone interaction — the stylus offers more precision, but the finger feels more natural and is convenient.

Another big problem with the current SDK is its lack of support for MacOS or Linux. One reason for the Kinect's success is the open source hacking that granted access to the hardware for thousands of dedicated hackers, tinkerers, and students who hooked up the Kinect to Mac or Linux systems and wrote some really cool software. By currently sticking with the Windows platform, Intel's solution will directly compete with Microsoft Kinect for Desktop. When compared side-by-side from a developer-centric standpoint, Kinect is a better choice due to considerably tighter integration with the OS, supported programming languages (with considerably more examples to boot), and a more intuitive API.

One problem I have with current gesture-based computing approaches, beyond the current state of less than 100% accuracy by the hardware and software to correctly interpret and act on imaged results, is the simple physical fact of fatigue. Fatigue sets in quickly. When combined with the problems of imprecise gesture interpretation, I found myself fighting with the camera. It occasionally had a hard time accurately detecting my motions, especially since I often inadvertently went outside the camera's field of view. Over the course of my exploration with the technology, my skills did improve, but never to the point where I could sit down in front of the computer and master on-screen object manipulation confidently enough to unplug my mouse. And for the completely uninitiated (like my personal test-market group: my wife and kids), they were initially interested in waving their hands in front of the screen, but quickly tired of the novelty and walked away. So until surface and point detection become laser-precise, I would rather enhance the experience using reflective points on a glove or reflective rubberized fingertip thimbles. We will also need to see instantaneous and highly accurate responses to the gestures for everyday users to take a closer look.

I was pleased to see that Intel included a Privacy Notification Tool with the SDK to assist both developers and users by notifying them when the camera was in use. Given the alarming number of security risks that have grown in recent years due to iterated webcams and smartphone cameras, it's refreshing to see that privacy and security are given such visibility at the foundation of the SDK.

Even with the aforementioned early-adoption problems, there's no doubt that perceptual computing is no longer the province of science fiction. The question is which of the nascent choices in the hands of developers today will be embraced and demanded by consumers. Each new interface technology needs a killer app that drives its adoption. The mouse was widely adopted when GUI-based operating systems became the rage. Today's Smartphone was adopted when multitouch screens helped replace the first wave of PDAs from the turn of the century. After playing around with the hardware and SDK for the past month, it’s not yet apparent to me what killer app will generate the kind of must-have interest for this gesture-based technology to exist on a desktop-centric computing platform. But as a squat green character from a popular space opera once said when the main character flew off to face his destiny, "There is another."


While I applaud Intel's efforts with this SDK and the inroads they have made so far, its current implementation indicates several years of development ahead before this technology can be reliably and inexpensively packaged into laptops, tablets, and wearables where its inclusion makes the most sense. In the meantime, Intel is committed to the perceptual computing concept. As recently as this year's Consumer Electroncis Show, Intel displayed demonstrations of a "Virtual Hand" enhancement (close-range, 10-finger articulation tracking that allows manipulation of 3D objects) coming to the SDK and Tobbi (an eye-tracking technology demonstrated in a "Where's Waldo" style game). Intel also announced at the show that they are working to integrate perceptual computing cameras into Ultrabooks and All-in-One PCs, further broadening the market for this initiative. And to further prime the innovation pump, Intel is currently sponsoring the Perceptual Computing Challenge featuring one million dollars in prizes for developers who can come up with that killer app so necessary for propelling the technology from curiosity to everyday peripheral. Visit for more details.

Ultimately, I see the gesture sensors not as a desktop PC appliance, but as part of a tablet or better yet, a wearable device where waving a hand or gesturing with a finger path will elicit a desirable hands-free effect. Imagine an augmented-reality application similar to the Sony Playstation's Wonderbook while wearing something like Google Glass equipped with depth-perceiving sensors. You could virtually pull down window shades of data and wave away overlays you didn't need to see — a completely natural and intuitive virtual space representation. The future is almost here, it's just not fully rendered yet.

To learn more about Intel's Perceptual Computing initiative, visit their website.

Mike Riley is a blogger for Dr. Dobb's and has been a judge for the Jolt Awards for more than 15 years.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.