It's always interesting to follow technologies as they make their way from the lab to the real world. An example is Intel's Ct, a programming language that lets you abstract data-parallel programming away from the hardware while supporting deterministic behavior to avoid races and deadlocks. Intel recently announced Ct will be incorporated in real products by year's end.

Another example is the SPADE programming language out of  IBM's T.J. Watson Research Center. In fact, SPADE (short for "Stream Processing Application Declarative Engine") has been around at least since 2007 when IBM rolled out its System S stream computing solution, a system with 800 or so processors. The System S software lets application software split up tasks, then reassemble the data into something that makes sense. In other words, stream computing leverages SIMD (Single Instruction, Multiple Data), letting applications achieve some degree of parallel processing. System S is designed to analyze hundreds or thousands of simultaneous data streams -- stock prices, retail sales, weather reports, etc. -- and deliver nearly instantaneous analysis to those who need to make split-second decisions. You don't need me to tell you that the amount of data is skyrocketing -- and not just structured information found in databases, but also unstructured, incompatible data captured from electronic sensors, web pages, email, blogs and video.  By 2010, the amount of digital information is expected to reach 988 exabytes. According to IBM that's roughly the equivalent of a stack of books from the Sun to Pluto and back.

So what is SPADE? Again, it is a programming language and a compilation infrastructure, specifically built for streaming systems. It is designed to facilitate the programming of large streaming applications, as well as their efficient and effective mapping to a wide variety of target architectures, including clusters, multicore architectures, and processors such as the Cell/B.E. processor. The SPADE programming language lets  stream processing applications be written with the finest granularity of operators that is meaningful to the application, and the SPADE compiler appropriately fuses operators and generates a stream processing graph to be run on the System S Stream Processing Core. Details of the language are available in the SPADE Language Specification.

(Not that it is specifically related to SPADE, but I found the following interesting: As part of the System S development at the T.J. Watson Research Center, IBM researchers used fuzzy logic to develope a risk adaptive access control model described in this paper titled Fuzzy MLS: An Experiment on Quantified Risk--Adaptive Access Contro l.)

Of course, IBM isn't the only player in the stream processing game. RapidMind bases its platform on the SPMD (Single Program, Multiple Data) Stream Programming model while also mapping efficiently onto multicore CPUs from AMD and Intel. Coincidentally, both the RapidMind platform and IBM's System S are based on the Cell/B.E. multicore processor. Likewise, vendors developing GPUs (graphics processors) are also deep into stream processing. AMD and NVIDIA come to mind. The reason why is that GPUs are designed to accomodate a data-centric model (like images, video, and DSP) -- like that GPUs are originally designed for.

As for SPADE, it is an interesting language that's designed to solve a massive problem.