Welcome to EMC Consulting Blogs Sign in | Join | Help

Simon Munro

Where Microsoft StreamInsight Fits In

As SQL Experts and DBAs start looking at what is new in SQL 2008 R2, they inevitably try and figure out where StreamInsight comes from and how it fits into their existing view of the database world.  The short answer is that the confusion is unnecessary as it does not fit in the world of most SQL DBAs.  Allow me to briefly describe some of my observations – and remember, my standard disclaimer applies - I don’t have any MVPness or access to information that is not publicly accessible and my views are not endorsed (or maybe not even shared) by anybody.

  • Microsoft Organizational and Marketing Structures

It would seem that members of the StreamInsight team have some background on the SQL Server team and hang around in the same building/group/division.  How Microsoft works and how organizational structures influence products, packaging and marketing is unclear but it does influence the positioning of their outputs (just look at Entity Framework vs Linq to SQL).  The organizational relationship, the marketing relationship and the technical relationship between the broad SQL product and StreamInsight should not be confused.  Regardless of the organizational alignment, technically StreamInsight is quite distant from the rest of SQL.

  • Relational Algebra != SQL

Because the clever people on the StreamInsight team worked with the database research team to apply relational algebra with temporal semantics in the StreamInsight product, it does not mean that the relational part is the ‘R’ in RDBMS/SQL Server.  The relational model, relational algebra and the SQL implementation of the relational model need to be discussed in very different contexts.

  • Specific Solutions

SQL databases are so broadly useful that a SQL expert will naturally question where the latest SQL feature broadly applies to their problem space.  StreamInsight is a product that has a very narrow application that falls outside the problem space that most SQL people are used to.  Like SQL CLR and SQL Service Broker, which solve less common data oriented issues and people struggle to position, StreamInsight solves problems that are much, much more narrow and further away from the generic solutions that SQL provides.

  • Read Once

SQL implementations are generally architected for a write once/read many scenario, resulting in the focus on storage of data and the concerns about disk IO, indexes and other things.  StreamInsight does not really care about data being stored, it just reads the data (once) and forgets about it.  This puts it outside the mental models that most DBAs have developed in terms of handling data.  Just because you may want to store data once StreamInsight is finished with it, it doesn’t mean that a preoccupation with data storage should affect how and where StreamInsight is used.  The clue is in the name – StreamInsight allows for queries (insight) to be run against a stream of data.

  • .NET

StreamInsight doesn’t run in SQL Server and is accessed using .NET libraries with some LINQ magic.  Most DBAs gloss over anything .NET related (such as SQL CLR) and assume that it doesn’t belong in the database world.  StreamInsight development is not going to be done by the database developers and T-SQL rockstars – you are going to need .NET development skills.

  • This Kind of Solution is not Cheap

There are a lot of ways to process data and, when it comes to small amounts of data or where latency is not a problem, plain ‘ol SQL will be good enough.  StreamInsight only comes in the Datacentre and Parallel Data Warehouse editions of SQL server – costing in the region of US$57,000.  You have to have a real problem to solve to spend that sort of money per node.

Update: It seems that since this I created this post, Microsoft has changed the licensing on StreamInsight as pointed out by the comments below and detailed by Simon Sabin in his post ‘StreamInsight is in all editions (except express)’ 

  • MS SQL Pedigree

StreamInsight, while not applicable to a broader market, fills a very important gap for Microsoft in enterprise engagements.  The ability to offer solutions for complex event processing that are backed by Microsoft and ride on the success of SQL Server is a big deal and being part of SQL Server, instead of some side project, makes marketing sense.

  • Data Professionals, Not DBAs

While the bulk of MS SQL instances are used for the database engine, there is a lot more than just SQL in the box.  Analysis Services is SQL only in name and tools for BI, such as PowePivot step away from how we generally work with data.  The concept of storing data in tables and viewing it with reports will be with us for a while, but there is more to data than just SQL.  StreamInsight is part of a family of tools that deal with data that data professionals need to come to terms with – leaving database administration and backups to the DBA.

If you work with data every day and think you need to be more of a data professional than a DBA, I suggest you have a look at StreamInsight (even if it is just the documentation) and form your own opinion on where it fits in to the SQL world and, more importantly, your own environment.



Published Monday, January 25, 2010 11:29 AM by simon.munro

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS



Asvin said:

StreamInsight will be available with SQL SERVER 2008 R2 Standard and Enterprise editions too. You can follow the product blog at http://blogs.msdn.com/streaminsight/archive/2010/02/10/streaminsight-versions-and-sql-server-skus.aspx

March 24, 2010 3:55 AM

Shashi said:

Stream Insight has nothing to do with DBA's.  I spoke to the program manager of the Stream Insight, the reason it is release with SQL Server R2 is because their group that developed this technology falls under the SQL Server Team in the organization

May 28, 2010 5:37 PM

Fahad said:

CEP fall under Business Intelligence, examples are clearly provided in the doc itself. Comparing it with SQL DBA stuffs, SQL CLR etc., is not quite right too. Its problem domain is to understand the flow of data, visualize it etc.,


June 7, 2010 1:45 PM

Saggi Neumann said:


Since your post comes up pretty high on Google when you search for StreamInsight, you might want to correct the bullet "This Kind of Solution is not Cheap", since it comes with all editions except express, with different event processing capacities (see here: http://sqlblogcasts.com/blogs/simons/archive/2010/06/10/streaminsight-is-in-all-editions-except-express.aspx)


S. Neumann

June 12, 2010 7:07 PM

AndyAnsryan said:

You can use streamInsight standard with an enterprise license of SQL Server which is around 23K. streamInsight standard can process under 5,000 messages / second with a up to 5 second latency . You only need streamInsight  enterprise if you need to process more than 5,000 message a second with 1 second latency.

November 16, 2010 6:52 PM

Leave a Comment

Powered by Community Server (Personal Edition), by Telligent Systems