As SQL Experts and DBAs start looking at what is new in SQL 2008 R2, they inevitably try and figure out where StreamInsight comes from and how it fits into their existing view of the database world. The short answer is that the confusion is unnecessary as it does not fit in the world of most SQL DBAs. Allow me to briefly describe some of my observations – and remember, my standard disclaimer applies - I don’t have any MVPness or access to information that is not publicly accessible and my views are not endorsed (or maybe not even shared) by anybody.
- Microsoft Organizational and Marketing Structures
It would seem that members of the StreamInsight team have some background on the SQL Server team and hang around in the same building/group/division. How Microsoft works and how organizational structures influence products, packaging and marketing is unclear but it does influence the positioning of their outputs (just look at Entity Framework vs Linq to SQL). The organizational relationship, the marketing relationship and the technical relationship between the broad SQL product and StreamInsight should not be confused. Regardless of the organizational alignment, technically StreamInsight is quite distant from the rest of SQL.
- Relational Algebra != SQL
Because the clever people on the StreamInsight team worked with the database research team to apply relational algebra with temporal semantics in the StreamInsight product, it does not mean that the relational part is the ‘R’ in RDBMS/SQL Server. The relational model, relational algebra and the SQL implementation of the relational model need to be discussed in very different contexts.
SQL databases are so broadly useful that a SQL expert will naturally question where the latest SQL feature broadly applies to their problem space. StreamInsight is a product that has a very narrow application that falls outside the problem space that most SQL people are used to. Like SQL CLR and SQL Service Broker, which solve less common data oriented issues and people struggle to position, StreamInsight solves problems that are much, much more narrow and further away from the generic solutions that SQL provides.
SQL implementations are generally architected for a write once/read many scenario, resulting in the focus on storage of data and the concerns about disk IO, indexes and other things. StreamInsight does not really care about data being stored, it just reads the data (once) and forgets about it. This puts it outside the mental models that most DBAs have developed in terms of handling data. Just because you may want to store data once StreamInsight is finished with it, it doesn’t mean that a preoccupation with data storage should affect how and where StreamInsight is used. The clue is in the name – StreamInsight allows for queries (insight) to be run against a stream of data.
StreamInsight doesn’t run in SQL Server and is accessed using .NET libraries with some LINQ magic. Most DBAs gloss over anything .NET related (such as SQL CLR) and assume that it doesn’t belong in the database world. StreamInsight development is not going to be done by the database developers and T-SQL rockstars – you are going to need .NET development skills.
- This Kind of Solution is not Cheap
There are a lot of ways to process data and, when it comes to small amounts of data or where latency is not a problem, plain ‘ol SQL will be good enough. StreamInsight only comes in the Datacentre and Parallel Data Warehouse editions of SQL server – costing in the region of US$57,000. You have to have a real problem to solve to spend that sort of money per node.
Update: It seems that since this I created this post, Microsoft has changed the licensing on StreamInsight as pointed out by the comments below and detailed by Simon Sabin in his post ‘StreamInsight is in all editions (except express)’
StreamInsight, while not applicable to a broader market, fills a very important gap for Microsoft in enterprise engagements. The ability to offer solutions for complex event processing that are backed by Microsoft and ride on the success of SQL Server is a big deal and being part of SQL Server, instead of some side project, makes marketing sense.
- Data Professionals, Not DBAs
While the bulk of MS SQL instances are used for the database engine, there is a lot more than just SQL in the box. Analysis Services is SQL only in name and tools for BI, such as PowePivot step away from how we generally work with data. The concept of storing data in tables and viewing it with reports will be with us for a while, but there is more to data than just SQL. StreamInsight is part of a family of tools that deal with data that data professionals need to come to terms with – leaving database administration and backups to the DBA.
If you work with data every day and think you need to be more of a data professional than a DBA, I suggest you have a look at StreamInsight (even if it is just the documentation) and form your own opinion on where it fits in to the SQL world and, more importantly, your own environment.
Simon
@simonmunro