Allow me to be the second to congratulate Jamie Thomson on reaching #2 in the Google search results for SQL Server Integration Services (SSIS).
Jamie is Conchango's celebrity blogger, and has developed something of a cult following in cyberspace. He sees himself as the David Beckham of the blogging world and has taken to wearing those big 70’s style sunglasses and, if we’re really lucky, a sarong around the office.
Anyway, back to San Francisco, we did our pitch to the client which went rather well. I tend only to speak when I think there is something worth saying – I put this down to the fact that I was an only child and had nobody around to listen to me when I was growing up. If this theory is accurate, Simon must be the youngest of fourteen, but between us we did a convincing job of letting the customer know that we knew what SOA and BI was all about.
At Conchango the EAI and ETL folks had had a number of discussions around where the two disciplines overlapped (mainly focused around the Microsoft technologies, Biztalk and DTS), and the market was converging in terms of the major ETL players acquiring EAI capability through acquisition. However much of the talk was idealistic – yes you can use Biztalk to load data into a data warehouse in an event-driven world, but its rare to come across an application that has been built to support that. You are more likely to find a system that doesn’t even record when data has changed in an environment where the users can change any of the data in the system. In those circumstances you are definitely in the heavy-lifting ETL world. However, there is a grey area between ETL and EAI products – as a message gets longer and less frequent, it begins to look like a scheduled data stream. As an ETL data stream gets shorter and more ad-hoc, it begins to look like a message.
There’s a pretty good overview on Microsoft’s integration technologies here.
So, we had spent time discussing the theory , but without a real customer requirement to prove or disprove our ideas.
After two days of workshops, Simon and I retired to a bar in Union Square and over beer started to piece together what eventually became Service Oriented Business Intelligence (SoBI). The key shift in our thinking was not to focus on where the disciplines of Service Orientation and Business Intelligence are mutually exclusive, but on the key strengths of each discipline and the areas where there is a synergy between them that can be exploited. From the outset we tried to be pragmatic – for example acknowledging that there will always be a case for pure ETL where large data volumes are required to be moved in batch.
We brought the ideas back to the UK and the brains of Rob Grigg, legendary socialite and leading architect from Conchango and Sean Gordon, architect in the Microsoft Scotland office, and a very handy man to have in a quiz team (excluding sport) were added to the mix.
The results of our efforts have been published as Service Oriented Business Intelligence in the Microsoft Architecture Journal: Be warned, it’s a touch on the dry side.
We’ve also had the opportunity to present a couple of times. Rob and I at the Connected Systems conference, and Rob and Sean at the Microsoft Architect Insight Conference
Have a look, we're always interested in feedback on this stuff.
That’s enough for now. I write this in the KLM lounge at LAX airport and my Heineken is getting warm.