There is an interesting article today on The Reg authored by a researcher from Bloor Research who gives his views on IBM's lack of visiblity in the ETL arena. Whilst IBM's efforts (or lack thereof) in this area are unlikely to affect me as a Microsoft BI practitioner it is pleasing to see that researchers are beginning to wake up and see the benefit that ETL can bring to an enterprise and, moreover, the advantage of adopting a enterprise wide data integration strategy [In other articles I have seen mention of Data Integration Centres of Excellence within an enterprise - an interesting concept].
Previously ETL has been thought of as a mechanism for getting data into a data warehouse but there are bound to be lots more opportunities to leverage the heavy lifting capability of ETL tools within an enterprise as data volumes increase and with it the need for centralised data management. In time, I can see that the definition of an ETL tool is going to change, or at least the areas that it encompasses will.
In the article the author touches on an emerging need for synergy between 2 areas that were previously thought to be seperate, those being ETL & EAI/EII/Data Integration/Data Federation (call it what you will); it seems that in this day and age the lines are blurring. We at Conchango have started to think about these issues and will be providing thought leadership in this area. Watch this space!
The author states: "ETL ... probably isn't the best approach if you have heterogeneous databases across your organisation". In my humble opinion...this is wrong. ETL tools are, these days, designed to be able to access heterogenous data sources and they do it well. Indeed, the tool that I know and love, SQL Server Integration Services (SSIS), has a number of tools available for accessing data that does not necassarily exist in traditional relational format. I will concede that it probably doesn't handle unstructured data as well - perhaps that is the next step! However I'm sure that Don Farmer would want me to point out that SSIS together with BizTalk presents a pretty compelling picture for enterprise wide data integration. What is key here is that the tools are just that, tools. The overall message will be that the Windows Server system can fulfill this capability and it can probably do it cheaper than anyone else.
-Jamie