Hello and welcome to my blog.
It’s been a war of attrition between me and the Conchango establishment to get me to write a blog. Clearly I’ve lost. I’m currently sat in a hotel room in Bakersfield and since I have no interest in Country & Western music or guns, this seems to be the most constructive way of wasting time until the jet-lag kicks in and I can go to bed.
I have to declare in advance that I am no longer involved in development; hence this blog will be 100% fluff. If its geekery you’re after, I’m afraid you’ve come to the wrong place – however help is at hand, there are literally dozens of uber-techy types writing all sorts of stuff in strange languages on this very site.
I’m currently data architect on a global upstream program for one of the super-major oil companies. It all started about two years ago when they came up with the strange idea of implementing an information architecture that enabled Business Intelligence (BI) but used Services as the primary means of data integration. It was a Microsoft account, but we’d been put forward as implementation partners and I guess someone in Resourcing, saw “BI” in the description of the opportunity and I got the gig.
We flew out to San Francisco for a couple of days of workshops to go through requirements, principles, objectives, roles and the scoping out of the initial architecture. It was over breakfast in the Fairmont Hotel, that I met Simon Thurman from Microsoft. He introduced himself as the SOA expert for the project and he almost choked on his eggs Benedict when I said I was there to talk about the data warehouse we were about to build.
It goes like this – when you build a data warehouse, you connect to the most readily available source of the data you need, you understand the schema of the system you are connecting to, you drag out as much data as you can, you clean and transform it and push a nice shiny new version of it into the data warehouse, all done in the shortest time possible. Oh, and if the source of that data changes, that’s OK, we just rebuild the ETL against the new system. There isn’t much room for services, abstraction, systems of record or contracts in there. In fact as I’ve found to my cost, ETL developers tend to growl at you if you even mention these words in the context of data integration.
The flip side of course is that in an SOA, the true source of data is identified in a system of record (SoR), and that data is owned by that system. The schema belongs to the SoR and shouldn’t be visible to consumers of that data. The SoR should be abstracted from data consumers to ensure consumers are insulated from change. Data is exchanged via services and changes to the SoR should have no impact on those services or the consumers of the data they provide.
So even though there are aspects of BI and SO that appear to be mutually exclusive, what the customer was asking for sounded sensible. Provision of a single interface into an application that future-proofed consumers from changes to the system. BI without the need to store data that had already been stored multiple times around the business unit. Integrating data from a diverse application landscape.
I’ll talk more about how we resolved this fundamental issue (it involved beer), Bakersfield and occasionally data architecture in future posts.