Conchango is making strenuous efforts to reduce its carbon footprint. Initiatives include an energy efficient kettle, low-carb light bulbs, recycling bins, and a scheme to ensure all non essential electrical items are switched off in the evening. In a more radical step, plans are afoot to plug all the developers into a giant fart-catcher* – the obvious results of a sedentary lifestyle fuelled by cakes and coffee means developers are probably second only to cows in terms of methane production. The collective emissions will be processed, converted into bio-fuel and used to power the central heating system at Conchango-Central during the winter months.
The UK’s efforts to reduce the effect of the greenhouse effect came to mind whilst I was sat in my 4-litre V6 Ford Mustang rental car, nose to bumper on the 12 lane 405 interstate freeway that heads north from Los Angeles towards Bakersfield. When the booking was made I specified my meagre budget, the fact that there would be only me and a single suitcase and that I wouldn’t need anything too big. I was a little surprised at the result, however when I look out of the driver's window and all I can see are the wheel-nuts of the army of 12 litre monster trucks, maybe the assistant at Avis got it right. This is America after all.
Anyway, enough about the serious environmental challenges facing the planet, let's talk about data.
As the number and diversity of data sources and potential systems of record increases, the greater the integration challenge. It’s also increasingly unlikely that any new system will make any effort to solve the wider data integration challenge – this is an enterprise problem and not usually something that can be addressed within the budget of an individual project – hence without a data architecture strategy, systems are implemented in isolation and any required integration is done point-to-point.
Hence the need for a well defined data architecture; as interdependencies between systems grow, so does the importance of managing and maintaining these interdependencies in a structured and strategic way.
For a SoBI project to work a structured approach is required; SoBI demands common data exchange formats, master data definitions, official and trusted Systems of Record (SoR) for critical data, common and rationalized application tools and common interfaces to data and performance metrics. Data architects need to understand and plan for information flows through the IT environment much the same way that process engineers plan for the fluid flow through a plant or logistics specialists plan for movement of goods and services through a supply chain.
Above all SoBI demands a huge investment in data governance. Without a willingness on the part of the customer to clean up their SoR’s and to have a roadmap in place to migrate to applications that can support SOA, SoBI can’t work. The SoBI architecture includes a project database but this is a data cache – for business intelligence data or for data that simply cannot be accessed on demand from the SoR (for which there may be any number of reasons) – but definitely NOT a traditional data warehouse where data can be scrubbed, de-duped and cleansed. In SoBI, the data owners can’t hide a sloppy implementation or poor data quality behind a corporate data warehouse. Data integrity starts at the SoR.
Most companies that are yet to embrace an Information Architecture focus on a point-to-point type of integration. The problem with such an approach (even in systems which currently operate successfully) is that the participating systems are susceptible to the development of a complex lattice of inter-application connections, resulting in a large number of inter-application dependencies. In such scenarios it is not uncommon for the scale and the complexity of the inter-connections to become a barrier to the addition of new business functions.
In such scenarios, the nature and complexity of the interconnections, coupled with the fact that in many cases not all of the dependencies are well understood or well documented, can lead to a situation where the architecture becomes highly brittle; although the system continues to function in it’s present (stable) state, changes to any of the component applications can often have unforeseen and unpredictable consequences on other applications in the enterprise.
The point to point approach usually results in similar data being stored in multiple applications within the business. Applications broaden their scope from their intended purpose for the convenience of their users and because different logic may be applied to the data during the integration process, it is not unusual for different systems to hold different values for data that should be the same.
These symptoms inevitably result in a do-it-yourself approach to data integration within the business community. If data can not be made available in a timely fashion, or data sources cannot be trusted, it is natural for people to create their own data stores to hold and process the data they need. In some cases, departmental databases are used as data stores. Desktop databases such as Access are also convenient, but for most business users the data is stored in a semi-structured format in applications such as Excel.
It’s not unusual for a complex network of linked Excel spreadsheets to evolve alongside the point-to-point integration between applications. At this point the reliability or effectiveness of the data and any control over its integration, format or quality has effectively been lost.
Web services and related internet standards are enabling companies to utilize legacy applications and create a new web enabled workflow structures. These technologies are ready for use and provide an opportunity for companies to deploy them as part of the “plumbing” their architecture for data integration.
You may have guessed that I’m back in Bakersfield for a couple of weeks. Thommo fans will be pleased to learn that he is adapting well to his new home. I happened to be behind him in the queue for sandwiches when he ordered his foot-long lard & cheese Subway, complete with “toe-may-doe” and “bay-zil”. No, really. A Yorkshireman putting on an American accent - it almost made the trip worthwhile.
Happy New Year and I note that none of you sent me a birthday card.
*for those of you who have young children, I can heartily recommend the Walter series of books.