Welcome to EMC Consulting Blogs

It was nice to start the day drinking Starbucks in Seattle.  It has that feeling of authenticity about it like drinking Boddingtons in Manchester or getting head-butted in Glasgow.  Except of course, that Boddingtons is no longer brewed in Manchester since the Belgian owners moved production from Strangeways, where it had been for 200 years,  to South Wales.  That’s why Boddies isn’t advertised as the Cream of Manchester anymore, merely The Cream.  Not that I’m bitter (boom boom).

It was equally exciting to walk into the main hall for the first keynote this morning to be hit by an ageing guitar band belting out cover versions.  The PA was so loud that the distortion made hearing the between-songs banter difficult, but I think they were called The Deadly Hanjdob Quartet.  They finished with a rousing rendition of Rick Astley’s “Never Going to Give You Up”, which was bad when it was released in 1987, hasn’t aged well and suffers further when butchered by an old bloke sweatting  in a suit.  0830.  Slightly better than an 0530 semi-naked rain filled taxi encounter, but not much.  Mornings just aren’t for me.

So onto the keynote and the first hour was a little disappointing.  Stephen Elop, president of Microsoft’s Business Division spoke about Microsoft’s vision for BI.

This started by stating that 10 years ago, BI was a relatively immature discipline. 

Eh?  Microsoft’s offering (SQL Server 6.5) may have been immature but the rest of the world was moving on apace.  There were multiple ROLAP technologies out there , several OLAP vendors like Essbase, Oracle Express, Holos, Cognos and BusinessObjects, though at the time BO had this crazy idea that pulling data back to the desktop, building a micro-cube on the fly and using that to render reports was the way forward – it was called Desktop OLAP (DOLAP). 

I’d only just stopped coughing when he went on to say that Microsoft had transformed the market (fair enough) and with Analysis Services 2008 they had a product which in scalability terms was “virtually unlimited”.    Hello?  I’m not even going to rise to that one. 

So having let myself get wound up by the marketing pitch, I was knocked sideways by the announcements from Ted Kummert, VP of Data and Storage platform.  First off we had a demo of how the DatAllegro purchase might work with SQL Server.  A 24-server shared-nothing SQL Server cluster with DatAllegro orchestrating things was demoed against a partitioned 150TB, one trillion row fact table database using some simple Reporting Services reports to return data in seconds.  OK, so it was a sanitised demo and the queries were selected to run quickly but 150TB in SQL Server?  Better news for us is that Microsoft is working to select standard hardware for a reference architecture and the storage aspect is covered by EMC.   It brings a warm glow to the Galloping Data Architect's cockles - if only I knew the company song. 

It was suggested that this would be licensed as a separate SQL Server SKU and if you’re looking for more information, its called Project Madison.  Of course a cynic would ask what the point of this expensive acquisition was if they already have “virtually unlimited” scalability in SSAS 2008 – 150TB cube anyone?

We got a demo of Project Gemini from Donald Farmer; this is the Microsoft vision of “Self Service BI”, where users can load data from varying sources into Gemini using an Excel Add-in and then create pivot tables based on the data sourced by Gemini.  Results/reports/applications can be published back to SharePoint for use by other users.  In his demo, Donald pulled back a 20 million row fact table into Gemini (which sorted and filtered the data almost instantaneously), combined it with an “external” data source, showed the integrated data model created dynamically in the background, created a pivot table report and simple drill down dimension structures in Excel and finally formatted and published the results as a reasonably tidy looking dashboard application back to SharePoint in a matter of minutes.

The UI that is created can include dimensions as slicers for basic report filtering and any selection within the slicer propagates to all pivots and charts on the sheet.  Simple drill down is also supported (within the context of your Gemini data set).

 Apparently SSAS is the engine doing the work behind the scenes so there must be some sort of dynamic cube created which attaches itself to the workbook to serve up the data so quickly.  It was a blinding demo, typical of Donald who is clearly excited about the capabilities and power that it can bring to the desktop – the 80% of users that Stephen Elop targeted.  Microsoft is putting their subprime mortgage on Excel.

The Microsoft view is that it is better for users to go out and get their own data that for that overworked IT department to source it for them and add it to the data warehouse which would take forever to prioritise and implement.

After a few hours reflecting on this I began to have a few doubts.

The Microsoft BI vision is one of guided analytics, lightweight dashboards with Excel as the tool of choice for "power users" and Reporting Services becoming more of a user-oriented tool for self service reporting.  I get the impression that Microsoft is targeting Business Objects here and form is definitely taking precedence over function.  ProClarity is sadly dead and train of thought or high-end analytics doesn’t appear to get a look in.

I’m not sure I want all my users going out and dragging multi-million row result sets into Excel to work on locally.  If I have 500 users in my company, how many times would the same result set have to be stored into multiple Excel reports for things to become unmanageable? Repeat that for the set of available data.  And all this posted back to SharePoint? 

Where is the single version of the truth in this architecture?  I’ve just spent 4 years of my life trying to convince users to stop using Excel as a data store and here are Microsoft positively encouraging it.  Hell will freeze over before this capability is used responsibly in most organisations.

Is it really too much to ask for external data that is useful to the business to be included in the data warehouse?  Surely we have gone past the point where we spend 3 years building monolithic databases that don’t have the flexibility to incorporate new requirements as they are discovered? 

I also have severe doubts about having users integrate data at the UI.  Again, how many users will have to pull the same external data source into their spreadsheet before that has cost more than having it sourced once centrally.  Where is the ability to share or reuse that integration?  Where do the metadata mappings take place – because in the real world external reference data isn’t going to exist in the same context as the stuff inside the organisation, someone will have to provide mapping.  I have this horrible vision of Stratature being served up in Excel for users to do this on the desktop.  Personal MDM.

The other flaw in the argument is that the IT department is supposed to monitor what the users are up to and identify reports that are being shared by multiple users, have become business critical or which are becoming too large and bring them back into the IT department.  This is obviously a different IT department to the one that was too busy to bring that external data source into the corporate data warehouse in the first place but have time to watch chaos unfold on SharePoint.

All the BI focussed updates will be packaged into a SQL Server update code named Kilimanjaro.  This is separate to the 24-36 month cycles of major SQL Server releases and the date given for release is first half 2010.

In other news, there will be an October feature pack featuring Attunity connectors for Oracle and Teradata and the release of Report Builder 2.0 which makes several significant enhancements to the usability of the product including a rather nifty looking shared component library which allows users to select components developed by others and also highlights any changes made back to the original author to give them the option of pulling the revised version into their report.  All the handling of data sources and fields is done automatically in the background.  This functionality comes from Microsoft’s acquisition of 90 Degree.

Microsoft have also bought Zoomix which will appear in the next major release as SQL Server Data Quality Services, part of the SSIS stream.  This sounded interesting and fills an obvious gap in the portfolio but no further details were given today.

So, as I strolled back from the Microsoft BI partner event last night, where its worth saying I was in a minority of one with most of these concerns, I reflected on what seems to the revival of DOLAP as the future of Microsoft BI (I'm really hoping to be proved wrong on this one), the dumbing down of analytical capabilities in the front end and a world where 150TB data warehouses are fronted with Reporting Services.    Time to dig out my old MicroStrategy 4 certification perhaps?

The really scary thing though – other than the demographic that walk Seattle’s streets in an evening which seems to be 30% business people, 30% geeks and 30% tramps and hobos - is that most of this stuff is at least two years away.  By then we will be 13 years into Microsoft’s strategy for BI, the competition will have had plenty of time to react to these latest announcements anyway and there will be a 60% chance that I have been bored or bludgeoned to death on my way back to the hotel.

    I'm back!

It’s been a while since my last blog post but reader-pressure has forced me to come out of retirement.

 ….I couldn't believe I was only at the half way point. After bitter arguments, stress, rain lashing into my face, arms aching and sweat running into my eyes due to being overburdened by the amount of gear I was carrying, I started the long trudge back to the agreed meeting point….

This was my journey back to rain soaked Preston train station after finally managing to locate an off-licence in the near vicinity. I'd received emergency in-transit telephone instructions from Dan Perrin and 24 cans of Stella were duly acquired and somehow forced into the non-existent space in my hold-all and rucksack.

The purchase almost caused a diplomatic incident when the shopkeeper refused to let me take all 24 from the cold-bin, insisting "you can't take them all, they are for customers", and then made me unpack them all to double check I wasn't stealing any.

Lesson 1: Get the beers in, in advance of departure.

Half an hour later a big white van pulled up at the other end of the car park. My eyesight isn't what it used to be but I could see the words "Budget" emblazoned down the sides. People I didn't recognize started pouring out, holding their backs and stretching. "Can't be them", I thought, "I'll wait for the van with the 'Luxury' signage". Unfortunately they turned and headed my way.

The journey from Egham had taken almost 8 hours. It's about 240 miles, so they would probably have made better progress if they'd cycled. Unfortunately Michael Knight had insisted on stopping at every service station so he could search out a McDonald's Vanilla Thick Shake, and the vans were limited to 60mph. They also had really uncomfortable seats in the back.

I'd been collected by Conchango Team 2 (call sign "Asia" for the rest of the challenge). My team ("Africa") had continued ahead up the M6 in search of a service station with the widest selection of fast food outlets. Michael Knight still needed a McDonald's. Dan Perrin needed a beer.

After introductions, we rejoined the M6 and eventually caught up with Africa at the next services. Unfortunately there were no McDonald's but plenty of other unhealthy options presented themselves, which were gratefully consumed.

At this point I swapped vans to join my team, leaving a few cans of Stella behind to help numb the pain of the van seats for team Asia.

Beer consumption on the move is a delicate balance between the brain and the bladder. I managed a beer in each van before eventually dropping off…..for about 3 minutes, by which time the rigid, upright, solid seats and badly positioned headrests meant I woke up with a crick in the back of my neck. It's difficult to imagine the moment when the person in charge of designing those seats stood back, looked at his handiwork and announced to the world that he's achieved what he set out to do. Unless of course the Torquemada family have gone into the van refitting business.

Lesson 2: Make sure the vehicle wasn't assembled by a sadist. You're going to spend 3 days living in it.

The journey up the M6 was fairly quiet. I'm not sure whether it was boredom, fear or just the fact that nobody liked me, but conversation was scarce. Fortunately Jon George has had the decency to grow a comedy hair style, so staring at the back of his mullet kept me entertained for several hours as it swung majestically from side to side in tune with the bumps in the road.

We passed an hour playing Trivial Pursuits. Saz had brought along the "family fun" version so we had 150 questions along the lines of "What colour is mentioned in the title of the children's TV series Blue Peter?" and "Who is the Queen of England?" I think I came third. Team Asia played the "yellow car" game for three hours, which involved looking for yellow cars on the motorway and shouting "yellow car". Are you getting a good sense of the boredom yet?

We did at one stage manage to connect an MP3 player up to the van's sound system. The speaker (singular) sounded like it had been assembled by stretching cling-film over an empty margarine tub but we did manage to enjoy something other than the sound of a diesel engine for a while. The MP3 player clearly had it in for us as it served up "Misty Mountain Hop" (Led Zep), "Rain" (The Cult), "King of Pain" (Police), "Rain When I Die" (Alice in Chains) and the sing-a-long classic that is Cradle of Filth's "Thank God for the Suffering". Eventually Mel took control of the music at which point we got an hour of only hearing the first seven seconds of each song before she skipped to the next track. Why do women do that?

The last hour driving into Stirling was through torrential rain. I was sat on the first row of seats behind The Mullet and the rain was so heavy that I couldn't see the road. At around 2330 we did finally make it to Stirling Travel Tavern. Allan Partridge eat your heart out. As I staggered in clutching the remnants of a warm can of Stella, soaking wet from the rain and feeling queasy from the excess of deep fried food I'd consumed over the last seven hours, I was starting to feel decidedly Scottish. The receptionist informed us that the local pub would be open until midnight. A hopeful look around revealed no takers so I consoled myself with a can of Stella in bed before getting the last few comfortable hours of sleep for the rest of the weekend.

Saturday morning started with breakfast at the local services. All the boys had a fry-up and all the girls had exotic forms of Muesli. And I include Michael Knight in the latter group. It was then back to the Travel Tavern, pack up and head back to the van. Now, at this point I decided not to squeeze the pillow I'd been carrying with me back into my bag, I just carried it to the van (there wasn't really room with all the remaining Stella in there). This led to accusations of "Travel Tavern pillow theft" for the remainder of the weekend, though I suspect much of this was borne out of jealousy for what was a spectacularly good idea given the lack of comfort offered by the van.

Lesson 3: Take something with you that will make sleep easier and more comfortable. Don't worry about the smell.

The proof of this being that as we drove into Fort William, the ladies insisted on an impromptu shopping expedition and Saz came back with enough sleeping accessories to run a B&B out of the back of the Africa van; for team Asia, it looked like Lorraine and Linda had bought a king sized travel water bed with duvet and pillow set. I'm sure it was only the lack of available time that saved the vans from being decked out with floral curtains, scatter cushions and a host of scented soft furnishings.

Just for the record, the pillow came from the Tardis (yes we have a Tardis) in the playroom at home. Amongst other things, it's the emergency store for spare bedding whilst the kids are going through potty training. My wife wouldn't let me take a decent one and she sent me off towards the playroom with the words "Sod off. You can take one of the old ones but it might smell of p***".

For those in team Africa that borrowed the pillow during the course of the weekend – I should probably have told you that earlier.

We had one more stop on the way to the meeting point - the excitement of an endless supply of Mrs Perrin's lemon drizzle cake meant Team Asia couldn't contain themselves for the whole journey so we had to stop for them to relieve themselves.  The girls set off on a two mile hike to find an appropriate level of seclusion, unfortunately to no avail - other than for Heidi who could wait no longer and so provided brief roadside entertainment for the traffic heading towards the Scottish Open Golf tournament.  I believe the Scottish Tourist board have set up a hotline number for anyone travelling on the A82 that day who now need counselling.

Lesson 4: Make use of all available facilities before you get in the van.

And so we finally arrived at the Glen Nevis visitor centre at the foot of Ben Nevis.  I don't know what the Nevis brothers did to get so famous, but their mother must have been very proud.  However if I was Glen, I would be somewhat distressed to see my brother get a mountain named after him whilst all I got was a small car park and some public toilets.   

All that remained was to avoid eating Lozza's protein balls, strap on the equipment - which included a pair of red Y fronts (to be worn on the outside of the trousers obviously), a gold cape and a mask - get our kit checked by the marshalls, clock in at the start point, pick up a radio and we were off!

To be continued.....

-----------------------------------------------------------------------------------------------------------------------------

Find out what we were doing here.

Find out why we were doing it here. 

For the last 20 years, the “largest, heaviest and most unusual object” ever found at the summit of Ben Nevis is a piano.  On  July 14^th, that record will be broken not once, but twice by the 2007 Conchango 3-Peaks Challenge team*.

The challenge consists of climbing the UK’s three highest mountains – Snowdon, Scafell Pike and Ben Nevis – in a 24 hour period.  The time objectives are as follows:

1. Ben Nevis – 5hrs 30 mins
2. Drive 6 hours to Scafell Pike
3. Scafell Pike - 4hrs 30 mins
4. Drive 4hrs 30mins to Snowdon
5. Snowdon – 4hrs

Conchango will enter two teams of five people.  The concept seems to be simple.  We drive to the middle of nowhere at some ungodly hour, spend 5 or so hours climbing up a hill, then back down it, then spend 5 hours travel-sick in the back of a transit van, before repeating .  Twice.  What’s worse is that two of the climbs will be in the dark and one will be in bad weather so we won’t actually be able to see where we’ve been.  The proof will be the seeping blisters, aching limbs and the slightly warm feeling that results from an exhaustion-based loss of bladder and bowel control.   If I have the misfortune to suffer a Paula Radcliffe moment myself, I’m hoping we’ve at least made it across the border into Wales**.

Why anyone would volunteer for this is something of a mystery.  The mystery is compounded when you consider that most of the people who will be involved don’t know each other.   Or had any idea who else might volunteer before they volunteered themselves.  Here are the rest of the team: Shane Collins, Dan Perrin, David Höhn, Michael Knight, Peter Hay, Sarita Ward, Michael Jones, Heidi Lin, Peter Murphy.  I know one of them.   I’m guessing that amongst the rest there is at least one Australian, a sci-fi buff, some quirky facial hair that would get you killed in Burnley,  the biggest geek in the company and someone with no interest in football.  And probably the same amongst the men.

We’ve been advised to start training 12 weeks before the event, with some basic groundwork eventually building up to hard training to build up stamina.  Since we’ve only just signed up for the challenge, there are only 8 weeks left.  Personally I think it’s important to get the basics right, so I’ve opted to miss out the last 4 weeks of training rather than skip the early essentials.   I’m now jogging to the pub on a Thursday evening (50 yards) and because I live in the North, we have stairs on the inside of our house, which I now climb two at a time.  Eight weeks of that should see me in peak physical condition.

Those of you familiar with Conchango will have spotted the distinct lack of involvement from the Conchango board of directors.  Advancing years, bulging waistlines and 30 years of sendentary lifestyle rules out some.  Others would struggle to make the distance in platform shoes.  Some would find it impossible to travel in such working class conditions and are awaiting the release of a Porsche-Transit before volunteering.  Chris Saul did volunteer at one point but later withdrew when the promise of an M&S retail outlet at the top of Snowdon turned out to be a hoax.  I’m sure in their absence they will at least show their support by sponsoring the team and that each of them will individually match the current single highest donation we have received.

Let’s not forget the dedicated support team who will accompany the hikers on the challenge.  The volunteers are Jon George, Saqib Barlas, Lozza and Guy Sturgess.   If you think spending 24 hours within six feet of nine people you have nothing in common with, halfway up a mountain in the dark, whilst someone hammers nails into your legs doesn’t sound like much fun, spare a thought for these four.  Their job is to drive the rest of us between locations – that’s 10 or so hours of ten people whining – interspersed by 15 hours of waiting in a cold transit van at the bottom of a foggy mountain and cooking a curry for 12 (thanks Saqib).  Unless of course the romance of the bleak landscape becomes overpowering, in which case – for Lozza at least,  it could be like jumping into a barrel of warm play-dough.

Our reward for completing the challenge is a disco.  I’m sure after 24 hours of no sleep, climbing up (and down) three mountains, travel sickness, incontinence, sprains, boils, blisters, aching limbs and an unending stream of fascinating conversation about C# development, we will all be looking forward with relish to spending four hours dancing to 70’s classics with a bunch of tree-hugging bearded 50-somethings with bad dress sense and a strong opinion on green issues.  I suppose it’s for times like these that beer was invented.

And so on to the point of this post – sponsorship.  We are raising money for CARE international.  CARE works in 70 countries with more than 48 million poor and marginalized people each year to find a way out of poverty.  It is one of the largest development agencies in the world with 91% of it’s donations going directly to help fight poverty, it’s marvelous, heartwarming stuff.  So, if you think CARE are great, please send them some money via the link below.

For those of you who couldn’t care less about CARE, please spare a thought for your colleagues that have signed up for this.  Whether trudging up a mountain in the rain or rolling about in the back of a transit van, they would appreciate your support so please do sponsor them.

Finally, if you don’t care about CARE and have never heard of any of the people taking part in the challenge, the sponsorship page allows you to enter a name and a line of text along with your donation, so please use this as an opportunity to spend a couple of quid to anonymously insult someone you don’t like at Conchango, safe in the knowledge that the whole of the company will see it.

http://www.justgiving.com/conchango2007

*Dan Perrin's beer gut and my backside.

**This is not a slur on Wales.  It is an acknowledgement that if an unexpected deposit of natural fertilizer occurs, where better to distribute it than the fabulous Welsh countryside where it can help enhance the already stunning natural beauty of the landscape.    Alternatively, take your mind off it by reflecting that Italy finished above you in last season’s Six Nations.

It has been a while since my last post.  There are a number of reasons for this - lack of time, the fact that blogging comes quite low on my list of fun things to do and mainly because the blog posts I had lined up on my laptop were dealt a severe blow following a United Airlines flight from LAX to Oakland (where my laptop bag was taken care of by their Valet Service) and a taxi ride from Oakland to San Ramon (where my laptop went in the boot).  I can't actually prove when or where something speared a hole through the external casing, however what I can say is that I sent emails immediately before I boarded the flight at LAX and the next morning the laptop was dead.  I'm also fairly sure that there wasn't a man hidden in the boot of the taxi with a portable hammer drill or someone jumping up and down on it in my hotel room while I slept.

So, I've been back to Bakersfield.  Another quite eventful trip, starting with me arriving in San Ramon at midnight and being unable to get into the flat I was supposed to be borrowing.  This meant a lengthy cruise round San Ramon by taxi discovering that all the hotels over 1 star were full, and eventually being dropped off at hobo-motel, checking into a room for the night, pushing my way past the local chavs hovering round reception and trying desperately to get some sleep despite the hip-hop festival that was unfolding three rooms down the corridor.   That was the first night, the second, I got an earthquake, but at least I had managed to break into the flat by this stage and so was able to cower under an overturned settee until the shaking stopped.

From San Ramon, work had kindly booked me on the 7am flight down to Bakersfield - so a 4am start on a Saturday morning.  The payback was the fact that I had the whole plane to myself on the flight down.  As the stewardess said, "nobody wants to go to Bakersfield on a weekend but we have to fly down there because there are quite a lot waiting to get out".  Imagine what you'd do with a private jet....well, I slept.

As I casually mentioned the fact I was cruising around in a Mustang on my last visit, it's only fair to report that awaiting me in the Bakersfield Avis car pool was a cream Chrysler PT Cruiser.   It looked like a mobile cream puff and with the high driver's seat, tinted windows and hugely underpowered engine felt like I was driving round in a clown's car. The early arrival meant I couldn't check into the hotel, so there was nothing for it but to hop in the fun-mobile and cruise Bakersfield. 

I would have felt less self-conscious wearing a clown's costume.  Fortunately for me, David Seymour - stalwart of Conchango's Business Team and winner of the "Conchango's Hairiest Man" competition at the last Christmas party - has joined the team in Bakersfield.  Size 16 shoes, garish shirts and oversized suits are a core part of his business-casual collection, so once David had leant me the necessary outfit I was good to go.

This post introduces one of our integration patterns, we have lots of these and this one shows how a legacy application or an application that expects to receive data using a locally defined reference data dialect rather than the common reference definitions that our MDM solution contains. It takes data from a service which communicates with a legacy system which itself stores data with a locally defined reference data dialect.

I'll use an Oil & Gas example – well production data (the amount of oil produced by a well) . We have a production facade which will return data in a common dialect and our MDM service will provide a translation and cross-reference service that is available to data producers and consumers who may identify oil wells differently to the common definition in our MDM application. This is in line with service oriented principles which encourage a common dialect to describe common entities – this is the basis of the ability to easily integrate SO services because the service (or façade) does not know about the legacy dialects.

To overcome the fact that it does not understand the common definition of wells, our consuming application may need to build an adaptor (a facade in reverse) to consume the common format and determine how it maps to the local client dialect in conjunction with the cross referencing provided by our MDM solution. In this way, only the consuming application is concerned with how to translate the common dialect into their own legacy or internal dialect – again this is in line with service oriented principles of encapsulation, where the translation functionality is kept as close to the required service as possible and not spread around to the possibly multiple service facades. Such distributed transformation functionality would increase the management overhead and reduce the agility of the overall solution. In scenarios where an application may request data of the same type from multiple different systems, the transformation logic would need to be present in every system that the application might use.

Similarly, the service façade will only accept requests in common dialect so the client application may need to go through an adaptor to translate the client request from local to common dialect when the data is returned from the facade. 

Adaptors are specific to the consuming system and are not part of the service façade.  They care only about translating the calling applications' dialect into common dialect.  MDM sits in the centre of the architecture and, by being aware of all dialects, can offer services to both adaptors and facades.

If a SoR speaks common dialect, its facade doesn't have to worry about MDM, it can just issue a straight pass-through query to the SoR and return the result to the calling application. In our example, this isn't the case – the SoR has its own way of identifying wells which also doesn't match the commonly agreed definition, so MDM is required to translate transactions from the SoR into the common dialect as they are exposed by the service facade. 

The consequence of these patterns is that our MDM solution will need to be aware of the dialects available within the business but the facades will not.

Here's the pattern. Underneath it are descriptions of the numbered steps.

1. Forecast application issues request for data – "give me yesterday's production for the well I know as W01"
2. Adaptor captures the request and issues call to MDM service to translate local well identifier into common dialect – "what's the common name for the well I know as W01?"
3. MDM service returns well identifier in common dialect – "It's called WELL001" 
4. Adaptor forwards query to Production service façade – "give me yesterday's production for the well WELL001"
5. Production Service Façade issues call to MDM service to translate common well identifier into SoR dialect – "What does the allocation system call WELL001?"
6. MDM service returns result in SoR dialect – "it's called PRDWL1"
7. Façade issues query against SoR – "give me yesterday's production for the well I know as PRDWL1"
8. SoR Returns results to facade – "PRDWL1 – 1000 barrels"
9. Façade issues call to MDM service to translate SoR well identifier into common dialect – "what's the common name for the well I know as PRDWL1?"
10. MDM service returns results to façade - "It's called WELL001".
11. Façade returns results to adaptor - "WELL001 – 1000 barrels" 
12.  Adaptor issues call to MDM service to translate common well identifier into local client dialect – "what does the forecasting application call WELL001?"
13. MDM service returns results to adaptor – "it's called W01" 
14.  Adaptor returns the results to the forecasting application "W01 – 1000 barrels" 

On the face of it, it looks complicated, and for a single application calling a single system of record, it is - in fact there is no point building services if no other applications are interested in the data you expose, you may as well stick to the simple point-to-point approach.  Where the value of this pattern becomes apparent is when there are multiple consumers of the data, each potentially with its own set of reference data, combined with multiple SoR's behind the facade serving up the data from different systems on different assets in different businee units.  In this case, you have eliminated a complex lattice of point to point integration solutions and by isolating producers from consumers, made a brittle architecture much more resilient. 

The group I’m working with is responsible for defining strategy, principles, practices and re-usable components for upstream data integration projects globally.  Currently I spend half my time in the UK and half in California. 

It’s not as glamorous as it sounds.  Nice as it would be to have an oilfield on the outskirts of San Francisco, unfortunately its in the middle of the Californian desert, close to Death Valley and just far enough away from Vegas, San Fran or LA to make them weekend-only destinations.  And on top of that, it seems I’m flying myself to an early grave!

In the UK I spend my time on delivering stuff for the global program that I work on.  When I’m in California its working on the latest project to adopt the SoBI approach and principles.

If you’ve read the first two blog posts and managed to wade through the SoBI white paper, you may be wondering what we have been doing in the 12 months since it was conceived, written and published.   The short answer is implementing it.

A longer answer is that this is a big project – the project in Aberdeen is approaching two years since conception with around 25 people staffing it at peak times.  The architecture is new, and fundamentally delivering an architecture is in itself quite a challenge in Upstream O&G.  Upstream is the part of the Oil business that delivers the dollars.  Basically, the more hydrocarbons that can be extracted, the more revenue the company makes.  Historically this has led to quite a tactical approach to integration – the relative wealth of Oil & Gas companies ensures that best of breed applications are bought, but they are implemented to solve specific business problems, often without a holistic view on data integration between applications (“Give me the shiny things NOW!!”).  It also means that in comparison to companies in other verticals I've encountered, upstream O&G has lots of applications out there, many of them storing similar data and accessing data from other applications in a very tactical, point-to-point manner. 

Upstream Oil and Gas is also a very complex environment in which to work.   In almost 10 years of working in the area of data warehousing and business intelligence, I’ve never come across a combination of:

• an environment so littered with candidate data sources
• systems implemented so myopically
• with such a reliance on Excel as a data store
• supporting such a complex business

Add to that a new architecture and throw in agile as a delivery mechanism and you have a significant challenge.

"Upstream" refers to the process of exploration and production of oil up to the point where it is transferred on for sale.

I’m surprised by the number of people who think that producing oil involves drilling a straight hole down to a conveniently located pool of oil and catching it as pressure forces it to the surface, before selling it on at huge profit.

Firstly, there is no magical pool of oil.  What you have, are layers of sedimentary rock into which the oil flows and sits, along with water and gas and lots of other chemicals in pores between the molecules of rock.  Think of it as the process of sucking water out of a sponge.  The amount of stuff that the rock can hold is its porosity.    The ease at which the stuff can flow through the rock is its permeability.

Skipping over how a potential oil reservoir is identified, the next thing to do is to drill.  Depending on where you are in the world, this can be an expensive exercise.   In the Californian desert where the oil reservoir isn’t too far below the surface, things aren’t too bad, but if you are working in deepwater exploration, things are significantly more complex and expensive. Take a look at the Jack #2 well announced by Chevron in deepwater Gulf of Mexico where the well was drilled to a depth of 28,000 feet.  To put that in perspective, Mount Everest is 29,028 feet high.  This is space age technology, and it doesn’t come cheap.  Cost estimates for field development in the vicinity of Jack #2 are about $80-120 million per well drilled, with an additional $1.3 to $1.5 billion for subsea facilities.  Upstream Oil and Gas is very much a risk/reward business and we tend only to hear about the rewards (which admittedly can be substantial).

It’s also surprising to find out that wells don’t have to be sunk vertically.  The drills are directional so it’s possible for a well to take a circuitous route to the oil bearing formation.  In some circumstances wells are drilled horizontally – in this case the well starts off vertical but curves as it goes down eventually flattening out to a horizontal line.  It’s also possible that each surface well could have a number of wellbores within it – i.e. multiple paths leading off from the main well to different parts of a reservoir.  Each wellbore also needs to be completed – completing a well consists of a number of steps;

• installing the well casing
• completing the well
• installing the wellhead
• installing lifting equipment
• treating the formation

The process of completion usually involves blowing holes in the casing at the bottom of the wellbore to expose it to the oil bearing rock.  Each wellbore can be completed multiple times and you may wish to complete the well such that hydrocarbons from two or more formations may be produced simultaneously, without mixing with each other.

So what’s left?  Stand at the top of the well with a barrel and collect the oil as it flows to the surface?   If you’re lucky, the pressure in the reservoir will force fluid to the surface.   Usually however this isn't the case, and even when oil does flow under its own pressure, this reduces over time as the fluid is extracted, reducing the pressure in the reservoir.  This is your bottomhole pressure and it’s something you have to keep an eye on.  It’s checked regularly by performing a well test which gives an indication of the pressures and temperatures at the bottom of the wellbore and gives an indication of the theoretical production rate of the well.

A good reference for O&G buzzwords is the Schlumberger Oilfield Glossary.

I think that’s enough for one post.  Next time I’ll cover all the stuff that happens once the fluid is actually out of the ground – and what on earth any of this has to do with data integration.

Some news to finish.  The head of the Technology Team, Iyas AlQasem, has just informed us that Conchango will end 2006 larger than it’s ever been.  I presume this relates to number of employees now in the company rather than the average weight of its consultants, though it should be noted that with so many developers working in Egham during the day, the man who runs the donut shop is now driving round in a Bentley Continental. 

Hello and welcome to my blog. 

It’s been a war of attrition between me and the Conchango establishment to get me to write a blog.  Clearly I’ve lost.  I’m currently sat in a hotel room in Bakersfield and since I have no interest in Country & Western music or guns, this seems to be the most constructive way of wasting time until the jet-lag kicks in and I can go to bed.

I have to declare in advance that I am no longer involved in development; hence this blog will be 100% fluff.  If its geekery you’re after, I’m afraid you’ve come to the wrong place – however help is at hand, there are literally dozens of uber-techy types writing all sorts of stuff in strange languages on this very site.

I’m currently data architect on a global upstream program for one of the super-major oil companies.  It all started about two years ago when they came up with the strange idea of implementing an information architecture that enabled Business Intelligence (BI) but used Services as the primary means of data integration.  It was a Microsoft account, but we’d been put forward as implementation partners and I guess someone in Resourcing, saw “BI” in the description of the opportunity and I got the gig.

We flew out to San Francisco for a couple of days of workshops to go through requirements, principles, objectives, roles and the scoping out of the initial architecture.  It was over breakfast in the Fairmont Hotel, that I met Simon Thurman from Microsoft.  He introduced himself as the SOA expert for the project and he almost choked on his eggs Benedict when I said I was there to talk about the data warehouse we were about to build.

It goes like this – when you build a data warehouse, you connect to the most readily available source of the data you need, you understand the schema of the system you are connecting to, you drag out as much data as you can, you clean and transform it and push a nice shiny new version of it into the data warehouse, all done in the shortest time possible.  Oh, and if the source of that data changes, that’s OK, we just rebuild the ETL against the new system.  There isn’t much room for services, abstraction, systems of record or contracts in there.  In fact as I’ve found to my cost, ETL developers tend to growl at you if you even mention these words in the context of data integration.

The flip side of course is that in an SOA, the true source of data is identified in a system of record (SoR), and that data is owned by that system.  The schema belongs to the SoR and shouldn’t be visible to consumers of that data.  The SoR should be abstracted from data consumers to ensure consumers are insulated from change. Data is exchanged via services and changes to the SoR should have no impact on those services or the consumers of the data they provide.

So even though there are aspects of BI and SO that appear to be mutually exclusive, what the customer was asking for sounded sensible.  Provision of a single interface into an application that future-proofed consumers from changes to the system.  BI without the need to store data that had already been stored multiple times around the business unit.  Integrating data from a diverse application landscape. 

I’ll talk more about how we resolved this fundamental issue (it involved beer), Bakersfield and occasionally data architecture in future posts.