Welcome to EMC Consulting Blogs

If I interpret the Wikipedia entry correctly, Moore’s law started as an observation that perhaps transitioned into a self fulfilling prophecy: “…[this law] is now used in the semiconductor industry to guide long-term planning and to set targets for research and development.”. I wonder if there isn’t a more human explanation and lucky dose of coincidence for the longevity of this law – i.e. it was initially true for one reason but soon gave way to another that happened to be similar.

If our civilisation, and all its economics, generates the demand for improving products and services then is it also naturally limited in its ability to successfully cope with a certain rate of change? And, therefore, will the demand for change remain constant? If throughput in innovation is properly matched to this demand then it would be no surprise that we don’t deviate much from Moore’s law (or variants of it). I suppose, in terms of Theory of Constraints, I am describing our current state of civilisation as the constraint to adopting innovation rather than a constraint of our ability to innovate. After all, why invest and innovate faster than you can ship it?

If, tomorrow, Intel brought to market a chip with 1000 cores what would we do with it? We couldn’t really exploit it. Throughput would have massively outmatched demand and Intel would have missed the opportunity to sell 16, 32, 64, 128 core systems. What if Canon had released a 28MP DSLR as one of the early generation digital cameras? Not only would USB 1.0, hard drives and narrow band internet connections have struggled but the mass market computers couldn’t cope with that resolution. Again innovation throughput would have exceeded demand.

Another scenario of innovation throughput exceeding demand would be Canon releasing a new camera every month doubling resolution. Logistics and marketing are obvious impediments but also consumers couldn’t afford that rate of change. It’d be a Greek geek tragedy and a form of “future shock”. Medical innovation also often has to throttle back whilst cultural/religious/legal alignment takes place.

War is often cited as a time that demand for innovation rises, and we see innovation throughput rise too. There is usually less impedance as activities such as marketing get dropped and people’s willingness to adapt is heightened through necessity. This shows us that human ability to innovate is not the bottleneck – it is elsewhere in the system.

Maybe Moore’s law is much more an observation than a law. It just represents our current civilisation’s rate of adopting change during steady state (i.e. excluding total world war and imminent environmental doom) and not our ability to innovate. I certainly see this all the time within our consultancy. Our ability to technically innovate and deliver advanced solutions is never the bottleneck. It almost always sits further on in the value chain. The Theory of Constraints not only tells us to match throughput to demand, but also to address those constraints. This is why as a technology consultancy we also focus on Business Change Management and User Centred Design to directly address the constraint of adopting the change. This ensures that our clients and their consumers can successfully adopt innovation for their advantage and not settle for the plodding pace set by averages. If we didn’t we’d be delivering future shock! For example: delivering technology to bring social media experiences such as Facebook and Twitter within an enterprise is not hard for us, but there’d be no demand for this change if the business and the users couldn’t understand the vision and integrate it into their working practices.

We’re ready when you are…

In an ever more agile world with test driven development and emergent design, has cloud computing come along at a perfect time to solve some of the age old problems of performance and scalability management? Let’s first look at a couple of the classic problems:

At the start of the project, management are pulling together budgets as they build a business case. They need to know how much the physical infrastructure is going to cost? How many web front ends? How much storage? How much memory and CPU in the database server? How many environments and how similar do they need to be to the production environment? System Test, End to End Test, Integration Test, Operational Acceptance Test, User Acceptance Test, Pre-Production, Disaster Recovery, Development and Build etc. The thing is, you only have a high level architecture at this point and some high level requirements. Unless the solution is just like the solution you just built then how on earth can you accurately answer these questions? Perhaps you plan to use a new technology stack on a new range of infrastructure. Perhaps the expectation is that this is going to define the next generation of rich highly personalised web chocolaty goodness that has never been attempted before. You probably don’t have all the requirements yet and there is no point denying it. Agile tells us to expect change – it is inevitable. But what if an apparently subtle change will devastate performance/scale? Perhaps you won’t be able to cache as much as you’d hoped and therefore you won’t be able to offload stress on the database. Perhaps concurrency will force everything to be “serialisable” when everyone expected it to be “read uncommitted”. Either way, in many scenarios there can be a very low level of certainty in defining the production infrastructure let alone all the supporting test environments that support it. Problem number 1: being forced to accurately specify all environments a long time in advance of developing the solution which will then be fixed for all time and most likely leads to big overestimation or under-estimation.

Those responsible for ensuring the performance and scalability of the solution insist on having a replica of the production environment (that you aren’t yet able to specify) so that they can reliably test and “accept” the product. Given that the team responsible for prematurely guesstimating the infrastructure are gigantically unconfident they compensate by scoping the infrastructure as huge as they can imagine they could ever need. And someone else then doubles it. The replica performance / scale environment then becomes a totally untenable proposition (unless it can be justified to double up as the Disaster Recovery environment) and is forced to become a scaled down version and probably virtualised. Clever people then try to create a model that interpolates the difference between the big real environment and the small virtualised environment and behind their backs they cross their fingers and touch wood. Problem number 2: to genuinely performance/scale/load/stress test the solution will require an environment that is an exact replica of production; the prohibitive cost of which leads to dissimilar environments and undesirable consequences.

Enter cloud computing. Sprint after sprint the solution takes shape as does the infrastructure required to support it. Because the solution is targeting the cloud it is simple to have an exact replica for both production and performance testing. The performance testing environment only exists, and therefore paid, for the time that tests are being conducted. As tests provide feedback and the solution changes shape in response, so does the required infrastructure in a perfect feedback loop. The provisioning and payment of infrastructure is perfectly matched to the demand for it. When the solution goes live it is accurately known that the amount of infrastructure in use is exactly what is needed (as proven) with no risk to capital expenditure – and that is if we ignore the big cloud benefit of elasticity that continually matches production resources with actual demand as opposed to a best effort business model (yet another estimate). Also, if the current solution demands 50 web front ends it is easy to balance this expense with the labour costs that may be incurred to improve efficiency and reduce infrastructure to 20 WFEs. As efficiency is improved, infrastructure and operational expense are released. Again a direct correlation between effort to optimise and the affect on the bottom line is clear.

Ultimately the cloud can provide a clear cost model that solution architects can optimise towards. Continuously evolving solutions can have continuously evolving infrastructure. This can be fed back into the sprint planning and prioritised (i.e. “it is clear the solution is going to require considerable resources to meet demand and this needs to be addressed now as a matter of priority”). It also means that you can easily build test environments to support performance and scalability (including “how would the solution handle 10x the demand”) and ultimately know that you only use what you need when you have met your performance and scale goals.

Early indications of Microsoft’s Azure pricing are out. As Simon Munro pointed out they seem an awful lot like Amazon’s AWS pricing and to some degree Google’s pricing. Perhaps people are already building Excel spreadsheets and AJAX apps to help model your solution like on Amazon’s Simple Monthly Calculator!

Microsoft have to move into this space to defend many of their franchises that would otherwise be eroded or sunk by the likes of Google and Amazon AWS. With MSFT on the scene running the most advanced⁽¹⁾ mega data centres, owning the entire technology stack as well as having experience of operating several of the biggest web properties on the planet⁽²⁾, will Amazon AWS be able to compete with that as well as Google? They will also have to compete with the likes of GoGrid and any other Managed Service company that wish to restructure around a Cloud business model. After all, their core business is the world famous ecommerce site and not Cloud Computing. What started as selling off excess capacity has evolved a great deal, matured and established a growing ecosystem with many big names putting their business on it. But is that enough to compete with pure technology giants who have set out to build this from the ground up to specifically address this need? Is there enough desire and a good enough business case within Amazon to do this? Will AWS need to become a fully separate business with Amazon as one of its biggest clients? However they justify it I hope they do because we need the choice and innovation they all offer.

(1) I expect – I don’t know.
(2) 2B Bings(!!) per month, 10B MSN page views per month, 30B Live ID authentications per month, 240B Messenger messages per month. These numbers have probably grown somewhat given Bing is boasting a 8% boost last month.

Shrink yourself to the size of an ant and think of a classic server as a mini data centre. Banks upon banks of memory, multiple disks, many CPU cores and even network interfaces. The OS manages all of the many processes and threads, sharing the resources amongst them and when they demand more resource it makes it available to them – on demand and apparently instantly. Importantly all the resources in my mini data centre genuinely appear as one enormous pool available to all my applications at once. My application runs in a process and doesn’t care whether it consumes one core or 8. So what is a cloud OS?

Cloud OSes do appear to be quite different because somehow I need to play a part traditionally played by the OS. When my application demands more resources I get involved. Typically I need to allocate more virtual machines or storage and quite likely I need to tell my application to extend itself into this new resource (i.e. install my application components and bring it into the pool/farm/cluster/you-name-it). Then when that demand passes I need to consider releasing those resources.

Consider for a moment too the utility model, or to use a common analogy: “cloud computing is like the power grid – on demand, just in time, scale up and down”. When I need to consume more power, the statistically multiplexed grid allows me to draw as much as I want. There are even special burst providers like pumped storage schemes to handle the gigantic surge of kettles going on after a football game. If I need a milliamp for a second or 10 amps for an hour it makes no odds to me(!).

Again the cloud seems to break this analogy. My application will consume up to the limit of pre-allocated resources and stop. The data centre might have a potential for a billion petaflops but the cloud OS can’t just grant them to my app. It is a bit like knowing that I am about to plug in my electric car and ringing the grid to provision me more current.

Once upon a time the server OS used to be quite crude but they evolved and we take their sophistication for granted now. Most people struggle to even describe what an OS does. At the moment I need to resort to special software like Appistry to help me virtualise my application so that when I auto scale up/down my Amazon machine images the application can do so too. How long before these capabilities are part of the cloud OS allowing applications to genuinely scale up and down very discretely? Will all the CPUs in the cloud data centre eventually present themselves as a single virtual resource with the cloud OS presenting abstractions, much like a regular process, as execution containers or in the cloud OS will the virtual machine always get in the way?

It feels as if the virtual machine is a convenient way of *currently* spanning large amounts of hardware, but is it a stop gap? Will a genuine cloud OS come along that behaves like a regular server OS atop a giant data centre? It will be happy to present 24TB of apparently contiguous RAM and 1000 cores to a single “process”. It will be like NUMA and move application data across the server farm to the CPU that has been scheduled to work on it. Do we need it? Are we content for the next few decades to worry about how to span virtual machines and make them collaborate and auto-scale? Is the VM the new Process?

P.S. I appreciate that in many clouds, storage capacity seems to have been the first to become a truly virtualised and discrete resource, although storage performance hasn’t (i.e. most solutions are too blunt to configure for different performance characteristics etc.)

When you have a web farm, monitoring can become a little bit more tricky. When you deploy a monitor  it is only monitoring the behaviour of the server chosen by the load balancer to respond. You need to create a monitor that is engaging each server directly, bypassing the load-balancer. Obvious, yes. Hard, no.

When you have a web farm to serve your web site you want to make sure that they are all serving the same content. Maybe post-deployment, or just on a regular interval you want to check that one server hasn’t failed to pick up a configuration change or been accidentally interfered with. Or perhaps, when another monitor detects a failure you want to see if the wrong content is displaying across all web servers or just one.

One thing you can do is to create a web page that includes an iFrame of each web server (bypassing the load balancer obviously). This is great for a quick eye-balling and may give the answer you need straight away. Sometimes the difference in a web page is not visual, or too subtle to spot in a hurry…

I feed my habit by coding on the train home and I knocked this PowerShell script together to line by line compare the output of a page across a farm. It is a start and has proved handy a couple of times to spot a problem. It can be vastly improved of course. For example: by repeating the script and passing different HTTP headers (emulating a different browser type) it can see whether Web Server A is serving different content to IE8 to Web Server D. That’d be hard to spot. Anyway, here it is:

$inputDocuments ="http://192.168.0.10/default.aspx","http://192.168.0.11/default.aspx","http://192.168.0.12/default.aspx","http://192.168.0.20/default.aspx","http://192.168.0.21/default.aspx"             

# Use this line if you would prefer the script actually ignored known patterns that different in each file
$patternList ="^\s*<input\s+type=""hiden""\s+value=""[^""]+""\s+/>","^[\s]*$"

functionMatchesKnownPattern ($inputLine)
{
   ForEach ($patternin$patternList)
    {
       $pat=[regex]$pattern
       if($pat.match($inputLine).Success)
        {
           return $TRUE
       }
    }
   return $FALSE
}

# For each document create a hash table of line number and the actual line. Exclude all lines that match a pattern known to be different in each document.
$parsedDocuments =@()
ForEach ($inputDocument in$inputDocuments)
{
    [regex]::Split((New-Objectnet.webclient).DownloadString($inputDocument),"\r\n") |ForEach-Object -begin{$count=0;$webLines=@{}}-Process{if(MatchesKnownPattern($_)){$webLines[$count]="KNOWN DIFFERENCE";$count++;}else{$webLines[$count]=$_;$count++;}}-End{$parsedDocuments +=$webLines}
}

# Look at each document and see if after all known different patterns are excluded if they have the same line count:
$lineCounts =@{}
ForEach ($parsedDocument in$parsedDocuments)
{
   $lineCounts[$parsedDocument.psbase.keys.count]++
}

if($lineCounts.Count-gt 1)
{
   Write-Host"Line counts are different"
   $documentCount =0;
   ForEach ($inputDocument in$inputDocuments)
    {
       Write-Host"Document ["$inputDocument "] has a count of: "$parsedDocuments[$documentCount].Count
       $documentCount++
   }
}

# Having agreed that they all have the same line count then go through each line
# and then compare:
# file 1 line 1 with file 2 line 1
# file 2 line 1 with file 3 line 1
# file 3 line 1 with file 4 line 1
# file 4 line 1 with file 5 line 1
# file 1 line 2 with file 2 line 2
# ....
$agreedLineCount =$parsedDocuments[0].Count
for($index=0;$index-lt $agreedLineCount;$index++)
{
   for($innerIndex =0;$innerIndex -lt $parsedDocuments.Length-1;$innerIndex++)
    {
       if($parsedDocuments[$innerIndex][$index]-cne $parsedDocuments[$innerIndex +1][$index])
        {
           Write-Host"Compared Document ["$inputDocuments[$innerIndex]"] Line Number ["($index+1)"]:"$parsedDocuments[$innerIndex][$index]
           Write-Host"To Document          ["$inputDocuments[$innerIndex +1]"] Line Number ["($index+1)"]:"$parsedDocuments[$innerIndex +1][$index]
           # If the files are horrendously different (i.e. out of step by one line) then you may just want the script exit on first error
            #Exit
       }
    }
}
Write-Host Done

I had a wry smile on the way to work the other day. I was passing a crossing, the lights were changing to green. A motorbike that had been crawling past the cars had reached the front, but people were straggling across. He blipped the throttle a few times. Nice aftermarket pipes – lovely tone, but probably not road legal. Anyway, the stragglers looked across with annoyed expressions and one of them blurted a few rude words. I carried on past the junction smiling.

It wasn’t that I was enjoying those melodic pipes. Nor was it that I missed my biking days. I had insight into the situation that others may not have. I knew that the biker had slowly controlled the bike past the line of traffic. So that he could make a nice smooth move away from the lights when they turned, he had timed it nicely to be still slowly moving as he reached the front and the lights changed. Trailing the rear brake to keep it nicely controlled so the forks wouldn’t dive from using the front brakes and make him wobble. Proper forward planning and control. What he hadn’t planned for were people crossing as the lights were going green for the traffic. As a result he was nearly at a standstill when he should have been moving away. His throttle blipping was to keep the bike balanced for a little longer so he did not have to put his feet down and for it to end up being a clumsy affair. Blipping the throttle gives the motorbike a little bit of angular momentum which stops the bike toppling over so readily – just enough to let the stragglers to clear the way. You know the trick with the big spinning disk in a briefcase that people struggle to turn? Or the plain spinning top that only falls over when it loses enough of its angular momentum.

So where the guilty stragglers saw an act of aggression I saw an act of very fine low speed handling and forward planning – hence my smile.

I saw parallels with consulting and presales. The frustrating scenarios where your skill and experience are misunderstood by clients who sometimes see time, budget and resources as excessive. Sometimes all you need is a bit of time to explain the low speed riding, forward planning and angular momentum so they can see it for the very desirable excellence and not an act of aggression, or in consulting terms: greedy profiteering. Sometimes you don’t get that chance and the client selects a supplier who doesn’t either and you just have to keep walking. Sometimes that is my job – to help my future clients really understand how we couple innovation with the rhythm and quality of manufacturing whilst building their awareness of the risks and complexity where it exists. All of this we do so that we can engage with a complete understanding – eyes wide open. Not just a proposal to win the work. A proposal to work together with understanding and transparency and empowered to succeed.

I should have mentioned it earlier, but the Microsoft offerings are based around Windows Server 2003 R2 and SQL Server 2005. I wonder why this is not Windows Server 2008 and SQL Server 2008? Were Microsoft unwilling? Did it take a long time to certify the Amazon Machine Images and the 2008 editions are on the way?

James Simmonds mentioned to me earlier that he is using GoGrid. Looks like a good offering for Windows Server 2008 and SQL Server 2005. They are still in Beta but they are promising 100% uptime and 10,000% Guaranteed SLA... They are also showing some GoGrid vs Amazon EC2 comparisons.

I'd like to correct myself too: Amazon's cloud offering is not just on-premise software. Clearly Simple Storage Service, Simple Queue Services and SimpleDB are offering more of a pure cloud platform approach as opposed to the hosted image approach. Both will have a role to play.

The rate at which .Net advances and improves is fantastic. Everything from LINQ to ASP.Net MVC via F# and parallel programming advances. The roadmap for on-premise and cloud computing shows that the future will be far from dull. Alongside the platform, the core Microsoft server products, such as Windows and SQL Server, are also advancing very favourably. Beyond Microsoft we are also getting ever more fabulous tools and frameworks ranging from ReSharper, Inversion of Control and Aspect Oriented Frameworks to Mocking and Test Frameworks. Around all of this, maturity is growing in Test Driven Development, Continuous Integration, Domain Driven Design and a whole range of Agile delivery methods. There is a growing reality of custom building solutions with repeatable success, considerable flexibility and quality that exactly meet the needs. Good times indeed.

What's my point? With such continuous and powerful advances being made I think it is time to re-evaluate many of the Build vs Buy decisions made a few years ago and maybe reset some of our defaults. Now I am not saying that something so radical has happened that the pendulum has decisively swung towards Build. However, I do think it has moved in this direction and I am expecting this to bring pressure on Product vendors to respond. That goes for Microsoft too who are ironically instrumental in handing the builders this power. You may argue that this power is also available to the product vendors, but too often I don't see that materialising. Maybe this is just the lag between harnessing the power and shipping products. That's me being generous.

No longer can products rely on (or even hide behind) arguments such as vendor support, upgrades and a resource pool of trained professionals. More than ever the products are lagging much further behind the technology curve, don't deliver exactly what you want and often represent significant compromise. Where the product contributes to the market differentiation of your business proposition it has to be super compelling and leading edge.

For example, some of these may resonate:

• How long do you have to wait for a product to take advantage of SQL Server 2008  - not just "supported on" or "compatible with", but actually exploit it?
• How long before the product adopts an architecture that better reflects how you want to deploy it in a Virtualised data centre - and changes its licensing to make sense?
• How long before it recognises that at scale might mean a large cluster of commodity hardware - e.g. a web farm of 50 pizza boxes each with their own local web and instance of a federated and co-operating database?
• How long before the product caters for and exploits the emerging and different Cloud computing platforms?
• How long before it truly exploits multi-core developments and has parallelism built right into its fabric as opposed to just using ubiquitous multiple cores as a means to extract more licensing?
• How long before the product is architected and licensed to match your increasing need to deploy for high levels of availability and business continuity? This is a regular expectation not an extraordinary demand.

In summary: Development platforms, methodologies and tooling is aggressively getting better and looks set to continue. This is certainly shifting the balance of power in the Build vs Buy argument towards Build. This should put pressure on product vendors to demonstrate clear advantages ("value-add") and more aggressive roadmaps; they have no excuse not to because they should be harnessing the same power! The Build vs Buy balance of power should remain mostly in equilibrium except during periods of arbitrage during the time lag when advances in the core technologies are harnessed by vendors. The ideal is to spot arbitrage and exploit it, or mitigate it. Right now, builders have the opportunity to make market share whilst vendors have the opportunity to leap frog competitors whilst ensuring they invest so as not to lose market share to builders!

N.B. Taking true advantage of these advances puts more emphasis on smart, skilled and experienced people.

I know. I ramble.