James Saull's Blog

Cloud DevOps where is your voice?

2010-11-29T11:59:00Z

To be honest I am completely overwhelmed by the onslaught of technologies emerging as serious contenders for building the cloud generation of architectures and applications. On the one hand I am truly grateful that there are super-smart engineers building a new generation of technologies but I am suffering Future Shock!

It appears to me that most conversations are led by the application development community. I am referring to conversations about applications on Google’s AppEngine, Microsoft’s Azure, Amazon’s Web Services, Joyent, GoGrid, Heroku, EngineYard and so on.

Naturally the next generation of applications being built are probably formed first in the minds of developers and architects. My bias probably means I gravitate towards those conversations too.

What seems to be missing is the voice of Ops. Sure - there are people talking about Puppet, Chef, Capistrano, Mcollective and the like. But I mean the voice of the people talking about running a 24/7 Network Operations Centre and an ITIL service around apps deployed on App Engine (for example). It won’t be long before all the Python, MongoDB, Hadoop, GigaSpaces^*1 systems are being demanded by developers to deploy on Eucalyptus/AppDrop^*1 in regular on-premise data centres when they can’t do it in the public cloud. This won’t be an issue if the Ops knows how to deploy, patch, monitor, fix, recover, capacity plan and generally manage these sorts of topologies for business critical systems. I’d take a wild guess at this not being the case… Never met a MongoDBA?

I think what I am saying is: if the forward planning Ops folks can wade in vocally and share their view of emerging technologies, then they can help influence the process of technology natural selection. They can help identify those that are fit for operations; and in turn not be asked to support the unsupportable! Wade in or forever hold you peace!

Change is happening at an accelerated pace and compelling economics will cast aside all who stand in its way. Be part of the change.

^*1 Just a smattering of unrelated technologies that probably aren’t as well known in the on-premise application development

Managing future shock to find commercial advantage

2010-04-13T15:35:15Z

If I interpret the Wikipedia entry correctly, Moore’s law started as an observation that perhaps transitioned into a self fulfilling prophecy: “…[this law] is now used in the semiconductor industry to guide long-term planning and to set targets for research and development.”. I wonder if there isn’t a more human explanation and lucky dose of coincidence for the longevity of this law – i.e. it was initially true for one reason but soon gave way to another that happened to be similar.

If our civilisation, and all its economics, generates the demand for improving products and services then is it also naturally limited in its ability to successfully cope with a certain rate of change? And, therefore, will the demand for change remain constant? If throughput in innovation is properly matched to this demand then it would be no surprise that we don’t deviate much from Moore’s law (or variants of it). I suppose, in terms of Theory of Constraints, I am describing our current state of civilisation as the constraint to adopting innovation rather than a constraint of our ability to innovate. After all, why invest and innovate faster than you can ship it?

If, tomorrow, Intel brought to market a chip with 1000 cores what would we do with it? We couldn’t really exploit it. Throughput would have massively outmatched demand and Intel would have missed the opportunity to sell 16, 32, 64, 128 core systems. What if Canon had released a 28MP DSLR as one of the early generation digital cameras? Not only would USB 1.0, hard drives and narrow band internet connections have struggled but the mass market computers couldn’t cope with that resolution. Again innovation throughput would have exceeded demand.

Another scenario of innovation throughput exceeding demand would be Canon releasing a new camera every month doubling resolution. Logistics and marketing are obvious impediments but also consumers couldn’t afford that rate of change. It’d be a Greek geek tragedy and a form of “future shock”. Medical innovation also often has to throttle back whilst cultural/religious/legal alignment takes place.

War is often cited as a time that demand for innovation rises, and we see innovation throughput rise too. There is usually less impedance as activities such as marketing get dropped and people’s willingness to adapt is heightened through necessity. This shows us that human ability to innovate is not the bottleneck – it is elsewhere in the system.

Maybe Moore’s law is much more an observation than a law. It just represents our current civilisation’s rate of adopting change during steady state (i.e. excluding total world war and imminent environmental doom) and not our ability to innovate. I certainly see this all the time within our consultancy. Our ability to technically innovate and deliver advanced solutions is never the bottleneck. It almost always sits further on in the value chain. The Theory of Constraints not only tells us to match throughput to demand, but also to address those constraints. This is why as a technology consultancy we also focus on Business Change Management and User Centred Design to directly address the constraint of adopting the change. This ensures that our clients and their consumers can successfully adopt innovation for their advantage and not settle for the plodding pace set by averages. If we didn’t we’d be delivering future shock! For example: delivering technology to bring social media experiences such as Facebook and Twitter within an enterprise is not hard for us, but there’d be no demand for this change if the business and the users couldn’t understand the vision and integrate it into their working practices.

We’re ready when you are…

Trendy brands still make mistakes. How many will be tolerated?

2009-11-09T20:54:23Z

Quite a day: Google Chrome/Gears bug could have allowed hackers to execute malicious code on the user’s system and the iPhone gets worms down under. Trendy brands can't escape the reality that building robust/secure software is not trivial. Will they be smart enough to avoid the pit Microsoft fell into (and climbed out of) or will they go through the same cycle?

Have these firms managed to combine innovation with robust security engineering and will they support the software they release today for the next 5+ years and keep their customers safe? Let’s hope so because it would be a shame not to learn from the mistakes of others.

Agile infrastructure and how Cloud Computing can help Performance and Scalability

2009-08-17T10:49:00Z

In an ever more agile world with test driven development and emergent design, has cloud computing come along at a perfect time to solve some of the age old problems of performance and scalability management? Let’s first look at a couple of the classic problems:

At the start of the project, management are pulling together budgets as they build a business case. They need to know how much the physical infrastructure is going to cost? How many web front ends? How much storage? How much memory and CPU in the database server? How many environments and how similar do they need to be to the production environment? System Test, End to End Test, Integration Test, Operational Acceptance Test, User Acceptance Test, Pre-Production, Disaster Recovery, Development and Build etc. The thing is, you only have a high level architecture at this point and some high level requirements. Unless the solution is just like the solution you just built then how on earth can you accurately answer these questions? Perhaps you plan to use a new technology stack on a new range of infrastructure. Perhaps the expectation is that this is going to define the next generation of rich highly personalised web chocolaty goodness that has never been attempted before. You probably don’t have all the requirements yet and there is no point denying it. Agile tells us to expect change – it is inevitable. But what if an apparently subtle change will devastate performance/scale? Perhaps you won’t be able to cache as much as you’d hoped and therefore you won’t be able to offload stress on the database. Perhaps concurrency will force everything to be “serialisable” when everyone expected it to be “read uncommitted”. Either way, in many scenarios there can be a very low level of certainty in defining the production infrastructure let alone all the supporting test environments that support it. Problem number 1: being forced to accurately specify all environments a long time in advance of developing the solution which will then be fixed for all time and most likely leads to big overestimation or under-estimation.

Those responsible for ensuring the performance and scalability of the solution insist on having a replica of the production environment (that you aren’t yet able to specify) so that they can reliably test and “accept” the product. Given that the team responsible for prematurely guesstimating the infrastructure are gigantically unconfident they compensate by scoping the infrastructure as huge as they can imagine they could ever need. And someone else then doubles it. The replica performance / scale environment then becomes a totally untenable proposition (unless it can be justified to double up as the Disaster Recovery environment) and is forced to become a scaled down version and probably virtualised. Clever people then try to create a model that interpolates the difference between the big real environment and the small virtualised environment and behind their backs they cross their fingers and touch wood. Problem number 2: to genuinely performance/scale/load/stress test the solution will require an environment that is an exact replica of production; the prohibitive cost of which leads to dissimilar environments and undesirable consequences.

Enter cloud computing. Sprint after sprint the solution takes shape as does the infrastructure required to support it. Because the solution is targeting the cloud it is simple to have an exact replica for both production and performance testing. The performance testing environment only exists, and therefore paid, for the time that tests are being conducted. As tests provide feedback and the solution changes shape in response, so does the required infrastructure in a perfect feedback loop. The provisioning and payment of infrastructure is perfectly matched to the demand for it. When the solution goes live it is accurately known that the amount of infrastructure in use is exactly what is needed (as proven) with no risk to capital expenditure – and that is if we ignore the big cloud benefit of elasticity that continually matches production resources with actual demand as opposed to a best effort business model (yet another estimate). Also, if the current solution demands 50 web front ends it is easy to balance this expense with the labour costs that may be incurred to improve efficiency and reduce infrastructure to 20 WFEs. As efficiency is improved, infrastructure and operational expense are released. Again a direct correlation between effort to optimise and the affect on the bottom line is clear.

Ultimately the cloud can provide a clear cost model that solution architects can optimise towards. Continuously evolving solutions can have continuously evolving infrastructure. This can be fed back into the sprint planning and prioritised (i.e. “it is clear the solution is going to require considerable resources to meet demand and this needs to be addressed now as a matter of priority”). It also means that you can easily build test environments to support performance and scalability (including “how would the solution handle 10x the demand”) and ultimately know that you only use what you need when you have met your performance and scale goals.

Peculiar behaviour in PowerShell with dot sourcing and $LASTEXITCODE

2009-08-07T13:29:35Z

Here is a script called SimpleExit.ps1

Exit 99

And here it is in use:

PS C:\temp> . .\SimpleExit.ps1 (Dot sourcing it so that variables in the script are made “global” )
PS C:\temp> $LASTEXITCODE
99 (You expected that – a variable set in the script was made “global” and therefore accessible by the calling shell/scope)

PS C:\temp> $LASTEXITCODE = 0 (Let’s reset it now)
PS C:\temp> $LASTEXITCODE
0

PS C:\temp> .\SimpleExit.ps1 (If we don’t dot source it, then $LASTEXITCODE in the script is NOT global and therefore not accessible by the calling shell/scope)
PS C:\temp> $LASTEXITCODE (so let’s see what the value of $LASTEXITCODE is now…)
99 (Yikes! So even though $LASTEXITCODE was set to 99 in a child scope it is visible in the parent – is this the magic of the Exit Keyword?)

Here is a script called MainTest.ps1

Write-Host "Started MainTest and here is the call stack:"
Get-PSCallStack
$LASTEXITCODE = 100
Write-Host "LASTEXITCODE just set to 100"
Write-Host "Now going to execute InnerTest by dot sourcing it..."
. .\InnerTest.ps1
Write-Host "Finished executing InnerTest and the LASTEXITCODE is: $LASTEXITCODE and it should be 10"
Exit 1

and a script called InnerTest.ps1:

Write-Host "Started InnerTest and here is the call stack:"
Get-PSCallStack
Write-Host "LASTEXITCODE is $LASTEXITCODE and it should be 100"
Write-Host "Setting LASTEXITCODE to 10"
Exit 10

Now observe how dot sourcing affects the behaviour of calling these scripts:

PS C:\temp> C:\temp\MainTest.ps1
Started MainTest and here is the call stack:

MainTest.ps1
prompt
LASTEXITCODE just set to 100
Now going to execute InnerTest by dot sourcing it...
Started InnerTest and here is the call stack:
.
MainTest.ps1
prompt
LASTEXITCODE is 100 and it should be 100
Setting LASTEXITCODE to 10
Finished executing InnerTest and the LASTEXITCODE is: 100 and it should be 10

PS C:\temp> . C:\temp\MainTest.ps1
Started MainTest and here is the call stack:

.
prompt
LASTEXITCODE just set to 100
Now going to execute InnerTest by dot sourcing it...
Started InnerTest and here is the call stack:
.
.
prompt
LASTEXITCODE is 100 and it should be 100
Setting LASTEXITCODE to 10
Finished executing InnerTest and the LASTEXITCODE is: 10 and it should be 10

The crux here is that when you DON’T dot source MainTest.ps1 the variable $LASTEXITCODE flows down into the child script (InnerTest.ps1) which can see the value is 100, but when it calls “Exit 10” that does not modify the value of $LASTEXITCODE in the parent script MainTest.ps1. However, when we call MainTest.ps1 by dot sourcing it then the value is modified by the child script because it is global. But if you go back to the simple example at the top you will notice that it doesn’t matter whether I dot source or not, Exit 99 modified the value $LASTEXITCODE of the root shell!

Dot sourcing is like saying “make everything in the executing script a global variable” and this would explain why if I don’t dot source MainTest.ps1 then it has its own locally scoped version of $LASTEXITCODE and why the value stays at 100. This makes it super annoying too! If I use “Exit 10” in a child script then it won’t be “bubbled up” properly because the calling script itself was not dot sourced! And it is apparently inconsistent with my first example which indicates that if the regular PowerShell shell calls a script that uses “Exit 10” it will bubble up! AGH!

This is made all the more strange by the fact that when you run the PowerShell ISE it is as if the ISE was itself dot sourced. By that I mean when you step through a script in ISE you WILL get different results to just running the script WITHOUT dot sourcing it on the command line.

Will Amazon AWS be able to continue to compete?

2009-07-15T10:02:01Z

Early indications of Microsoft’s Azure pricing are out. As Simon Munro pointed out they seem an awful lot like Amazon’s AWS pricing and to some degree Google’s pricing. Perhaps people are already building Excel spreadsheets and AJAX apps to help model your solution like on Amazon’s Simple Monthly Calculator!

Microsoft have to move into this space to defend many of their franchises that would otherwise be eroded or sunk by the likes of Google and Amazon AWS. With MSFT on the scene running the most advanced⁽¹⁾ mega data centres, owning the entire technology stack as well as having experience of operating several of the biggest web properties on the planet⁽²⁾, will Amazon AWS be able to compete with that as well as Google? They will also have to compete with the likes of GoGrid and any other Managed Service company that wish to restructure around a Cloud business model. After all, their core business is the world famous ecommerce site and not Cloud Computing. What started as selling off excess capacity has evolved a great deal, matured and established a growing ecosystem with many big names putting their business on it. But is that enough to compete with pure technology giants who have set out to build this from the ground up to specifically address this need? Is there enough desire and a good enough business case within Amazon to do this? Will AWS need to become a fully separate business with Amazon as one of its biggest clients? However they justify it I hope they do because we need the choice and innovation they all offer.

(1) I expect – I don’t know.
(2) 2B Bings(!!) per month, 10B MSN page views per month, 30B Live ID authentications per month, 240B Messenger messages per month. These numbers have probably grown somewhat given Bing is boasting a 8% boost last month.

Megastructure Datacentres

2009-07-01T20:31:10Z

I suggested last year that National Geographic would probably not include Cloud Datacentres in their megastructures series. I am in Dublin just as MSFT launch their “mega datacentre”. Perhaps they should!

How cloud operating systems aren’t quite there yet

2009-07-01T19:25:00Z

Shrink yourself to the size of an ant and think of a classic server as a mini data centre. Banks upon banks of memory, multiple disks, many CPU cores and even network interfaces. The OS manages all of the many processes and threads, sharing the resources amongst them and when they demand more resource it makes it available to them – on demand and apparently instantly. Importantly all the resources in my mini data centre genuinely appear as one enormous pool available to all my applications at once. My application runs in a process and doesn’t care whether it consumes one core or 8. So what is a cloud OS?

Cloud OSes do appear to be quite different because somehow I need to play a part traditionally played by the OS. When my application demands more resources I get involved. Typically I need to allocate more virtual machines or storage and quite likely I need to tell my application to extend itself into this new resource (i.e. install my application components and bring it into the pool/farm/cluster/you-name-it). Then when that demand passes I need to consider releasing those resources.

Consider for a moment too the utility model, or to use a common analogy: “cloud computing is like the power grid – on demand, just in time, scale up and down”. When I need to consume more power, the statistically multiplexed grid allows me to draw as much as I want. There are even special burst providers like pumped storage schemes to handle the gigantic surge of kettles going on after a football game. If I need a milliamp for a second or 10 amps for an hour it makes no odds to me(!).

Again the cloud seems to break this analogy. My application will consume up to the limit of pre-allocated resources and stop. The data centre might have a potential for a billion petaflops but the cloud OS can’t just grant them to my app. It is a bit like knowing that I am about to plug in my electric car and ringing the grid to provision me more current.

Once upon a time the server OS used to be quite crude but they evolved and we take their sophistication for granted now. Most people struggle to even describe what an OS does. At the moment I need to resort to special software like Appistry to help me virtualise my application so that when I auto scale up/down my Amazon machine images the application can do so too. How long before these capabilities are part of the cloud OS allowing applications to genuinely scale up and down very discretely? Will all the CPUs in the cloud data centre eventually present themselves as a single virtual resource with the cloud OS presenting abstractions, much like a regular process, as execution containers or in the cloud OS will the virtual machine always get in the way?

It feels as if the virtual machine is a convenient way of *currently* spanning large amounts of hardware, but is it a stop gap? Will a genuine cloud OS come along that behaves like a regular server OS atop a giant data centre? It will be happy to present 24TB of apparently contiguous RAM and 1000 cores to a single “process”. It will be like NUMA and move application data across the server farm to the CPU that has been scheduled to work on it. Do we need it? Are we content for the next few decades to worry about how to span virtual machines and make them collaborate and auto-scale? Is the VM the new Process?

P.S. I appreciate that in many clouds, storage capacity seems to have been the first to become a truly virtualised and discrete resource, although storage performance hasn’t (i.e. most solutions are too blunt to configure for different performance characteristics etc.)

Dilbert’s Scott Adams is on the trail of Device 0

2009-05-19T18:30:04Z

I read this post on Scott Adams’ blog today: http://dilbert.com/blog/entry/your_phone/ It echoed my sentiments about the evolution of the mobile phone that I posted last year here: http://blogs.conchango.com/jamessaull/archive/2008/09/22/windows-mobile-and-cloud-computing-will-it-erode-laptop-and-books.aspx

Perhaps, with his considerable readership, he’ll provoke Steve Jobs into action. However, Microsoft have a much broader technology portfolio to actually materialise this. Like Veruca Salt I want this right now!

Making sure each web server in the farm is serving the same content.

2008-12-15T20:37:40Z

When you have a web farm, monitoring can become a little bit more tricky. When you deploy a monitor it is only monitoring the behaviour of the server chosen by the load balancer to respond. You need to create a monitor that is engaging each server directly, bypassing the load-balancer. Obvious, yes. Hard, no.

When you have a web farm to serve your web site you want to make sure that they are all serving the same content. Maybe post-deployment, or just on a regular interval you want to check that one server hasn’t failed to pick up a configuration change or been accidentally interfered with. Or perhaps, when another monitor detects a failure you want to see if the wrong content is displaying across all web servers or just one.

One thing you can do is to create a web page that includes an iFrame of each web server (bypassing the load balancer obviously). This is great for a quick eye-balling and may give the answer you need straight away. Sometimes the difference in a web page is not visual, or too subtle to spot in a hurry…

I feed my habit by coding on the train home and I knocked this PowerShell script together to line by line compare the output of a page across a farm. It is a start and has proved handy a couple of times to spot a problem. It can be vastly improved of course. For example: by repeating the script and passing different HTTP headers (emulating a different browser type) it can see whether Web Server A is serving different content to IE8 to Web Server D. That’d be hard to spot. Anyway, here it is:

$inputDocuments ="http://192.168.0.10/default.aspx","http://192.168.0.11/default.aspx","http://192.168.0.12/default.aspx","http://192.168.0.20/default.aspx","http://192.168.0.21/default.aspx"

# Use this line if you would prefer the script actually ignored known patterns that different in each file
$patternList ="^\s*<input\s+type=""hiden""\s+value=""[^""]+""\s+/>","^[\s]*$"

functionMatchesKnownPattern ($inputLine)
{
   ForEach ($patternin$patternList)
    {
       $pat=[regex]$pattern
       if($pat.match($inputLine).Success)
        {
           return $TRUE
       }
    }
   return $FALSE
}

# For each document create a hash table of line number and the actual line. Exclude all lines that match a pattern known to be different in each document.
$parsedDocuments =@()
ForEach ($inputDocument in$inputDocuments)
{
    [regex]::Split((New-Objectnet.webclient).DownloadString($inputDocument),"\r\n") |ForEach-Object -begin{$count=0;$webLines=@{}}-Process{if(MatchesKnownPattern($_)){$webLines[$count]="KNOWN DIFFERENCE";$count++;}else{$webLines[$count]=$_;$count++;}}-End{$parsedDocuments +=$webLines}
}

# Look at each document and see if after all known different patterns are excluded if they have the same line count:
$lineCounts =@{}
ForEach ($parsedDocument in$parsedDocuments)
{
   $lineCounts[$parsedDocument.psbase.keys.count]++
}

if($lineCounts.Count-gt 1)
{
   Write-Host"Line counts are different"
   $documentCount =0;
   ForEach ($inputDocument in$inputDocuments)
    {
       Write-Host"Document ["$inputDocument "] has a count of: "$parsedDocuments[$documentCount].Count
       $documentCount++
   }
}

# Having agreed that they all have the same line count then go through each line
# and then compare:
# file 1 line 1 with file 2 line 1
# file 2 line 1 with file 3 line 1
# file 3 line 1 with file 4 line 1
# file 4 line 1 with file 5 line 1
# file 1 line 2 with file 2 line 2
# ....
$agreedLineCount =$parsedDocuments[0].Count
for($index=0;$index-lt $agreedLineCount;$index++)
{
   for($innerIndex =0;$innerIndex -lt $parsedDocuments.Length-1;$innerIndex++)
    {
       if($parsedDocuments[$innerIndex][$index]-cne $parsedDocuments[$innerIndex +1][$index])
        {
           Write-Host"Compared Document ["$inputDocuments[$innerIndex]"] Line Number ["($index+1)"]:"$parsedDocuments[$innerIndex][$index]
           Write-Host"To Document          ["$inputDocuments[$innerIndex +1]"] Line Number ["($index+1)"]:"$parsedDocuments[$innerIndex +1][$index]
           # If the files are horrendously different (i.e. out of step by one line) then you may just want the script exit on first error
            #Exit
       }
    }
}
Write-Host Done