Welcome to EMC Consulting Blogs Sign in | Join | Help

Random Ramblings of a Platform Architect

  • I’ve moved

    I apologise for the silence  over the past few months but My posts have moved to Devops and the Public Cloud

  • How to set up a windows AMI to use with elastic bamboo

    Jira Studio uses elastic bamboo as its build controller which in turn uses elastic agents (AWS ec2 instances that are spun up by the build controller to run the build) rather than the standard remote agents. This means you will need to have an Amazon AWS account and will have to create a custom windows AMI to be used to create your elastic-agents. Unfortunately there are no available instructions on creating a custom elastic-agent for windows users so you have to use the Linux instructions as guide. This post outlines what I did to get this working for a recent project. Please note that this is unsupported by Atlassian but hopefully they’ll be supporting windows soon as from my experience it does all seem to work so far.

    Please note that this post assumes you have some basic knowledge of using AWS.


    Start a Windows 2008  instance from an  Amazzon base windows 2008 AMI in the us-east-1  region

    It needs to be in the US-East region as this is where Jira Studio expects to find the build agent.

    Using AMI



    Turn off windows firewall or ensure you have port 22626 open



    Ensure jre &/or  jdk installed and files from jdk that are needed are also in jre folder  if you decide to use the jre as your JAVA_HOME

    You will need tools.jar in your JAVA_HOME path so   you will have to install the jdk to get that

    Install whatever components you need to be able to undertake a build e.g visual studio, msbuild etc and other bits



    Use the link here as a  guide




    Set up the Amazon ec2 API tools as outlined in the Atlassian guide ( section 5.4)

    Specific guidance for windows  can be obtained from the AWS documentation

    To check the version of bamboo click on administration then expand under system \ system information 


    Scroll down to see the Bamboo version






    Download the bamboo-elastic-agent that matches the version of Bamboo that is being used in Jira Studio


    Make sure you click show all.



    Create folder c:\bamboo-elastic-agent



    Unzip the bamboo-elastic-agent-2.6.2.zip to folder created above

    To the folder : c:\bamboo-elastic-agent

    Add bin to path variable


    Download  the latest zip of ant and unzip into c:\ant\


    Set up environmental and path variables



    Download the latest zip of maven


    Unzip maven  into  c:\ apache-maven-2.0.11



    Set up environment variable and path as done for ant



    Summary of relevant Environment variables

    ANT_HOME  C:/ant

    EC2_CERT  c:\ec2-api-tools\YOUR-cert.pem

    EC2_HOME  c:\ec2-api-tools

    EC2_PRIVATE_KEY c:\ec2-api-tools\YOUR-pk.pem

    JAVA_HOME  C:\Program Files\Java\jre6

    MAVEN_HOME c:\apache-maven-2.0.11

    Path  variables

    .......  c:/ant\bin;C:\Program Files\Java\jdk1.6.0_23;c:\apache-maven-2.0.11\bin;C:\ec2-api-toolsbin;c:\bamboo-elastic-agent\bin

    Create batch file that  consists of two lines

    Line 1: the Java classpath ( This was obtained by using a simple Powershell script to scrape the lib folder under bamboo-elastic-agent  and setting the result as  a CLASSPATH  variable)

    Line2: This will run the actual elastic-agent

    Note there are over a  hundred jar files but to date Atlassian have been unable to let me know which ones are actually needed hence the snippet of the batch file I used  rather than the full list.



    SET CLASSPATH=acegi-security-1.0.4.jar;activation-1.1.1.jar;activeio-core-3.1.0.jar;activemq-core-5.2.0.jar;activemq-ra-5.2.0.........



    java -server -Xms32m -Xmx512m -XX:MaxPermSize=256m  -classpath %CLASSPATH% com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap 2>&1 > c:\bamboo-elastic-agent\bamboo-elastic-agent.log

    Test that everything is set up okay by running the batch file interactively. You should see output similar that  opposite. The key points are it trying to get userdata and attempting to create the tunnel. The agent needs to be started by the bamboo controller hence the errors.


    .........  Java.io.FileNotFoundException:
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown So
    at java.net.URL.openStream(Unknown Source)







    0 [main] INFO com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap  -

     Using tunnnelling. Registering 'httpt' and 'httpst' protocols.

    577 [com.sun.sungrid.service.tunnel.server.TunnelServer] INFO com.sun.sungrid.se

    rvice.tunnel.server.TunnelServer  - Waiting for tunnel connection.

    The agent needs to start automatically on starting the instance



    I looked at using the wrapper obtained via this url but that failed miserably


    I used the nssm  to wrap the batch file as a service


    Set it to start automatically


    Create the AMI



    Do the bamboo controller configuration & do some basic testing

    Register the AMI,

    Set up the capabilities , test that it will spin up the instance , test a basic build


































  • Configuration Management and the Cloud part III–Cooking with Chef

    Before I go on I apologise up front but I just can’t resist the cooking theme Embarrassed smile

    The purpose of this series of blog posts is to explore using Chef to configure instances in the cloud. My particular cloud for this just happens to be AWS.

    In this post I’ll be describing how to actually use Chef to spin up & configure an ec2 instance. I am making the assumption you are familiar with the contents of the previous two posts in this series. Even if you don’t have any experience with  AWS I hope it will still be easy enough to follow.

    In my last post I described the configuration of a windows workstation to prepare for this but I hit a problem which meant I have had to resort to using a Linux instance running on AWS. I have a bug logged with OpsCode. For those of you who are keen to make use of your windows machines as your user environment I will post an update when the bug is resolved.

    I am starting off with spinning up a Linux instance as it will be easier to grasp the concepts and what you need to do to use Chef effectively. My last post in this series will deal with using chef with a windows target.

    The first thing I needed to do was walk through what I wanted to achieve , in what order .

    I wanted to  spin up an ec2 instance and install tomcat so Chef will need to:

    ·         Fire up an ec2 instance

    ·         Bootstrap it so it can run a chef-client

    ·         Install java  if required ( Note as I am using an aws linux instance java is preinstalled)

    ·         Install tomcat

    Yes I know this can all be easily done using cloudinit and passing user data to your instance but I feel it’s pretty self evident how Chef or a similar tool would be a more appropriate enterprise solution with a large target ec2 estate with various roles .  The simplicity of the example is to illustrate the steps that need to be gone through.

    The really cool thing with Chef is that you can rerun cookbooks against a node and it will not do anything it has already done i.e it will not change the end result on the target node as defined by the recipes being run against it. So you will always get the same outcome no matter what state the node and actions will not be taken if already done (and conversely run if detected it has not been run).  When reading about Chef you will see this described as being idempotent (There I’ve saved you looking it up).

    At this stage I have a chef ready user environment, an OpsCode organisation set up and now I want to start by spinning up an ec2 instance.

    I will just be talking about the key chef configuration steps for the rest of this post but if you are following along then use the table at the end of this post that details all he steps I followed. I will not be going into any depth re the ec2 specifics as that would make this post far too long.

    The first thing I needed to do was create a cookbook for installing tomcat.  The Chef community (admittedly in the opensource world) though is full of off the shelf recipes that you can use to get started with.

    I did a quick search from the OpsCode cookbooks section of the web console to see if there were any tomcat cookbooks 


    I could have downloaded it from the console but I feel it was neater to use knife .


    I also need the cookbook java  and apt cookbooks .

    Before proceeding it would probably be a good idea to take time out and read the Opscode  Chef Recipe wiki   which has a nice clear explanation on cookbook name spaces. Also remind yourself of the components that make up a cookbook  it’s worth noting that recipes manage resources and those resources will be executed in the order they occur.

    As my target platform will be ‘amazon’ I decided to amend the default.rb to reflect that platform.  The Amazon AWS linux ami is based on centos so it was a minor change to add the platform.  I  did not need to do this modification as the centos bootstrap will work fine with no changes  but I felt it was a worthwhile exercise as it shows the sorts of steps you would need to go through typically.

    I then downloaded the Chef repository so that I could obtain a template bootstrap file. The purpose of the bootstrap file is to install chef on the target node and set the node up as a chef client

    At this stage I have a cookbook with a recipe ready to upload and run and I have a bootstrap file.  Nearly ready to try it all out but before I stitch it all together I need to introduce you to the concept of roles.


    A role is basically a user defined description of how a node should be configured and this description can then be applied to multiple nodes.  A role is normally defined from recipes defined in a cookbook. A role can have a number of cookbooks that need to be run against it.

     Typically a node will be carrying out a pre-determined role.

    I am describing in this post a tomcat server role which is an ec2 instance that has Tomcat installed. You can define numerous roles so later on in his series of posts I will be describing the creation of a windows web server role.




    I then uploaded my modified tomcat cookbook to my OpsCode chef instance and created a role that would basically run the tomcat recipe in this cookbook.


    I then ran the following command to instantiate my ec2 instance:

    knife ec2 server create "role[tomcat]"  --region eu-west-1  -Z eu-west-1a  -i  ami-75d4e101 -f t1.micro -G GraceHome -I ~/.ssh/GM-EC2.pem -S GM-EC2 --ssh-user ec2-user -d amazon-gems

    Note: see table at end for explanation of the ec2 specific flags

    On running the command a long series of actions occur which you can review via a log file



    When I saw  the following I then verified everything was  completed as expected:


    The new node appears in my node list:


    I can now consistently reproduce as many tomcat nodes as I want. What I’m not too sure of is how I can tie this in with AWS  auto scaling  but figure that’s something to look into later.

    The table below assumes that your user environment workstation is also set up to use the ec2 api tools

    Table 1: Configuration steps for starting an  AWS Amazon  Linux ec2 instance and installing tomcat


    Edit knife.rb  to add aws access key and secret access key

    # EC2:


    knife[:aws_access_key_id]     = "Your AWS Access Key"


    knife[:aws_secret_access_key] = "Your AWS Secret Access Key"

    Download tomcat  cookbook using knife

    knife cookbook site vendor tomcat  {-u username here }

     As I am reusing my windows keys I set up previously I need to add my username to all my knife commands,  but if you have set up your environment from scratch on your linux instance you should not need to do this

    Review cookbook recipe to understand what it’s doing &  what needs modiying

    Need to get  java  cookbook and also upload that as it is a dependency also need apt cookbook which is a dependency for the java cookbook


    Edit  the tomcat cookbook default.rb to include “amazon” as a valid platform

    Replace :





    Obtain a  copy of the   chef repository  so can get bootstrap template file

    [ec2-user@gmadmin chef-repo]$ cd ~

    [ec2-user@gmadmin ~]$ git clone https://github.com/opscode/chef.git


    Create  a bootstrap folder under the chef-repo folder

    [ec2-user@gmadmin chef-repo]$ mkdir ~/chef-repo/.chef/bootstrap



    Copy an appropriate template  from chef repo to newly created bootstrap folder

    cp chef / chef / lib / chef / knife / bootstrap /fedora13-gems.erb  /chef-repo/.chef/bootstrap/amazon-gems.erb

    Upload cookbooks  to opscode

    [ec2-user@gmadmin chef-repo]$  knife cookbook upload tomcat

    [ec2-user@gmadmin chef-repo]$  knife cookbook upload java

    [ec2-user@gmadmin chef-repo]$  knife cookbook upload apt

    Create role which includes tomcat cookbook

    [ec2-user@gmadmin chef-repo]$  nano  roles/tomcat.rb

    name "tomcat"

    description "Tomcat  Role"





    [ec2-user@gmadmin chef-repo]$  knife role from file tomcat.rb

    Use chef to instantiate an ec2 instance that is a member of the tomcat role all on one line

    [ec2-user@ip-10-234-151-231 chef-repo]$ knife ec2 server create "role[tomcat]"  --region eu-west-1  -Z eu-west-1a  -i  ami-75d4e101 -f t1.micro -G GraceHome -I ~/.ssh/GM-EC2.pem -S GM-EC2 --ssh-user ec2-user -d amazon-gems



    Knife ec2  specific flags

    --region  : AWS region , -Z : AWS Availability zone , -i :AMI ID, -f: Instance type,

     -G: Security group, -I:ssh key  file used  to ssh into instance , -S: Amazon keypair ( normally that passed to  –I & this flag will be a match) , --ssh-user:ssh user name passed to bootstrapper





  • Configuration Management and the Cloud part II – Collecting the ingredients

    Before I could enter the world of Chef I needed to at least get a handle on Ruby which is the language used to write the recipes needed (I’ll explain a little about how Chef hangs together later in this post).

    If you’re used to C# or Java then Ruby shouldn’t be that scary as it’s an OO language and treats everything as an object and is very familiar i.e. there are classes, constructs and the usual OO suspects.

    If you are going to get to grips with Chef then it helps to have some OO programming skills. Powershell is a good starting point though for those starting from windows systems management as it treats everything as objects so the concepts should not be that hard to get used to. The good news is that it is possible to use Powershell   with Chef which I will cover in a later post in this series (you still need to have a basic familiarity with ruby though sorry about that J ).

    I found reading Ruby was all about getting my head round the semantics of the language.  

    Ruby does seem to be a handy language to have a smattering of familiarity with as it seems to pop up all over the place these days so the effort involved in getting to a basic level of understanding is not wasted whether you’re going to use Chef or not.  I would suggest having a read of The little Book of Ruby to get a basic familiarisation. Needless to say there is a bigger more extensive version of this e-book J

    The other concept I needed to get my head around was RubyGems . This is a package manager for Ruby. It provides a way to manage Ruby libraries and applications. I haven’t really had time to really get to grips with this aspect beyond what I needed to  install chef but it’s on my list of things to do.

    There are also Language specific versions of Ruby JRuby for Java and IronRuby for .NET but for this exercise I decided to stick to vanilla Ruby. I had enough new things to learn without getting distracted and I do really want to get to the end of this series of posts in the near future.

    I had to prepare my workstation for Ruby use and thus chef use as ideally I wanted to use it to write any recipes and upload changes to the Chef Server repository from there.

    The first thing you need to accept is that window is a second class citizen so I anticipated getting my workstation (windows 7 pro) configured properly was not going to be trouble free (plus I’m just getting to grips with ruby)  .The description below of how I got to  a working workstation has admittedly missed out some of my pain and blind alleys I went down  but will mean anyone else following along will have an easy time of it .

    To prepare my workstation I first installed Ruby 1.9.2-p0 from the Ruby Installer for Windows Download page

    I then had to install the DevKit from the same location following the instructions from here to install it.

    I am going to be using the OpsCode platform for my Chef server (see later in this post) I thus followed the instructions from chef installation on windows  with slight modifications.


    Make sure you install rdp-ruby-wmi not ruby-wmi and you do not need the win32-open gem if you are using the same version of ruby as I am.

    gem install  rdp-ruby-wmi windows-api windows-pr


    You also need to modify the file :

    C:\Ruby192\lib\ruby\gems\1.9.1\gems\chef-0.9.12\lib\chef\mixin\command\windows.rb so it doesn’t barf as it will be trying to include the win32/open3 gem rather than the open3 gem

    #require 'win32/open3'

    require 'open3'

    I verified that the Chef Client had installed successfully by running chef-client –v



    Then I created my first chef client using the last step from   chef installation on windows

    chef-client -c c:\chef\client.rb

    I then installed & configured git as described in the Setting Up your User environment . If using the OpsCode platform it’s optimised to use git. Even if you decide not to use the OpsCode platform having an appropriate repository is a must have anyway (Luckily I had git on my list of things to look at, this just means I’m looking at it a bit earlier than anticipated).

    I checked I could connect to the OpsCode Platform using knife

    C:\Users\grace\chef-repo>knife node list




    And my first node can now be seen from the opsCode Console



    Now is a good point to have quick look at the chef architecture:

    Chef itself consists of a number of components a high level overview provided by Opscode is shown below. The individual components are described on the OpsCode site (I would suggest having at least a quick scan of this page as I reference some of the definitions later on)



    Chef comes in several flavours:

    Chef-solo is a standalone version of Chef that runs locally on your node. All the information and cookbooks required to configure the node have to be present on the local disk.

    Chef-client is a Chef agent that, like Chef-solo, runs locally on your nodes. But it connects to a Chef Server to be told what to do on the local node.

    The Opscode Platform is a highly available, scalable Chef Server in the cloud .This is the flavour I’ll be using in conjunction with Chef-client.

    The diagram below pulls out some of the components that we will be getting used to if you stick with me and the journey.



    Nodes are the systems you are managing with Chef and the Opscode Platform. Nodes are typically single servers or VMs.

    Anyway back to finishing of preparing my workstation for using the OpsCode platform. From the console you can see my workstation as I showed above, but when you click on it you can see that there is the fun stuff to be getting on with next.


    This sets the scene for the next posts in this series where I hope I’ll be able to describe how I used Chef to manage an ec2 estate by exploring the use of cookbooks, instantiating nodes etc.

  • Configuration Management and the Cloud - Part 1 getting my feet wet


    Most of my projects I’ve worked on since joining EMC Consulting have involved ALM of some flavour but having worked on a couple of projects which utilised public cloud services  I have found that I need to start considering configuration management as a whole end to end process.

    You may well ask what configuration management is.

    Well I’m not talking about the ITIL definition:

    The process of identifying and defining the Configuration Items in a system, recording and reporting the status of Configuration Items and Requests For Change, and verifying the completeness and correctness of Configuration Items.

    Configuration management as examined in this post is defined as

    The tracking and controlling of changes of the infrastructure (be that a virtual machine on your local virtual server hosting solution or a public cloud image) and the software/ applications that constitute a service.

    The key point is that in the definition I am talking about we are talking about the how to as opposed to the recording of.  Related but definitely different activities and yes the how to does feed into the Recording of so it’s all part of the same requirement in the end.

    By using appropriate configuration management solutions one can automate and scale the deployment of services thus minimising the administrative overhead associated with this. 

    ALM tools are intrinsically part of configuration management but that is far too big a subject to try and shoe horn into this post so I won’t even attempt to. Needleless to say it’s assumed here that the applications are developed and deployed using an appropriate ALM toolset. The ALM process is part of the overall configuration management process.

    With the availability of Public cloud IaaS services like AWS which supplies autoscaling features configuration of your estate can be achieved by baking in core services into the AMI and then using scripts perhaps leveraging packages like cloudinit  to further modify the instances as they spin up. This is a simple approach and works well for small estates of around 10-15 server images or less. I have used this approach successfully but I am aware of the limitations of this approach namely in terms of managing a large number of EC2 hosts with varying roles .

    When you start looking at larger numbers of server images and perhaps a mixed environment of windows and Linux servers with a range of complicated applications an enterprise configuration management solution is needed.

    Using  an enterprise configuration management solution will require a  large commitment and will need complete buy in for it to be successful,  thus the selection needs to be carefully considered Some of the areas of a very large list  which need considering when making this selection are:

    ·         Skill set of  the staff who will  be using the tools

    ·         Target environment

    ·         What exactly do you want to automatically configure

    To give the requirements justice I would need to write a post just for that but as I want to get my hands dirty sooner rather than later I’m going to gloss over that.

    In the open source world there is good choice of configuration management solutions but selecting the right one is anything but straight forward. Whichever solution is chosen will need considerable investment in terms of getting up to speed, probably learning a new scripting language and creating the configuration files that allow you to manage the changes.

    Configuration management for windows systems tends to be a relatively straight forward affair where you can use Powershell to manage the various tasks, make use of various windows specific point solutions  and  ideally if you have the budget and resources System Center Configuration manager with System Center Operations manager as an enterprise solution. (You can of course resort to third party solutions).  

    If you use VMWare or Hyper-V you can hook powershell scripts into those. Powershell is embedded into products from Exchange to SQL server so any Windows System administrator soon learns to love Powershell which is an incredibly powerful scripting language.

    Over this series of posts I will be looking at how to use an open source configuration management solution to manage an EC2 hosted environment.

    I will be coming at this from a windows background being familiar with power shell and Operations manager.  My goal being to

    ·         Find & implement  an open source configuration management  solution that fits the flexibility offered by hosting on EC2

    ·         Allows me to automatically configure a set of servers and their associated services and  applications from a central point

    ·          is not too painful to get up and running  

    ·         Is cost effective in terms of resource required and costs

    ·         Is able to manage both linux and windows



    There are a number of open source configuration management solutions which could do the job. The viable open source solutions I have found all tend to be developed initially for the Linux arena and any support for windows is patchy to say the least still I’m known to persevere.

    I trawled through a list of potential solutions whittled it down and eventually settled on Chef. Again I could go on and write another post on why I selected Chef but that would delay my journey which begins in the next post in this series now that I’ve set the scene.

    I guess if you're a regular reader of this blog then you know how varied my role is and how much I enjoy it . If you're interested in joining us here at EMC Consulting and getting the same job satisfaction I do then look no further and contact Michelle as  we're  recruiting now for a Platform Architect to join our over worked team see  Michelle's blog  

  • Get on my Cloud


    With the release of the November Windows Azure 2010 release the decision about  which Public cloud provider to use to host your .NET applications has again been muddied ( I’m not going to discuss  IaaS  versus PaaS ). There used to be a number of clear cut decision points for  example  If you had legacy applications, or were unable to factor your application to run on Windows Azure then AWS or one of the other Cloud providers like Rackspace would be a good alternative to Azure. Now however with a number of new features included as part of the update a few more options need to be considered when deciding where to host your .NET applications. I’ve pulled out a few of these new features to illustrate my point.

    Virtual Machine Roles:

    To use this you need to be able to create a custom vhd locally which at the current time needs to be a windows 2008 R2 Enterprise based vhd (There are indications that windows 2003 and windows 2008 sp2 will also be supported in 2011).

    So if your application runs on windows 2008 R2 and cannot be refactored to take advantage of Azure web and worker roles  then you can probably use the VM role ,  but if it needs to run on a previous version of Windows then it’s probably worth considering AWS at this current time.  With great power however does come more responsibility though. Using the VM role means an overhead that in my opinion goes against the idea of a PaaS solution whereby you will now be responsible for the configuration of the O/S and the ongoing maintenance of the O/S in addition to the maintenance of the application.

    Remote Desktop:

    This is obviously required if you’re using the VM Roles but you can now  Remote desktop to an instance  running on Azure . This was in response to the calls for better ability to debugging. In previous versions it was all about the API’s but now you have the ability to jump onto the actual Azure instance. Debugging via remote desktop is a familiar approach which I guess means that Azure doesn’t feel so alien to new comers and is comparable to the capabilities available with IaaS solutions.

    Extra small instance:

    Designed for smaller workloads, 1.0 GHz CPU, 768 MB RAM, 20 GB storage low I/O capabilities and a lower price point of $0.05 per hour.  AWS offers its micro instances which are capable of running windows as well which is also at low cost of $0.035.  This extra small instance  was a well timed release for Windows Azure  to allow users to experiment without incurring prohibitive costs and also to allow the introduction of dedicated roles for smaller workloads

    Windows Azure Connect:

    Extension of cloud to on premise. This now allows you to create hybrid apps, join your Azure roles to your local domain etc. It requires TCP  port 433 outbound to be open,  is agent based  and uses IPSec and IPV6 . AWS has its AWS VPC which  take a different approach and you need to  essentially set up  a site to site IPsec VPN. There are no new networking components to install on your end nodes when using the AWS approach.

    Upgrading to 1.3 how it affects Visual Studio:

    If you are not a visual studio 2010 user then be careful. I got this:


  • The Public Cloud and Platform Agility


    I’ve been working on an AWS hosted project  and it’s been a lot of fun the only real problem has been keeping up with the changes that Amazon has introduced in the relatively short time span of the project

    In traditional projects where we typically deploy to Data Centres the platform is pretty much fixed early in the development process. Essentially this is  because a  contract will have been engaged for a specific set of services up front, it takes longer for Traditional Data Centre suppliers to provision a platform  and only small changes can be made subsequent to this such as  network changes, maybe a few more virtual machines etc . So a lot of my time would typically be involved with negotiations with the DC supplier and trying to get the platform sorted as far ahead as possible. This is not ideal for a number of reasons so the arrival of the Public cloud, with self service portals, pay as you go approach, introduction of new features and the ability to try stuff out without a commitment up front has meant that the platform evolves as the application does.  The platform within reasons is no longer a fixed constraint.  

    I will note here though that traditional Data Centres are stepping up to address the new levels of flexibility whether they are dressing this up as their own version of the cloud or just introducing  a more appropriate pricing and servicing model .

    This evolution doesn’t mean the platform is not defined up front as you do need to know what your platform will be not least because the developers need to know what they will be targeting; the scaling approach needs to be understood up front   but also so the client needs to have some idea of costs. This may sound contradictory to my earlier statement but the difference is that although all the usual decision points that apply to the platform for any project are still made the boundaries around the platform are a lot looser. Certain key decisions can now be made a lot later and so more flexibility is inherent without impacting on the total cost of the solution.

     Before this particular project I do not recall working on a project where the platform costs have gone down over the development cycle of the project.  The scary thing is how fast the features have been pouring out of Amazon which has meant that the actual platform provisioning has had to be as agile as the rest of the development process reflecting how the approach to my role has had to adapt as a result. The Amazon AWS feature changes have not really affected the ultimate target platform in any major way as defined early in the project but there were a number of changes and subtle decisions made as a result of some of the additions.  A few of which were considered and in most cases directly affected the solution I’ve listed below (all of which appeared since July).


    IAM: identity & Access management:


    This feature allows the creation of virtual users with delegated authorization all under a single AWS account

    This arrived too late for us as we had to set up individual AWS accounts for the development & test team at the beginning of the project which we then set up consolidated billing for.

     Micro instances:

     A lower priced instance type that could satisfy the needs of their less demanding applications. , It is a low cost instance type designed for lower throughput applications and web sites. Micro instances provide 613 MB of. Micro instances provide a small amount of consistent CPU resources, and also allow you to burst CPU capacity when additional cycles are available.

    Arrived just in time really. The scaled out topology meant the front end web servers could happily run on micro instances and the cost savings will make a significant difference.

    Amazon AWS Linux AMI:

    The Amazon Linux AMI is a supported and maintained Linux image  optimized for use on EC2. It is designed to provide a stable, secure, and high performance execution environment. It also includes several packages that enable easy integration with AWS, including launch configuration tools and many popular AWS libraries and tools. Amazon Web Services also provides ongoing security and maintenance updates to all instances running the Amazon Linux AMI.

    This also arrived just in time and although some time had been spent on creating a Linux AMI from scratch where I had adventures with kernel mismatches & creating a new Linux instance using EBS volumes the majority of work carried out was easily transferable to the Amazon Linux AMI.

    RDS Reserved instances and Multi-AZ Deployments:

    These were just two of a number of RDS enhancements over the lifespan of the project.

    Reserved instances are similar to reserved Ec2 instances. Reserved RDS instances provide a significant discount if a one off fee is made up front. This made sense as reserved instances were part of the upfront commitments for the solution

    RDS instances can be configured in multiple Availability Zones which provides asynchronous standby replica in a different availability zone.  The primary DB Instance is synchronously replicated across Availability Zones to the standby replica to provide data redundancy, eliminate I/O freezes during system backups, enhance availability during planned system maintenance, and help protect the databases against DB Instance failure and Availability Zone disruption. This is a feature that also needs to be implemented.


    Resource tagging:


    The ability to tag the following types of resources: EC2 instances, Amazon Machine Images (AMIs), EBS volumes, EBS snapshots, and Amazon VPC resources such as VPCs, subnets, connections, and gateways.

    With performance testing of different size instances, UAT, pre-prod , prod environments slave and master instances all running. A simple but needed feature to make it easy to identify what environment or size instances were running. Our Tester made copious use of this feature. It arrived in time for us

    Downloadable invoices:

    I had to include this one in as it was a step in the right direction for corporates to embrace AWS and it also  made the financial part of the project a lot easier.

    Free usage Tier:

    AWS Free Usage Tier a new free usage tier. Beginning November 1, new AWS customers will be able to run a free Amazon EC2 Micro Instance for a year, while also leveraging a new free usage tier for Amazon S3, Amazon Elastic Block Store, Amazon Elastic Load Balancing, and AWS data transfer.

    Bah humbug this arrived too late but I had to mention it as what an enticement to those too hesitant to step over the precipice to try out the public cloud  go on give it a  go .



  • Monitoring Applications deployed on PaaS (Windows Azure) … Business as usual

    Monitoring applications provided by PaaS such as Azure should in theory be all about the application as the underlying platform isn’t your problem.  Well it’s true that you don’t have control of the actual Platform (Platform in this context being the Virtual server host and the actual virtual machines), but you do still need to understand when there’s a problem with the underlying infrastructure even if it’s so you can harangue the supplier to sort the problem out; to implement contingency plans or to implement notification processes.

    Obtaining alerts related to the state of the infrastructure applies whether it’ IaaS or PaaS. Both Amazon and Microsoft provide dashboards indicating the status of their services. Amazing how similar they look!!!

    Amazon AWS dashboard :                                                             



    Microsoft windows Azure Dashboard: 




    In the case of Amazon AWS you are responsible for the Virtual machine and there is an API that you can take advantage of as described in my previous post on monitoring Amazon  AWS .

    Whether you develop an application that is to be delivered via on Premise, IaaS or PaaS you still need to be able to monitor the status of your application and implement appropriate action.  In addition to understanding what metrics would trigger some sort of scaling action you also need to understand the APIs that allow you to effect that scaling.

    The monitoring/management requirements are the same whatever the cloud platform of choice. Being in the cloud does not remove this requirement all it does is require a slightly different approach to what one may be used to. Providing the ability to collate useful diagnostic data about an application is a development task no matter what the delivery platform.  PaaS services generally provide API’s that allow your application to hook into diagnostic mechanisms. The data collated is then used to allow your application to scale appropriately ( e.g by starting more web roles) or to recycle.

    The rest of this post I’ll be talking about the use of Windows Azure Diagnostics as an example of how one can accomplish the typical tasks of collating appropriate metrics on PaaS (I am assuming a basic level of familiarity with Windows Azure from here on in now though).

    Windows Azure achieves this by providing access to Azure Diagnostics via an API. This allows you to collect diagnostic data from a service running in Windows Azure. Metrics that can be collected out of the box (with little bit of configuration) are  detailed at : http://msdn.microsoft.com/en-us/library/ee843890.aspx They include Windows Azure Logs, IIs 7.0 Logs, Windows Diagnostics Infrastructure Logs, event logs and performance counters . The sort of thing you’d expect from traditional deployments.

    To access the diagnostics API is relatively painless as its part of the Azure SDK.  Using Visual Studio to create an Azure Cloud application creates the stubs to allow you to hook into the Azure Diagnostics API.

    The Screen shot below shows the generated code for the web role. You get a corresponding entry in the worker role


    In the associated config file ( web.config for the web role or app.config for the worker role)  you will see that the hooks to the Diagnostics  listener is now also set .


    Great so how do you use it?

    I’ve got a simple application that queries SQL server for a bunch of security controls using a grid view to display the results. What the application does is irrelevant though for this post. It’s not pretty but at least  I  know my data and it’s one less thing to get to grips with as   I re-familiarise myself with Azure ,which I have I admit neglected as I’ve been working with AWS recently.


    The diagnostic data is written to Azure storage and this is where it’s analysed. The ServiceConfiguration.csfg file is where you change the storage account that is used. For this outline I’ll leave it pointing at Development storage


    So what happens when I now start my simple web app?


    In blob storage a container is created call wad-control-container which indicates what diagnostic data is being collated by default for your deployment



    This data is stored as XML and indicates the default configuration data collated:


    I want to be able to collect some  data on IIs logs & from  the event view so to do this I will need to go back to my application specifically the OnStart method of my Webrole which looks like this before I add any custom code :



    To start collecting the log data I then modified this method so it looks like this:


    The above example sets the transfer schedule of data to Table storage to 1 minute intervals.



    (I did try and collect performance stats but wasn’t having much joy. I’ll revisit when I have more time maybe when the application has evolved somewhat beyond a simple WebRole)

    The Log data gets written to a Table called WADLogsTable


    The Infrastructure logs get written to a table called WADDiagosticinfrsatructureLogsTable 


    So now I know how to add code to my Azure Application to collect metrics. The next logical step is to alert on that data appropriately as well as take action when certain thresholds are reached like starting restarting a  WebRole but that is probably best for another post.

    There is now a fair amount of information   on implementing Azure Diagnostics into your application so I won’t spend any more time on outlining the concepts but I would recommend reading http://msdn.microsoft.com/en-us/magazine/ff714589.aspx as an excellent introduction to implementing Azure Diagnostics.



     What is collected and how that data is used is intimately tied into the application and its behaviour.  If you are ultimately responsible for service delivery then you need to build a relationship with the development team and make sure your requirements are incorporated as part of the deliverables.  This requirement is the same whether the application is being developed for the cloud whatever particular flavour that may be or for on premise.



  • Monitoring your AWS estate using Cloudwatch

    Anyone tasked with setting up  IT Services be they in the cloud public or private, virtual or  physical will need to manage those resources .To be able to do that you need to be able to monitor stuff and use those results to provide proactive management.

    Amazon AWS is primarily an IaaS cloud service (although they are fast morphing into providing PaaS)  and as such you would be expected to be able to monitor typical  metrics. You can do this by accessing pef mon stats in windows or the equivalent on Linux but Amazon AWS offers a neater more secure way of obtaining this  type of data .

    Amazon AWS exposes a number of  metrics  via  a web service called  Cloudwatch . The use of these metrics enables Autoscaling which is a nice reason for using cloud services. You only incur Amazon CloudWatch fees for Amazon EC2 instances you choose to monitor. There are no additional charges for monitoring Amazon EBS volumes, Elastic Load Balancers or RDS Database Instances.

    It’s a simple checkbox via the AWS console to switch on Cloudwatch monitoring for instances . You can also indicate as part of the command line start-up commands that you want monitoring turned on . Monitoring for EBS and RDS is set up automatically. 

    My initial explorations into monitoring on AWS  took an interesting route as I explored using the AWS console, an app for my phone , the Command line tools and a plug in for a popular monitoring solution. I did start looking at rolling my own wrapper using java but that was just a little too far out of my comfort zone and  something I didn’t have time for especially as all the examples I looked at were using the deprecated Java Library for AWS rather than the AWS SDK for Java. I’d have to spend time getting to know he SDK  and attempting to write something from scratch or refactoring an example but we have a team of Java Devs who can do that sort of thing better than I ever will be able to  :-) .

    There is a .NET SDK and at  some point when I get a chance I’ll probably have a look at using that . The projects I was looking at this for were Java/open source  based  and I wanted to stay within that arena  so straying into .NET was off limits.

    The times are in UTC so for us in the UK fairly easy to work out . You can view data from the last hour through to the previous 2 weeks . So if you need  data for a longer period than 2 weeks to assist with capacity planning then you’ll need to archive it

    AWS Console

    Once you have monitoring switched on your instances you are able to use the AWS console to access graphs of the metrics. You can view data up to 2 weeks , and

    Instance  Metrics:


    You are then able to drill down on any particular graph:



    RDS Metrics:


    EBS Metrics:


    This is fine for initial set up/ development  but not really practical as a  way to monitor your AWS estate . It’s cumbersome , and there is no way to set alerting from here .  As nice as it is to have pretty graphs you will not be watching them 24 x 7 via this route .

    Command Line Tools

    If you’ve downloaded the API tools you can  Use the command line tools to see the list of metrics exposed via this. It exposes metrics for EC2 instances, RDS and EBS.  This entails setting up the API tools so you can run them from the command lines ( setting up env variables, telling it where Java is , where the command tools are, your authentication details etc )  you then run fairly ugly commands as the example below where I’m grabbing the CPU stats from my EC2 estate ( In this case a single instance)

    mon-get-stats CPUUtilization --start-time 2010-07-11T23:00:00.000Z --end-time 2010-07-12T23:00:00.000Z --period 3600 --statistics "Average,Minimum,Maximum" --namespace "AWS/EC2"

    Which spits out the metrics:



    Not pretty but quick and dirty and gets the job done. You can pretty much understand why wrapping this stuff up is a must do really.

    Monitoring from my phone (Decaff)

    Most people have their phone close by most of the time so using your phone as  an integral part of your monitoring solution makes sense. There are a couple of solutions for Android phones but I’d read good things about  Decaff so thought I’d try that one. So I paid my money downloaded and installed the app.  Decaff  allows you to monitor and manage many aspects of the aws services . It also includes the ability to set up a watch on specific ports.  The images below illustrate  a few of the features.

    Exif_JPEG_PICTURE                                                       Exif_JPEG_PICTURE                                                  Exif_JPEG_PICTURE                                              Exif_JPEG_PICTURE                                              Exif_JPEG_PICTURE                                               

    I found the interface very usable and responsive and soon opted for that to launch instances, stop and terminate them as well as using it to provide an at a glance over view of how the systems were doing.   Check out the 9Apps blog which has a great description of how they  approached creating the application .

    3rd party Solutions

    There is a whole eco system that has built up around the AWS services and as such there are a number of 3rd party solutions available either hosted or you can incorporate into your own environment. I  started by looking  at PRTG as I have used it before and its still  a popular monitoring solution. The fact they had a sensor for cloudwatch I admit  heavily influenced me  at wanting to  look at this as an example rather than any alternatives. Unfortunately  I was unable to get the sensor to pick up any data. There are only a  few things to set up  and so far I have no idea why it’s not working  and to date their support have been unable to help either. If it ever gets resolved I’ll update this post . A review of 3rd party solutions would deserve a post all of its own and would mean I’d never get this post actually posted so forgive the lack of in depth analysis and I promise to revisit another day.


    Cloudwatch exposes the core metrics you would expect and the fact it can be accessed in a variety of ways relatively easily should be another tick on the list for those Operational staff tasked with making sure they can manage their AWS estate as easily as their current infrastructure.  I love the fact you don’t have to install agents to obtain these metrics  and thus no extra ports required to be opened .The use of cloud watch to  enable auto scaling is what makes the use of  cloud watch in my opinion a no brainer. Yes it does  cost  $0.015  an hour which I guess soon adds up  but that is money well spent to allow you to provide a pro active solution for monitoring and managing your AWS estate with very little effort . The issue with the  cost of monitoring using Cloudwatch which may seem out of step with the actual cost of running an instance  but that is a subject for another day .

    A safe  solution is to make use of  3rd party tools  such as Cacti or PRTG ( I’m assuming if you were a  paying customer any teething issues would be sorted out quickly)  or possible  a  hosted solution, together with a mobile solution like Decaff.  I’m not that comfortable with writing Java and using eclipse  ( remember I banned myself from using the .NET AWS SDK)  but   if you have access to some Java Developers then it won’t be a problem for them to  write a wrapper to use  the Cloudwatch API (which is what I’d do given the time)  . What suits you will obviously depend on what you use today and what your requirements are in terms of managing the estate.

  • A matter of choice

    I am usually bemused by the almost fanatical attitude of Apple fan boys, Linux hardcore advocates , android lovers etc etc.  The reason why is I have always believed that it's a matter of choice and one size does not fit all.
    The fact I went for an android phone rather than an iphone was my choice as I prefer to have a little more freedom to install what I like ( and I'll accept the consequences) than what Apple give you , I'm also not a fan of itunes and I actually think my new phone is very very nice :-) . I appreciate the fact that anything coming out of Apple is likely to just work very well and be a beautifully designed bit of kit so for the majority of folks who just want something that will work that's good enough but hey guys you don't need to diss all non apple lovers its just not that big a deal that you have an iphone /ipod/ ipad  or whatever. Without choice innovation will be stifled. Just look at the tit for tat law suits and behind door deals between the mobile suppliers to see that one company cannot have a monopoly on ideas and innovation.

    So that comes to my walk on the dark side where choice is what it’s all about.

    My natural habitat is in the nice safe confines of the Microsoft world. I have in the past wandered over into the world of Linux and have  only recently retired my Linux laptop as in the end windows  was able to run the stuff I wanted to easier .

    So I was continuing my flight in the cloud  this time it's Amazons aws and using  Linux  as the starting point.  One of the first questions I had to ask was what flavour Linux ? Whereas in the windows world it's take the latest unless your desired solution will not work with it. With Linux it was well you can have a striped one or a maybe a  polka dotted  one .. The number of Linux distributions is quite overwhelming but luckily being agnostic as an application consultant I had plenty of access to  resources to discuss the pros and cons with while always bearing in mind the Clients requirements & desires . There was also the fact I needed a degree of confidence in the base Image I would start off with so taking an image provided by Joe Blogs probably wouldn't be the route anyway so that did limit the choice as well. So it's back to choices . Then there was the kernel version... This was key as only certain kernels are supported by the underlying Xen hypervisor used by the aws service  :-(  more on that later needless to say it's  a whole new set of challenges from here on in .

    Thanks to Simon for the chat that inspired this random ramble.

Powered by Community Server (Personal Edition), by Telligent Systems