Welcome to EMC Consulting Blogs

I apologise for the silence  over the past few months but My posts have moved to Devops and the Public Cloud

Before I go on I apologise up front but I just can’t resist the cooking theme

The purpose of this series of blog posts is to explore using Chef to configure instances in the cloud. My particular cloud for this just happens to be AWS.

In this post I’ll be describing how to actually use Chef to spin up & configure an ec2 instance. I am making the assumption you are familiar with the contents of the previous two posts in this series. Even if you don’t have any experience with  AWS I hope it will still be easy enough to follow.

In my last post I described the configuration of a windows workstation to prepare for this but I hit a problem which meant I have had to resort to using a Linux instance running on AWS. I have a bug logged with OpsCode. For those of you who are keen to make use of your windows machines as your user environment I will post an update when the bug is resolved.

I am starting off with spinning up a Linux instance as it will be easier to grasp the concepts and what you need to do to use Chef effectively. My last post in this series will deal with using chef with a windows target.

The first thing I needed to do was walk through what I wanted to achieve , in what order .

I wanted to  spin up an ec2 instance and install tomcat so Chef will need to:

·         Fire up an ec2 instance

·         Bootstrap it so it can run a chef-client

·         Install java  if required ( Note as I am using an aws linux instance java is preinstalled)

·         Install tomcat

Yes I know this can all be easily done using cloudinit and passing user data to your instance but I feel it’s pretty self evident how Chef or a similar tool would be a more appropriate enterprise solution with a large target ec2 estate with various roles .  The simplicity of the example is to illustrate the steps that need to be gone through.

The really cool thing with Chef is that you can rerun cookbooks against a node and it will not do anything it has already done i.e it will not change the end result on the target node as defined by the recipes being run against it. So you will always get the same outcome no matter what state the node and actions will not be taken if already done (and conversely run if detected it has not been run).  When reading about Chef you will see this described as being idempotent (There I’ve saved you looking it up).

At this stage I have a chef ready user environment, an OpsCode organisation set up and now I want to start by spinning up an ec2 instance.

I will just be talking about the key chef configuration steps for the rest of this post but if you are following along then use the table at the end of this post that details all he steps I followed. I will not be going into any depth re the ec2 specifics as that would make this post far too long.

The first thing I needed to do was create a cookbook for installing tomcat.  The Chef community (admittedly in the opensource world) though is full of off the shelf recipes that you can use to get started with.

I did a quick search from the OpsCode cookbooks section of the web console to see if there were any tomcat cookbooks 

I could have downloaded it from the console but I feel it was neater to use knife .

I also need the cookbook java  and apt cookbooks .

Before proceeding it would probably be a good idea to take time out and read the Opscode  Chef Recipe wiki   which has a nice clear explanation on cookbook name spaces. Also remind yourself of the components that make up a cookbook  it’s worth noting that recipes manage resources and those resources will be executed in the order they occur.

As my target platform will be ‘amazon’ I decided to amend the default.rb to reflect that platform.  The Amazon AWS linux ami is based on centos so it was a minor change to add the platform.  I  did not need to do this modification as the centos bootstrap will work fine with no changes  but I felt it was a worthwhile exercise as it shows the sorts of steps you would need to go through typically.

I then downloaded the Chef repository so that I could obtain a template bootstrap file. The purpose of the bootstrap file is to install chef on the target node and set the node up as a chef client

At this stage I have a cookbook with a recipe ready to upload and run and I have a bootstrap file.  Nearly ready to try it all out but before I stitch it all together I need to introduce you to the concept of roles.

I then uploaded my modified tomcat cookbook to my OpsCode chef instance and created a role that would basically run the tomcat recipe in this cookbook.

I then ran the following command to instantiate my ec2 instance:

knife ec2 server create "role[tomcat]"  --region eu-west-1  -Z eu-west-1a  -i  ami-75d4e101 -f t1.micro -G GraceHome -I ~/.ssh/GM-EC2.pem -S GM-EC2 --ssh-user ec2-user -d amazon-gems

Note: see table at end for explanation of the ec2 specific flags

On running the command a long series of actions occur which you can review via a log file

When I saw  the following I then verified everything was  completed as expected:

The new node appears in my node list:

I can now consistently reproduce as many tomcat nodes as I want. What I’m not too sure of is how I can tie this in with AWS  auto scaling  but figure that’s something to look into later.

The table below assumes that your user environment workstation is also set up to use the ec2 api tools

Table 1: Configuration steps for starting an  AWS Amazon  Linux ec2 instance and installing tomcat

Most of my projects I’ve worked on since joining EMC Consulting have involved ALM of some flavour but having worked on a couple of projects which utilised public cloud services  I have found that I need to start considering configuration management as a whole end to end process.

You may well ask what configuration management is.

Well I’m not talking about the ITIL definition:

The process of identifying and defining the Configuration Items in a system, recording and reporting the status of Configuration Items and Requests For Change, and verifying the completeness and correctness of Configuration Items.

Configuration management as examined in this post is defined as

The tracking and controlling of changes of the infrastructure (be that a virtual machine on your local virtual server hosting solution or a public cloud image) and the software/ applications that constitute a service.

The key point is that in the definition I am talking about we are talking about the how to as opposed to the recording of.  Related but definitely different activities and yes the how to does feed into the Recording of so it’s all part of the same requirement in the end.

By using appropriate configuration management solutions one can automate and scale the deployment of services thus minimising the administrative overhead associated with this. 

ALM tools are intrinsically part of configuration management but that is far too big a subject to try and shoe horn into this post so I won’t even attempt to. Needleless to say it’s assumed here that the applications are developed and deployed using an appropriate ALM toolset. The ALM process is part of the overall configuration management process.

With the availability of Public cloud IaaS services like AWS which supplies autoscaling features configuration of your estate can be achieved by baking in core services into the AMI and then using scripts perhaps leveraging packages like cloudinit  to further modify the instances as they spin up. This is a simple approach and works well for small estates of around 10-15 server images or less. I have used this approach successfully but I am aware of the limitations of this approach namely in terms of managing a large number of EC2 hosts with varying roles .

When you start looking at larger numbers of server images and perhaps a mixed environment of windows and Linux servers with a range of complicated applications an enterprise configuration management solution is needed.

Using  an enterprise configuration management solution will require a  large commitment and will need complete buy in for it to be successful,  thus the selection needs to be carefully considered Some of the areas of a very large list  which need considering when making this selection are:

·         Skill set of  the staff who will  be using the tools

·         Target environment

·         What exactly do you want to automatically configure

To give the requirements justice I would need to write a post just for that but as I want to get my hands dirty sooner rather than later I’m going to gloss over that.

In the open source world there is good choice of configuration management solutions but selecting the right one is anything but straight forward. Whichever solution is chosen will need considerable investment in terms of getting up to speed, probably learning a new scripting language and creating the configuration files that allow you to manage the changes.

Configuration management for windows systems tends to be a relatively straight forward affair where you can use Powershell to manage the various tasks, make use of various windows specific point solutions  and  ideally if you have the budget and resources System Center Configuration manager with System Center Operations manager as an enterprise solution. (You can of course resort to third party solutions).  

If you use VMWare or Hyper-V you can hook powershell scripts into those. Powershell is embedded into products from Exchange to SQL server so any Windows System administrator soon learns to love Powershell which is an incredibly powerful scripting language.

Over this series of posts I will be looking at how to use an open source configuration management solution to manage an EC2 hosted environment.

I will be coming at this from a windows background being familiar with power shell and Operations manager.  My goal being to

·         Find & implement  an open source configuration management  solution that fits the flexibility offered by hosting on EC2

·         Allows me to automatically configure a set of servers and their associated services and  applications from a central point

·          is not too painful to get up and running  

·         Is cost effective in terms of resource required and costs

·         Is able to manage both linux and windows

There are a number of open source configuration management solutions which could do the job. The viable open source solutions I have found all tend to be developed initially for the Linux arena and any support for windows is patchy to say the least still I’m known to persevere.

I trawled through a list of potential solutions whittled it down and eventually settled on Chef. Again I could go on and write another post on why I selected Chef but that would delay my journey which begins in the next post in this series now that I’ve set the scene.

I guess if you're a regular reader of this blog then you know how varied my role is and how much I enjoy it . If you're interested in joining us here at EMC Consulting and getting the same job satisfaction I do then look no further and contact Michelle as  we're  recruiting now for a Platform Architect to join our over worked team see  Michelle's blog  

I’ve been working on an AWS hosted project  and it’s been a lot of fun the only real problem has been keeping up with the changes that Amazon has introduced in the relatively short time span of the project

In traditional projects where we typically deploy to Data Centres the platform is pretty much fixed early in the development process. Essentially this is  because a  contract will have been engaged for a specific set of services up front, it takes longer for Traditional Data Centre suppliers to provision a platform  and only small changes can be made subsequent to this such as  network changes, maybe a few more virtual machines etc . So a lot of my time would typically be involved with negotiations with the DC supplier and trying to get the platform sorted as far ahead as possible. This is not ideal for a number of reasons so the arrival of the Public cloud, with self service portals, pay as you go approach, introduction of new features and the ability to try stuff out without a commitment up front has meant that the platform evolves as the application does.  The platform within reasons is no longer a fixed constraint.  

I will note here though that traditional Data Centres are stepping up to address the new levels of flexibility whether they are dressing this up as their own version of the cloud or just introducing  a more appropriate pricing and servicing model .

This evolution doesn’t mean the platform is not defined up front as you do need to know what your platform will be not least because the developers need to know what they will be targeting; the scaling approach needs to be understood up front   but also so the client needs to have some idea of costs. This may sound contradictory to my earlier statement but the difference is that although all the usual decision points that apply to the platform for any project are still made the boundaries around the platform are a lot looser. Certain key decisions can now be made a lot later and so more flexibility is inherent without impacting on the total cost of the solution.

 Before this particular project I do not recall working on a project where the platform costs have gone down over the development cycle of the project.  The scary thing is how fast the features have been pouring out of Amazon which has meant that the actual platform provisioning has had to be as agile as the rest of the development process reflecting how the approach to my role has had to adapt as a result. The Amazon AWS feature changes have not really affected the ultimate target platform in any major way as defined early in the project but there were a number of changes and subtle decisions made as a result of some of the additions.  A few of which were considered and in most cases directly affected the solution I’ve listed below (all of which appeared since July).

This feature allows the creation of virtual users with delegated authorization all under a single AWS account

This arrived too late for us as we had to set up individual AWS accounts for the development & test team at the beginning of the project which we then set up consolidated billing for.

 Micro instances:

 A lower priced instance type that could satisfy the needs of their less demanding applications. , It is a low cost instance type designed for lower throughput applications and web sites. Micro instances provide 613 MB of. Micro instances provide a small amount of consistent CPU resources, and also allow you to burst CPU capacity when additional cycles are available.

Arrived just in time really. The scaled out topology meant the front end web servers could happily run on micro instances and the cost savings will make a significant difference.

Amazon AWS Linux AMI:

The Amazon Linux AMI is a supported and maintained Linux image  optimized for use on EC2. It is designed to provide a stable, secure, and high performance execution environment. It also includes several packages that enable easy integration with AWS, including launch configuration tools and many popular AWS libraries and tools. Amazon Web Services also provides ongoing security and maintenance updates to all instances running the Amazon Linux AMI.

This also arrived just in time and although some time had been spent on creating a Linux AMI from scratch where I had adventures with kernel mismatches & creating a new Linux instance using EBS volumes the majority of work carried out was easily transferable to the Amazon Linux AMI.

RDS Reserved instances and Multi-AZ Deployments:

These were just two of a number of RDS enhancements over the lifespan of the project.

Reserved instances are similar to reserved Ec2 instances. Reserved RDS instances provide a significant discount if a one off fee is made up front. This made sense as reserved instances were part of the upfront commitments for the solution

RDS instances can be configured in multiple Availability Zones which provides asynchronous standby replica in a different availability zone.  The primary DB Instance is synchronously replicated across Availability Zones to the standby replica to provide data redundancy, eliminate I/O freezes during system backups, enhance availability during planned system maintenance, and help protect the databases against DB Instance failure and Availability Zone disruption. This is a feature that also needs to be implemented.

The ability to tag the following types of resources: EC2 instances, Amazon Machine Images (AMIs), EBS volumes, EBS snapshots, and Amazon VPC resources such as VPCs, subnets, connections, and gateways.

With performance testing of different size instances, UAT, pre-prod , prod environments slave and master instances all running. A simple but needed feature to make it easy to identify what environment or size instances were running. Our Tester made copious use of this feature. It arrived in time for us

Downloadable invoices:

I had to include this one in as it was a step in the right direction for corporates to embrace AWS and it also  made the financial part of the project a lot easier.

Free usage Tier:

AWS Free Usage Tier a new free usage tier. Beginning November 1, new AWS customers will be able to run a free Amazon EC2 Micro Instance for a year, while also leveraging a new free usage tier for Amazon S3, Amazon Elastic Block Store, Amazon Elastic Load Balancing, and AWS data transfer.

Bah humbug this arrived too late but I had to mention it as what an enticement to those too hesitant to step over the precipice to try out the public cloud  go on give it a  go .

Anyone tasked with setting up  IT Services be they in the cloud public or private, virtual or  physical will need to manage those resources .To be able to do that you need to be able to monitor stuff and use those results to provide proactive management.

Amazon AWS is primarily an IaaS cloud service (although they are fast morphing into providing PaaS)  and as such you would be expected to be able to monitor typical  metrics. You can do this by accessing pef mon stats in windows or the equivalent on Linux but Amazon AWS offers a neater more secure way of obtaining this  type of data .

Amazon AWS exposes a number of  metrics  via  a web service called  Cloudwatch . The use of these metrics enables Autoscaling which is a nice reason for using cloud services. You only incur Amazon CloudWatch fees for Amazon EC2 instances you choose to monitor. There are no additional charges for monitoring Amazon EBS volumes, Elastic Load Balancers or RDS Database Instances.

It’s a simple checkbox via the AWS console to switch on Cloudwatch monitoring for instances . You can also indicate as part of the command line start-up commands that you want monitoring turned on . Monitoring for EBS and RDS is set up automatically. 

My initial explorations into monitoring on AWS  took an interesting route as I explored using the AWS console, an app for my phone , the Command line tools and a plug in for a popular monitoring solution. I did start looking at rolling my own wrapper using java but that was just a little too far out of my comfort zone and  something I didn’t have time for especially as all the examples I looked at were using the deprecated Java Library for AWS rather than the AWS SDK for Java. I’d have to spend time getting to know he SDK  and attempting to write something from scratch or refactoring an example but we have a team of Java Devs who can do that sort of thing better than I ever will be able to  :-) .

There is a .NET SDK and at  some point when I get a chance I’ll probably have a look at using that . The projects I was looking at this for were Java/open source  based  and I wanted to stay within that arena  so straying into .NET was off limits.

The times are in UTC so for us in the UK fairly easy to work out . You can view data from the last hour through to the previous 2 weeks . So if you need  data for a longer period than 2 weeks to assist with capacity planning then you’ll need to archive it

AWS Console

Once you have monitoring switched on your instances you are able to use the AWS console to access graphs of the metrics. You can view data up to 2 weeks , and

Instance  Metrics:

You are then able to drill down on any particular graph:

RDS Metrics:

EBS Metrics:

This is fine for initial set up/ development  but not really practical as a  way to monitor your AWS estate . It’s cumbersome , and there is no way to set alerting from here .  As nice as it is to have pretty graphs you will not be watching them 24 x 7 via this route .

Command Line Tools

If you’ve downloaded the API tools you can  Use the command line tools to see the list of metrics exposed via this. It exposes metrics for EC2 instances, RDS and EBS.  This entails setting up the API tools so you can run them from the command lines ( setting up env variables, telling it where Java is , where the command tools are, your authentication details etc )  you then run fairly ugly commands as the example below where I’m grabbing the CPU stats from my EC2 estate ( In this case a single instance)

mon-get-stats CPUUtilization --start-time 2010-07-11T23:00:00.000Z --end-time 2010-07-12T23:00:00.000Z --period 3600 --statistics "Average,Minimum,Maximum" --namespace "AWS/EC2"

Which spits out the metrics:

Not pretty but quick and dirty and gets the job done. You can pretty much understand why wrapping this stuff up is a must do really.

Monitoring from my phone (Decaff)

Most people have their phone close by most of the time so using your phone as  an integral part of your monitoring solution makes sense. There are a couple of solutions for Android phones but I’d read good things about  Decaff so thought I’d try that one. So I paid my money downloaded and installed the app.  Decaff  allows you to monitor and manage many aspects of the aws services . It also includes the ability to set up a watch on specific ports.  The images below illustrate  a few of the features.

I found the interface very usable and responsive and soon opted for that to launch instances, stop and terminate them as well as using it to provide an at a glance over view of how the systems were doing.   Check out the 9Apps blog which has a great description of how they  approached creating the application .

3rd party Solutions

There is a whole eco system that has built up around the AWS services and as such there are a number of 3rd party solutions available either hosted or you can incorporate into your own environment. I  started by looking  at PRTG as I have used it before and its still  a popular monitoring solution. The fact they had a sensor for cloudwatch I admit  heavily influenced me  at wanting to  look at this as an example rather than any alternatives. Unfortunately  I was unable to get the sensor to pick up any data. There are only a  few things to set up  and so far I have no idea why it’s not working  and to date their support have been unable to help either. If it ever gets resolved I’ll update this post . A review of 3rd party solutions would deserve a post all of its own and would mean I’d never get this post actually posted so forgive the lack of in depth analysis and I promise to revisit another day.

Conclusions:

Cloudwatch exposes the core metrics you would expect and the fact it can be accessed in a variety of ways relatively easily should be another tick on the list for those Operational staff tasked with making sure they can manage their AWS estate as easily as their current infrastructure.  I love the fact you don’t have to install agents to obtain these metrics  and thus no extra ports required to be opened .The use of cloud watch to  enable auto scaling is what makes the use of  cloud watch in my opinion a no brainer. Yes it does  cost  $0.015  an hour which I guess soon adds up  but that is money well spent to allow you to provide a pro active solution for monitoring and managing your AWS estate with very little effort . The issue with the  cost of monitoring using Cloudwatch which may seem out of step with the actual cost of running an instance  but that is a subject for another day .

A safe  solution is to make use of  3rd party tools  such as Cacti or PRTG ( I’m assuming if you were a  paying customer any teething issues would be sorted out quickly)  or possible  a  hosted solution, together with a mobile solution like Decaff.  I’m not that comfortable with writing Java and using eclipse  ( remember I banned myself from using the .NET AWS SDK)  but   if you have access to some Java Developers then it won’t be a problem for them to  write a wrapper to use  the Cloudwatch API (which is what I’d do given the time)  . What suits you will obviously depend on what you use today and what your requirements are in terms of managing the estate.