|
|
-
Much of the resistance to cloud computing is by operational IT staff. Some of this resistance is based on real concerns; most of it is because of the fear of loss of control. IT Pros seem to fear that the moment the physical, on site datacentre disappears that they will lose their power base and jobs. Service rather than server The constant message from cloud computing experts is not that jobs will be lost, but jobs will change – dramatically in some cases. The biggest change is the shift to being responsible for the availability of a service, rather than a server. The subtle distinction means that skills of operational staff need to move up the stack and away from the hardware. Application Support IT Operations will need to understand and support more of the actual applications than just the web server and everything underneath. They will need to understand more about how the application behaves, how data is moved around various nodes, how the system handles failure scenarios and what their own role is when things go wrong. It will no longer be a case of palming of the support of the application to the vendor or application support team as there will be aspects contained within the overall cloud ecosystem that application people have no clue about. Vendor APIs In large enterprises it is likely that there will be many cloud computing platforms, from multi vendor private clouds to rogue applications hosted on any number of public cloud hosting platforms. Ops staff are going to have to master the nuances of how each platform behaves and understand the various APIs so that they can write their own automation scripts to perform tasks specific to the organization. IT Management The biggest challenge will be for IT management. Without a gleaming data centre with lots of impressive blinking lights to back them up they need to find ways to pitch their value to their own (internal) customers. While cloud computing allows anyone to whip out a credit card and bypass procurement of hardware, or filling in a requisition to the corporate data centre, we know that the provisioning of solution infrastructure is more than just ordering hardware and this point needs to be communicated to business. IT Ops management need to facilitate migration to cloud computing by engaging actively with business (so that they are seen as trusted advisors) and their own staff in order to keep the business specific knowledge while migrating skills to a more service based operational model. Service Levels While IT may be familiar with contracts with outsourced IT, the difference with cloud computing contracts is that they will generally be on the vendors’ terms. A cloud computing provider may only have an architecture to support a specific level of service and no amount of large corporate huffing and puffing will change five nines to six. The way IT has to combine hybrid environments and effectively front various different agreements will become quite colourful, never mind the billing issues when the costs are passed on to the business. Best Prepared IT Ops are probably the best prepared for cloud computing because they are already familiar with working with more abstract views of technical infrastructure. IT Ops spend little or no time deep in the server room any more and juggle hundreds of servers from the comfort of their desks, similar to the way that they would manage cloud based services. However, IT Ops needs to be far more proactive, particularly in dealings with large vendors, to insist that they have the tooling and operating framework to continue to deliver to the business in a cloud computing environment. After all, just because the need for specialised local hardware doesn’t exist, it does not mean that business does not need the fuller range of services offered by enterprise IT. Disclaimer: This is not a complete list and IT Operations is where all the big cloud marketing money is Simon Munro @simonmunro The ‘Who Should Know About Cloud Computing’ Series This post is part of a series of posts for non technical roles, which you can follow from the links below
|
-
Large enterprise databases that form the backbone of business are not going to move out to a cloud computing platform any time soon, but more and more data will be processed in and originate the cloud and data professionals need to understand how to work with data in the cloud.
Non Transactional Data Processing
The delay between transactions being processed and business intelligence (in the form of analysis and data mining) being available is largely down to the ability to process data. The processing of hundreds of millions of rows into a cube is processor intensive and is generally left to an overnight job on a machine that is usually underpowered because it spends most of its time idle. While the big cloud computing vendors are not providing BI services yet, the prospect of having huge amounts of processing power available to analyse data has to be compelling. Cloud computing based BI can provide near real time data analysis for one business while it’s competitors wait until the next day or the end of the month.
Scattered Data
As much as DBAs insist that their large well run database is the master of all transactions and the single version of the truth, the reality is that data is becoming decentralised and is scattered across the network, home computers and the Internet. By design, application architects have been making use of read-only data stores and caches but over time the rigidity of the centralised database has forced application developers to store some data elsewhere ‘temporarily’. The centralised database has lost its way in a sea of data and the traditional DBAs will be paddling aimlessly around with it. Regardless of whether it has to do with cloud computing or not, the data professional needs to take more responsibility for the data, not by locking down the database, but by understanding how it is used and where it is stored.
NoSQL
For most data professionals the NoSQL movement is seen as an aggressive attack of their fiefdom, which, in a way, it is. The SQL model has become a bottleneck and does not support the dynamic nature of cloud computing architectures. Data professionals should, if not embrace then at least understand, the NoSQL movement and the patterns that it represents. Data professionals need to see that in some cases NoSQL is a good fit for a particular requirement (such as search) and need to work with the NoSQL practitioners so that the data is meaningful, accurate, secure and ultimately finds its way back to the centralised data store in neccessary.
Data Risk
Understanding the risk of data loss or compromise is a big part of cloud computing and the answers are complex because of different types of data at varying levels of granularity represent different risks. The enterprise data professional is more intimate with every table and row in the database than anyone else in the organization and needs to help the business understand what atomic pieces of data are for so that the risk can be managed. Over time this will affect how databases are designed – optimised for distribution and risk rather than performance or the most logical scheme – and data professionals are key to making this happen.
Biggest Change
While cloud computing will change application architectures and user interfaces, by far the biggest impact will be on databases. After all, much of the high performance storage consumed by enterprises is dedicated to data and economic benefits of cloud computing will be applied to optimising the cost of data storage, processing and operations – yet it is data that is the most exposed to risk.
While most of this series has been dedicated to less technical roles, the data professionals deserve a special mention because I see the disparity between interest in cloud computing and the role that they need to play greater than any other group of people who need to understand cloud computing.
Disclaimer: This is not a complete list and seasoned DBAs know to keep developers away from their stuff
Simon Munro
@simonmunro
The ‘Who Should Know About Cloud Computing’ Series
This post is part of a series of posts for non technical roles, which you can follow from the links below
|
-
At first you would think that in a world cloud computing applications would look and behave the same as any other web application – after all it is the financial model for computing that has changed, not what is served up, right? For many of the large websites that we use today that is generally true – cloud computing doesn’t change much. What does change though is that applications that would normally get developed or would remain obscure because it isn’t financially viable can now be pushed out for the entire world to see, use and interact with.
Microsites
One class of applications affected by cloud computing are microsites that are tactical and functionality because the cost of providing the infrastructure and support of extensive functionality is unwarranted. With cloud computing, providing richer functionality is a matter of development cost rather than hosting and long term bandwidth costs. The UX designer needs to think about extra functionality demands placed on applications that would normally not be considered – so we should expect registrations, sharing with friends, linking into Facebook, uploading and viewing of videos and so on.
High Load User Journeys
Due to the scalability of cloud computing, applications that would normally have a fairly small user base can now be extended to huge numbers of users. While we may have resolved the expected user journeys for more traditional high load applications, such as web retail and personal banking, the patterns have not settled for more familiar complex applications such as bookkeeping or sales automation, never mind the new ideas presenting themselves every day.
Unexpected Behaviour
Technical solutions to scalability problems such as eventual consistency, asynchronous operations, service degradation and apology based computing mean that the behaviour of applications will not be as expected and the UX specialist needs to carefully define the interactions to either hide effects or help the user understand why they are happening. The frustrations that many of us feel with shoddily built web based enterprise systems such as time recording and expenses (my colleagues are nodding an agreement right now) will not be tolerated by customers.
Complex Process Interaction
Cloud computing will lay the foundation for the ability to interact with more complex business processes and user experience specialists have their work cut out to ensure that, at least from the users’ perspective, things run smoothly. The existing patterns will need to be extended and the UX community is going to have to move towards a set of expected behaviours so that a vast number of users can easily find their way through a twenty step asynchronous process or change their own privacy settings.
Disclaimer: This is not a complete list and we know how bad developers are at building UIs
Simon Munro
@simonmunro
The ‘Who Should Know About Cloud Computing’ Series
This post is part of a series of posts for non technical roles, which you can follow from the links below
|
-
Testers, who are normally derided as the poor cousins in IT, have an opportunity in cloud computing to assert their value to the development and operation of solutions. Any cloud computing project that implements the standard non-cloud test strategy (that may be working well for traditional systems) is in for a big surprise as they approach the go live date and thing are falling to pieces.
Drastic Changes to Testing Process
Some traditional steps in the QA process need serious rethinking or throwing away altogether. A well architected cloud computing solution should have no need for load testing as we know it. If unlimited computing power is available on demand, there is no hardware configuration to do tests against. But the assumptions about how the solution handles load need to be tested and the accurate testing of how much load can be handled per compute instance. This changes testing to a role away from failure testing and more towards the establishment of metrics for non functional requirements and finding ways to test against those.
Back to Basics
Because cloud computing encourages developers to architect systems differently, many of the things that we took for granted no longer work the same way and testers need to be more rigorous in some areas than normal. For example, testing the result of a multi-node query across an eventually consistent data store needs to be done, whereas the equivalent query on an enterprise RDBMS probably wouldn’t be tested at all. It sounds scary, but developers jumping into cloud computing may make mistakes with simple things such as generating sequential and unique invoice numbers – when did you last test that properly?
Testing Sooner
Testers normally get their hands on servers as configured for production very late in the project, if at all. Because cloud computing has minimal procurement issues and uses pre-packaged commodity servers, testing on a ‘production’ platform can be done very early on in the projects – from the first day if there is something useful to test. This turns testing upside down, where non functional tests, such as load tests, security, latency and availability can be tested before the solution can be considered functionally mature. Smart project managers and QA specialists will use this to their advantage in an attempt to de-risk the project early in the development lifecycle.
Additional Criteria
Testers also need to think about new quality criteria that may be added by cloud computing, many of which are not considered by developers. Aspects such as the impact of latency, geographic distribution, synchronisation of data and intentional service degradation are aspects that we tend to gloss over when testing traditional systems.
Crucial Role
The role of testing and QA in cloud computing cannot be underemphasised. With so much risk and exposure to new technology, business processes and service provider models, the need to have someone on the team who knows what to look for and can give their stamp of approval is imperative to the success of a cloud computing project.
Disclaimer: This is not a complete list and after all, it may only work on my machine
Simon Munro
@simonmunro
The ‘Who Should Know About Cloud Computing’ Series
This post is part of a series of posts for non technical roles, which you can follow from the links below
|
-
As cloud computing projects get under way, the demand for cloud savvy business analysts is going to outstrip demand. Cloud computing introduces more than just technical changes as it forces changes to IT processes and alters the way that the business interacts with its customers.
Compliance
The most obvious areas are in compliance, risk and security. While cloud computing may be satisfactory for many regulations and internal guidelines, those assumptions need to be checked against the particular business scenario. The myths surrounding any framework influenced by regulations, such as the offsite or offshore storage of data, need to be unpicked and understood in detail by referencing back to the original regulations and laws that may be relevant. This is no simple task and requires a high level of credibility, commitment and specialist knowledge.
Legacy ERP Out of Favour
The disassembly of centralised monolithic enterprise systems into a loosely coupled conglomeration of on and off premise solutions changes many of the underlying business processes that currently exist to enforce the quirks and whims of a particular package. Cloud computing, in it’s guise as the next stage of the largely unsuccessful SOA (Service Oriented Architecture), offers the promise that cloud computing will deliver the computing services needed by business rather than being constrained and force fed the services offered by the incumbent ERP system. While technically we may be able to describe the loose coupling easily, the business analyst has a crucial role to play in understanding how existing processes need to change and how to ensure that processes key to the success of the business do not become victims of the process cull.
Alignment of Business and Cloud Models
The scalability and operational cost models of cloud computing allows business cases to be considered that would normally be considered too expensive or risky and now these opportunities can be snapped up by smaller departments and smaller businesses. The fundamental question that the business analyst needs to ask is “If cost, availability and lead times of IT were not constraints, where are the bottlenecks and how would you do things differently?”
Vision Alignment
No amount of technical brilliance will get cloud computing solutions adopted without business analysts who understand the vision, possibilities and pitfalls of cloud computing. Cloud computing is considered by most to be over hyped and fraught with danger and a cloud computing team needs people who understand the business intimately in order to help them buy into the clodu computing vision.
Disclaimer: This is not a complete list and there is a long traditional of miscommunication between developers and business analysts which needs to be upheld
Simon Munro
@simonmunro
The ‘Who Should Know About Cloud Computing’ Series
This post is part of a series of posts for non technical roles, which you can follow from the links below
|
-
At first glance there would seem to be little that creatives need to know about cloud computing – after all, it is just another hosting platform right? But there are some key things that creatives need to think about.
I don’t wear the right clothes to know enough about the creative processes or roles, so I will, for the sake of convenience lump branding, advertising, marketing and design into this group (apologies to anyone who takes offense at being lumped with this anyone else – sort it out amongst yourselves).
Understanding the implications of provisioning and scalability
If computing power can be provided with a lead time of a few minutes and a site can handle any load that you can possibly throw at it, how would that change the programmes and campaigns that you develop? What if the creative process (including product development, branding, design and launch) was the only cost and time bottleneck, with cheap computing power waiting to do your bidding? This would probably change the creative and product development processes.
Working closely with technical people to plan peak demand
In a traditional environment, a computing platform can only handle a certain load and when creatives dream up some clever idea the response from technical implementers is either ‘No’ or ‘Yes, in n months time’. With cloud computing it will be possible to plan activities that put enormous load on the system but it will only work if it was developed properly in the first place and if the technical people have some lead time (maybe a few days so that tests can be run). Done correctly and with technical bottlenecks removed by cloud computing, the relationship between creatives and delivery could (should?) be much closer and more interactive.
Functional Microsites
Outside of the existing transactional system, microsites built for specific campaigns can have rich functionality if built on cloud computing platforms because the functionality is funded, not by outright hardware purchases, but by the usage of compute power as needed and for the length and success of the campaign. Microsites could have logins that use the identity services from the transactional site, they could have underlying databases that provide rich storage of data for the user or even some transactions. Microsites will be able to plug into relatively cheap pay per use content delivery networks to handle the upload and download of rich media. (I personally think that microsites are a huge market for cloud computing vendors over the next year or two and will drive the wedge into risk averse business being comfortable with cloud computing)
Branding Effects of Failing Fast
The costs of cloud computing allow products to be hosted and operated with a low upfront cost and this will provide the development of products to ‘try out the market’ with the expectation that if they don’t work, that they cost was so low that it is no big deal. Having new products appearing and disappearing creates some interesting branding issues in order to maintain trust.
Fail Whale
Twitter’s notorious ‘Fail Whale’ became an Internet meme and somehow helped foster a sense of sympathy and increasing user loyalty during troubled times. The fail whale was a brilliant piece of design that allowed Twitter to be buffered from customer rejection as they grew. Cloud computing, particularly when used in high load environments, will be new and mistakes will be made. The customer responses to those mistakes will be far reaching and good design will influence those responses.
Creative Explosion
Cloud computing removes some of the technical hurdles to getting a product developed, launched and operational so the rate at which ideas are turned into business models becomes the limiting factor. Previously marginal business cases are now viable and products which may previously have missed the window of opportunity (such as the response to a competitor) will now be able to be delivered in time. Unchecked this explosion of ideas can result in monumental brand failures and confused customers but creatives that understand how cloud computing can bring their ideas to life will give an edge to their customers.
Disclaimer: This is not a complete list and creative people are advised to question creative advice from technical people
Simon Munro
@simonmunro
The ‘Who Should Know About Cloud Computing’ Series
This post is part of a series of posts for non technical roles, which you can follow from the links below
|
-
Whether software development or infrastructure, the manner in which cloud computing projects are delivered is different enough to require a slightly different approach to project management. So while the same elements exist on cloud computing projects as any other, the sequence that they need to be done in is a bit jumbled up, the focus is higher on some areas than usual and additional risks present themselves. The instant availability of computing resources solves some problems, such as the dependencies on procurement and provisioning of resources, but replaced with far more legal and contractual issues that need to be managed earlier on as some sticking points may get in the way.
Moving Activities Upstream
The on demand availability of resources presents interesting opportunities to move activities, which have traditionally been downstream activities, to be moved forward. Testing on a ‘production platform’, security, billing and operational handover, which should be moved forward, reflect the areas of focus on cloud computing projects that are either glossed over or taken for granted on traditional projects. Clearly security and regulatory concerns that would normally be wrapped up in the incumbent datacentres credentials will have to be dealt with in detail, both technically as well as the management of perceptions around them. The project manager will need to spend a lot of time allaying fears, proving the solution and generally providing assurance and answers where few exist.
Dealing with New Risks
With any new set of technologies and concepts new risks emerge. The biggest risks to cloud computing projects may be non technical and may result from a senior stakeholder reading a sensationalised story about data loss in the cloud and, fearing litigation or brand negativity, wants to move the new cloud computing initiative on premise. Other risks include problems that result from vendor platform and tool immaturity, availability of skills, increased engineering costs, inability to satisfy certain requirements (such as latency), compliance issues and a whole host of other problems waiting to ambush the unsuspecting project manager.
Cloud Computing Projects Will Be Different
While vendors will go to great lengths to tell us how familiar cloud computing is to their existing offerings, things are different and, in many cases, significantly different. While project managers are equipped with the general skills to manage a cloud computing project, most project managers would not have thought about cloud computing in sufficient detail to foresee what needs to be done on the project and will manage it as he or she would manage any other project until it all falls apart. There is a need and an opportunity in the cloud computing world for project managers who at least have a conceptual understanding of the issues.
Disclaimer: This is not a complete list and good project managers are wary of advice from developers
Simon Munro
@simonmunro
The ‘Who Should Know About Cloud Computing’ Series
This post is part of a series of posts for non technical roles, which you can follow from the links below
|
-
As much as cloud computing is getting media attention and is being hyped as the way that IT is going to be done in future, it seems that many people within the IT industry are ignoring it for now. While technologists, CIO’s and data centre professionals are giving cloud computing a look, other important roles within the IT chain seem to feel that it will not affect their jobs yet and are adopting a ‘wait and see’ attitude.
Most cloud computing commentators are trying to make it seem fairly simple and easy to implement (if you use their product of course). Cloud computing does indeed seem simple until you start thinking more about the possible effects and ramifications. These ramifications reach beyond technical curiosity and the simple economic benefits being sold to business – the way we tackle solutions, engage with consumers and manage risk and change are going to be impacted in a cloud computing world.
Ten years ago, who would have predicted the issues that Facebook is dealing with at the moment? Not just technical scalability, which seems largely resolved, but major issues around privacy, regulation, IP law, perception management and so on – all on a massively global scale for a service that essentially provides nothing other than user generated content, with no transactional functionality that businesses depend on anywhere in sight.
The economies of the future will rely heavily on the consumerisation of non physical products, whether that be high quality media or simple dots on your mobile screen showing exactly where your friends are, and the large multinationals will inevitably move into and dominate (as yet undefined) future markets. The provision of the services to enable these economies is going to be in high demand and will, in turn, demand to be paid a premium. Being in demand and paid a premium is something that many IT professionals may be interested in.
People that are actively thinking about cloud computing are security specialists, developers, network engineers and storage people. There is a worryingly high lack of thought from technical people (the lack of interest from database professionals is of particular concern) which, I imagine will correct over time as the technology becomes more compelling. But apart from the technical people there are a whole bunch of other people directly or loosely related to IT that are needed to make solutions work – and they’re not thinking enough about the impact of cloud computing.
As a technical person, I have mainly focussed on the technology, but part of my somewhat lonely mission to get people to think about cloud computing beyond the hype, I have given some thought to what other roles and disciplines need to do when trying to deal with cloud computing. So over the next few days I’ll post some basic thoughts on the following roles:
Contrary to my style, I’ll try and keep the posts as short as possible.
Simon Munro
@simonmunro
|
-
I have recently been speaking about the reasons why NoSQL just won’t die and a common thread is the cost of keeping data. The basic idea is that SQL databases are inherently expensive places to store data, with costs coming from specialised hardware, software licences, networking and a whole army of associated skills to keep it all running smoothly. It is also the desire to store and process every single little bit of data that inhibits scalability as the processes and infrastructure simply cannot cope or change quickly enough to handle growth or peak demands. Deciding what to do with all our data is not straightforward, so keeping it in a good ‘ol SQL database seems to make sense. There is little desire and few available skills to assess each piece of data as to where, how long, what format and what level of aggregation it should be stored in, so the knee jerk reaction is to store it at its most granular level in a SQL database with ACID, failover, backups and everything else that may be part of the standard solution. I think it is safe to say that most systems store far more data than is necessary. I would also assert that in high growth environments that the inability to effectively scale will inhibit growth. So the question to ask ambitious business units and product teams is this: Are you comfortable that your ability to grow your business is inhibited by the large amounts of data sitting around that you never use and will probably never need? I would think that given the choice, businesses would not want IT systems inhibiting their growth, and even less so if those systems are providing something that they don’t need or use. So the basic principle is pretty straightforward, but how do you go about changing behaviours? Go and analyse the lifecycle of every piece of data and try and determine how we should store it, in what format and for how long? I think that this may be a futile exercise similar in magnitude and success to the ‘Enterprise Data Model Project’. Not only don’t we really know how to go about doing it, but the answers to the questions will often be the same, as the cognitive dissonance will remain, and data will land up in the same place it has always been. Recently I was discussing with someone the success that colleagues of mine had building a retail web site that only touched the SQL database when an order was placed and all application state, such as the current basket, was stored elsewhere such as distributed in-memory cache. Things such as baskets would normally be considered useful data that should be in the SQL database for analysis reasons, after all, how can you determine conversion rates (crucial in ecommerce) without basket vs order information? The answer, in this case, partially lay with the use of Google analytics, where the project had some really smart and highly skilled people that know how to make it work. Google analytics performs a very specific service, storing and processing data that serves a particular purpose. Ten years ago we would have collected this data in our own website, stored it in our own database and spent a fortune trying to write reports and make sense of it. Now we simply put some JavaScript in the web pages, pay a nominal fee and let someone else provide the storage, processing, front end, training, support, and documentation for, in the case of web retail, a huge chunk of the data ‘needed’ by the system. Is Google Analytics NoSQL? In a sense that data previously destined for a SQL database is no longer being stored in SQL, yes but, in the sense of the NoSQL movement, it is not a piece of software that needs to be mastered and installed on a bunch of servers. Google Analytics is, from an architectural perspective, a data service – you give it some data and it does a whole lot of things with it, at the very lest allowing for storage and processing of the data. I think that over time we will see the emergence of more specialised data services for storage and, perhaps more important, processing of distinct chunks of data. We already have data services in cloud computing platforms – Azure Storage, Amazon Simple Storage and so on, but these are more generic storage services, without offering (much) processing, rather than specialised services that provide specific functionality to a particular shape of data. I can imagine data services where you would unload a whole list of your system requirements. The Holy Grail is authentication and authorisation, but there are others that may be more niche and less contentious. You could offload product catalogues onto specific services that handle searching for you. How about content management services that can take all the supporting infrastructure and front-ends away? After all, your content is usually in cache anyway so occasional refreshes off a distant server shouldn’t be that much of a problem. Even those examples are generic, how about something that is even more specific? I’m currently developing an appreciation (the hard way) for the complexities in the mapping and geo-aware data world and can picture a few data services in that domain. I can imagine a data service that stored routes, only. I think there are a few mobile applications that could make use of that – like cloud-based bookmarks, but with complex geographic structures. And if that became a standard that could be shared among applications, the richness of photos, video, searching, directions, bookings and meeting friends could change dramatically. Opportunities arise, functionality increases and customers are happier because we chose to farm out (apparently) crucial data to a service that is better at working with it. Obviously there is risk associated with giving your data to someone else. Aside the overall trust issues (what else does Google do with your data?), whenever you pitch the idea there will be a flurry of emails quoting auditing or regulatory reasons why it shouldn’t be done. However, as attitudes change (a lot of individuals are happy with all of their email sitting with Google), more robust SLAs emerge and cloud computing interoperability improves the reasons why not to let someone else store and process your data will begin to be overwhelmed by the benefits in doing so. In the next year or two, the market for such niche data services will largely be the domain of startups (where funding may be better spent elsewhere) and offerings that need a reduced time to market and the idea of plugging in a service that ‘just works’ is the only way to deliver on time. There definitely is an opportunity for providers of data services as early entrants can wrap up their little corner of the market immediately. More importantly, to the broader market, there are the opportunities for new solutions to take advantage of the services style in order to accelerate development and reduce overall cost. Provided, of course, that we are prepared to alter our views about data location, format, control, value and lifecycles. Simon Munro @simonmunro
|
-
On Friday I delivered a presentation at SQLBits which was an awesome event attended by about 450 people. There were some really good speakers presenting who know SQL inside out (and even written part of it), so I always feel a bit anxious about presenting about SQL Server, which I don’t use on a daily basis, with and to people who have made it their careers to know better than the next guy. I presented a session entitled ‘Improving Database Performance by Removing the Database’ (a long title I know) and, in my first slide, emphasising that the presentation was not about NoSQL (pointing people to my previous presentation for a NoSQL overview). Obviously ‘removing the database’ has a NoSQL smell I explained that t he presentation was about the reasons why NoSQL doesn’t die – and went through some of the issues that we have when marrying data exclusively with SQL. I was more about the (often unnecessary) costs and effort in keeping SQL databases running rather than the standard developer/scalability position of the NoSQL movement. The SQLBits organisers gave me a small room to present in (which I will read something into) and it was literally full to the rafters (the gallery was also full). One positive thing about being in a smaller room is that it was more intimate and I could get immediate feedback from the audience – and it was the nods in agreement that most interested me. From the demeanour of the audience and the people who spoke to me and emailed me after the session I came away feeling that I had succeeded in my desire to get SQL people to think a bit and take serious responsibility for the data in their control. It seems that I struck a nerve and people were beginning to understand that the stickiness of NoSQL is about the misuse of SQL by ‘database experts’ more than any particular killer alternative technology. I even had one person ask me afterwards if there is a movement that follows the principles that I spoke about. I don’t know of any and suggested we start one – the thing about movements is that they need a good name to start with and I haven’t come up with one yet, so that is as far as I have progressed and am stuck. For those that attended my presentation, the slide deck is here in pdf (as with so many images it was 10MB) as well as the slideshare link below. The very popular trump cards are here. Keep an eye out on the SQLBits website for the videos if you missed the presentation – I certainly want to have a look at it to check on who I offended (I recall estate agents, Oracle people, auditors and most DBAs) because I may need to watch my back. I think that I am going to stick to this theme for the next SQLBits in September (probably) and hope to see you there. Simon Munro @simonmunro
|
-
I’ve just about had my fill of Azure demos and presentations. After more than a year in beta it seemed that the only people that could stand up in front of a crowd and talk about Azure were those from Microsoft or their hymn-singing partners. It is not that the presentations and videos are bad, it is just that they are a lot of the same – either a ‘Hello cloud’ introduction to the platform, release of new features that have been asked for or pimping how similar Azure development is to existing .net development. Most of the presentations that I have seen, although many of them very good and done by really smart people, have the sheen of marketing snake oil. It all seems to perfect and simple - all clinical, clean and fashionable like CSI Miami where everything works, rather than messy and grungy like the development world that we have to live in every day. It is to be expected. Microsoft has a well-oiled marketing machine that must carefully introduce the cloud to it’s enterprise developers (making it seem familiar and easy) and a platform that has only been live for a couple of months can’t be expected to have many development teams that have real production experience of developing and delivering a solution that is more than a proof of concept. Last year UKAzureNet, the London based Azure user group, launched with considerable success and had well attended meetings but there was something missing. Since Azure was unreleased (and in many cases the attendees had never seen Azure before) the meetings took on a more introductory tone that was a thin slice of the capabilities and issues – so the tone became that of the presentations done at any number of Microsoft organized events like PDC. Even the conversations in the breaks were filled with conjecture (and hope even) about an as yet unfinished solution to an as yet unidentified problem for an as yet unidentified market (it was in the middle of the recession after all). What was missing from the meetings last year was some Azure rockstars – people who have delivered something in Azure in a real project that a customer paid valuable money for. We needed to hear from people who had found both the valuable nuggets and the warts and had done some of the initial toiling and fighting with the tools on our behalf. So it is fortunate (and about time) that the first Azure meeting of the year is going to have presentations – not by some big name to attract the usual sheeple that marketing attracts, but by some people that have worked with Azure and delivered something that is real. I have had some chats with Simon Evans, James Broome and Grace Mollison over the last couple of months as they have developed a solution on top of Azure – a solution that would have no business case if it weren’t for the cloud model. I have heard from James as he wrestled the development fabric to fit in with his BDD style and the support of Grace in providing a build server for the team using a product that doesn’t have a server version. I watched from a distance as Simon got his head down and dealt with the persistence issues and had to ignore him after the n th time that he exclaimed how cool and easy the CDN is. Next week is a busy week for the Microsoft community. There is all the stuff that Microsoft is putting on around the launch of VS2010, SQL R2 and others. There is a Silverlight user group meeting and I will be presenting at SQLBits on Friday. Thursday night is the turn of UKAzureNet and even though it might be less central, it is being done in a cinema and our new emerging rockstars will be on a stage of sorts. We need your support and attendance to help get it as full as possible – we don’t expect it to be like the opening weekend of Avatar, but hope that we’ll have more than a handful of usual suspects throwing popcorn. You are guaranteed to hear some interesting stories from the trenches. You can register here UK AzureNET User Group: Phoenix from the Flames @simonmunro
|
-
The video of the presentation that I did at SQLBits Goes West is online. You can view it from the session details page : ‘SQL, NoSQL, SomeSQL - A look at non-relational databases’. I listened to the presentation this morning to check where I need to improve my delivery (answer: a lot of places) and to check if it is indeed a good introduction to NoSQL for the ProSQL (as I remembered it). It seems that I remembered it quite accurately and it has, IMHO, turned out well (also the technical quality of the video and audio is of a high standard). I made this presentation at a SQL conference and was aware of the risks of lynching – particularly since it was in Wales, which has a somewhat antagonistic history with England. I struggled to come up with something that made the point about the pros and cons of NoSQL to an audience that was overtly biased to SQL, but managed to deliver something that allowed me to survive unscathed and I also received positive feedback. If you are are familiar with SQL and want to get an idea what this NoSQL thing is about without trawling the interwebs for hours, I suggest you have a look. It is, after all, free as with all of the other good sessions at SQLBits that will continue to be free. Simon @simonmunro
|
-
As SQL Experts and DBAs start looking at what is new in SQL 2008 R2, they inevitably try and figure out where StreamInsight comes from and how it fits into their existing view of the database world. The short answer is that the confusion is unnecessary as it does not fit in the world of most SQL DBAs. Allow me to briefly describe some of my observations – and remember, my standard disclaimer applies - I don’t have any MVPness or access to information that is not publicly accessible and my views are not endorsed (or maybe not even shared) by anybody. - Microsoft Organizational and Marketing Structures
It would seem that members of the StreamInsight team have some background on the SQL Server team and hang around in the same building/group/division. How Microsoft works and how organizational structures influence products, packaging and marketing is unclear but it does influence the positioning of their outputs (just look at Entity Framework vs Linq to SQL). The organizational relationship, the marketing relationship and the technical relationship between the broad SQL product and StreamInsight should not be confused. Regardless of the organizational alignment, technically StreamInsight is quite distant from the rest of SQL. - Relational Algebra != SQL
Because the clever people on the StreamInsight team worked with the database research team to apply relational algebra with temporal semantics in the StreamInsight product, it does not mean that the relational part is the ‘R’ in RDBMS/SQL Server. The relational model, relational algebra and the SQL implementation of the relational model need to be discussed in very different contexts. SQL databases are so broadly useful that a SQL expert will naturally question where the latest SQL feature broadly applies to their problem space. StreamInsight is a product that has a very narrow application that falls outside the problem space that most SQL people are used to. Like SQL CLR and SQL Service Broker, which solve less common data oriented issues and people struggle to position, StreamInsight solves problems that are much, much more narrow and further away from the generic solutions that SQL provides. SQL implementations are generally architected for a write once/read many scenario, resulting in the focus on storage of data and the concerns about disk IO, indexes and other things. StreamInsight does not really care about data being stored, it just reads the data (once) and forgets about it. This puts it outside the mental models that most DBAs have developed in terms of handling data. Just because you may want to store data once StreamInsight is finished with it, it doesn’t mean that a preoccupation with data storage should affect how and where StreamInsight is used. The clue is in the name – StreamInsight allows for queries (insight) to be run against a stream of data. StreamInsight doesn’t run in SQL Server and is accessed using .NET libraries with some LINQ magic. Most DBAs gloss over anything .NET related (such as SQL CLR) and assume that it doesn’t belong in the database world. StreamInsight development is not going to be done by the database developers and T-SQL rockstars – you are going to need .NET development skills. - This Kind of Solution is not Cheap
There are a lot of ways to process data and, when it comes to small amounts of data or where latency is not a problem, plain ‘ol SQL will be good enough. StreamInsight only comes in the Datacentre and Parallel Data Warehouse editions of SQL server – costing in the region of US$57,000. You have to have a real problem to solve to spend that sort of money per node. Update: It seems that since this I created this post, Microsoft has changed the licensing on StreamInsight as pointed out by the comments below and detailed by Simon Sabin in his post ‘StreamInsight is in all editions (except express)’ StreamInsight, while not applicable to a broader market, fills a very important gap for Microsoft in enterprise engagements. The ability to offer solutions for complex event processing that are backed by Microsoft and ride on the success of SQL Server is a big deal and being part of SQL Server, instead of some side project, makes marketing sense. - Data Professionals, Not DBAs
While the bulk of MS SQL instances are used for the database engine, there is a lot more than just SQL in the box. Analysis Services is SQL only in name and tools for BI, such as PowePivot step away from how we generally work with data. The concept of storing data in tables and viewing it with reports will be with us for a while, but there is more to data than just SQL. StreamInsight is part of a family of tools that deal with data that data professionals need to come to terms with – leaving database administration and backups to the DBA. If you work with data every day and think you need to be more of a data professional than a DBA, I suggest you have a look at StreamInsight (even if it is just the documentation) and form your own opinion on where it fits in to the SQL world and, more importantly, your own environment. Simon @simonmunro
|
-
There has been a lot of discussion about whether or not Amazon EC2 is oversubscribed and as Reuven Cohen points out, oversubscription is used across a variety of services and industries. The most widely accepted and understood oversubscription model is that of airlines, where it is common practice to overbook flights by as much as 50%. The reason why airlines overbook is because it makes sense to have the occasional irate passenger and extra costs than flying with empty seats and the amount of overbooking for a particular flight uses specialised software and algorithms to work out the optimal overbooking. From weakonomics: … some of the inputs into these things would be the number of people flying for business and pleasure, where they’re going, the current weather that day, the time of the flight (surprise! People oversleep and miss flights!), even the odds of the plane breaking down When too many people turn up for a flight, some passengers get bumped and some are offered a financial incentive (vouchers) to get off the flight and wait for the next one and the people that are generally not bumped off the flight are those with the most expensive tickets (business travellers). You would be surprised at the number of IATA ticket classes available for airline travel, where economy/coach has 13 different classes – meaning that there are potentially 13 different prices for the same cheap seat! It is the business travellers (W class) that will almost always get on the flight – they understand that they have paid a premium to just pitch up at an airport and get on the next departing flight, leaving the cheap ticket tourists to sit for a few hours or days in the airport. In the EC2 post that started much of the discussion, Allan Williamson tells the story of how they initially used ‘small instances’ and had to move to higher instances because, he believes, they are not oversubscribed and offer better overall performance that is not related to the size of the instance. The core observation is that he feels cheated because ‘Amazon is forcing us to go to a higher priced instance just because they can't seem to cope with the volume of Small instances’. Chuck Hollis states that the problem is not over subscription, but a lack of control and suggest that the solution is to make more use of private clouds, where enterprise IT has more control of resources. I’m not big on this idea because in my enterprise experience, the allocation of scarce IT resources is done by politics (where the most politically adept business owner gets first refusal) or bureaucracy (where internal processes and interdepartmental billings mean that the department with the most budget gets what they want). This is hardly the kind of control that supports the zero touch model that is promised by cloud computing. What cloud computing needs, is a mechanism similar to the airlines where a premium is paid for a guaranteed seat. The problem with implementing this in cloud computing is that although the premium could be distilled into financial terms, the seat becomes a lot more complex. You could arguably pay a premium for higher spec instances, lower latency, better disk i/o, better network i/o, guaranteed completion time for a long running process, total bytes in or out and so on – all assuming that it is understandable and supported by the architecture. The first problem is trying to get the market (even if it is consumers of your private cloud) to understand the complicated pricing model. Most people, when confronted with too many choices will um and ah until selecting the one that is probably the least suited to their real needs. If competitors are offering three different packages, small, medium and large, your highly sophistication multi-factor and dynamic prioritisation model may be too complicated and too risky (because it cannot be budgeted for) – driving customers away. There is also the problem as to whether or not cloud computing platforms are able to measure these various performance points and make adjustments and decisions accordingly. The first culprit is the virtual machine which, as an abstraction, leaks through to all levels of the application – you may have some part of your process that is a lower priority but, because it is running on the same VM instance as everything else, cannot be prioritised differently. Do cloud computing platforms have a way to measure the applications (or even VM’s) that hog resources such as network and disk i/o and, more importantly, do something about it by balancing them out with less resource intensive applications? Finally, there is the question as to whether or not an application is architecturally capable of being bumped off the flight or onto business class. I have no doubt that your average legacy ERP application would not take kindly to being shuffled mid-process from one server to the next as an when some fabric controller deems it a really good idea. Most applications are not being developed with the idea of being started up, shut down and moved around, never mind having the necessary monitoring points that help the fabric controller to its job. I foresee that cloud computing platforms and their consumers will become more sophisticated, allowing the over subscription model to be an asset that allows us to use available resources optimally. Imagine being able to build an application where you can say ‘this step, from here to here, should take 0.5 seconds – if not, escalate to processing plan C1 if it costs less than $0.02/min, otherwise leave as is for 4 hours and reassess’? We are still a long way from there, but it is coming. I wonder who is building it? Simon @simonmunro
|
-
Cloud computing is obviously green and good for the environment. For example, - Computing power located near green resources – hydroelectric power or other renewable resources like they have in Iceland.
- Elasticity - resources don’t have to sit around consuming power when underutilised and can be (virtually) spun up when required.
- Cloud Computing makes use of commodity resources, reducing wastage and helping ensure that hardware components are used as long as possible.
So when vendors pitch cloud computing today, it seems a good idea to ride the one thing that may be hyped more than cloud computing – climate change, green energy and related topics. An environmentally responsible organization may want to do as much as they can to be more energy efficient (or at least make use of renewable energy resources) and cloud computing looks, on the surface, to be a good candidate. In assessing the viability of cloud computing, running the numbers about how green it actually is for a particular usage scenario becomes a little bit more complicated than ticking off the green credentials. That is because cloud computing encourages behaviour that may not be very green. Like a Prius driver that commutes 2 miles a day instead of walking – just because the Prius is green technology, how it is used and applied is more important. I thought I would make up some potential non green side effects of cloud computing… - Availability of cheap resources encourages poor optimisation – it is easier and cheaper to throw another cheap server at a process than to code and debug a highly optimised solution.
- Processing during peak times – because processors are available on demand, jobs that may have run at night can now be run any time, meaning that the energy required for processing is the most expensive and environmentally unfriendly.
- Over processing – Cloud Computing allows things that may never have been processed before to be processed without an impact on performance, for example, selecting a very large set of data for analysis because you can literally process the data in an hour where previously it could have taken days.
- Providing low value products and services – If cloud computing lowers the cost to provide services, it is possible to provide services that only generate a few pennies per transaction. While generally considered a benefit of the cloud, one has to question whether the value of the end product is worth its environmental cost.
- Insisting on low latency – A big part of the greenness of cloud computing is the availability of resources where energy is cheaper (and more sustainable), but an overzealous demand for low latency may mean that large data centres still need to be located in metropolitan areas where the environmental impact is high.
As we move from the ‘why’ in cloud computing to the ‘how’, claims about the green credentials of cloud computing need to to be clearly answered, motivated and calculated in order to substantiate the claims for a particular proposal. Reuven Cohen from Enomaly has recently started asking these sorts of questions and makes a call for supporting data of the environmental friendliness of cloud computing – and seemingly finding very little. There is little doubt that some organizations can pull off very efficient (cloud) computing but I imagine that it is tough. The efficient use of computing resources goes beyond the physical data centre and extends to the application architectures and usage scenarios around a potentially limitless and cheap supply of computing resources. My interest is to try and understand the influence of environmentally friendly approaches to application architectures and whether or not it is relevant. We could, for example, look at our application architecture and structure it to offload some processing (where latency is acceptable) to the most environmentally efficient data centre at that moment in time – say on the other side of the world where the data centre is making use of off-peak overcapacity. Of course, in the spirit of cloud computing, this should be done transparently and in a zero touch manner. As Google found out a year ago with the furore that erupted over the estimated carbon footprint of a Google search, people are concerned about their energy consumption (or at least aware). Perhaps, with ever increasing public awareness of climate change, those same concerns will be directed at organizations, where responsible use of energy will become expected. It is possible that a few years down the line that regulators will jump in and require audits and aggressive increase in data centre efficiencies. With the current initiatives in building out new cloud computing oriented platforms, it would be prudent to spare a thought for the environment – if not in the implementation of energy efficient processing, then at least in the ability to measure whether or not we are successful. Then can we use cloud computing as a platform to bring about the radical change in energy usage that is required to protect future generations from our insatiable appetite for computing resources. Simon Munro @simonmunro
|
|
|
|