Welcome to EMC Consulting Blogs

As a business analyst I often feel that one of my responsibilities is to act as an interface between the business and development team. This sometimes feels like speaking one language with the business and translating that to a language a developer will understand (and vice versa). Recently on a project, the development team suggested that we look at implementing Behaviour Driven Development (BDD). After experimenting with this, westrongly believe this is a powerful way of exploring and clarify business requirements and logic with the business and developers as it uses a common language that uses business terms but is inherently useful for coding.

At its core BDD is based on the following Practices:

1.    Involve stakeholders throughout the process

2.    Establish the goals of stakeholders

3.    Gather features to meet goals of the stakeholders

4.    Use examples to describe behaviour

5.    Automate the examples

In exploring this we have found two techniques for facilitating this process that employs Practice 1 throughout. For Practices 2 and 3, a user story can be defined with the business that expresses the stakeholder involved, their goaland the required feature...

As a Consultant (Stakeholder)

I want to capture when a Candidate has beenidentified as the person the client would like to employ (Feature)

So That the fact that a Candidate is under offer isclearly visible within an assignment so I can tell at a glance the status of my assignment (Goal)

Then in support of Practices 3 and 4 a number of “critical” scenarios can be captured in a structured language that identifies what needs to be done inorder for the user story to be accepted by the business...

Scenario 1

Given a person on an assignment who is in the role of Candidate, ex-Candidate or Successful-candidate. (Context)

When a user enters the under offer date and comment, (Event)

Then the under offer date and comment should be recorded against that person on the assignment (Outcome)

Scenario 2

Given a person on an assignment who is not in the role of Candidate, ex-candidate or Successful Candidate.

When a user enters the under offer date and comment

Then the under offer date and comment should not be recorded against that person on the assignment.

The beauty of this is that the developer can use these scenarios to implement the automated scripts that guide and validate the code they write. This is really easy for them as the coding language the automated scripts are written in are based on the same structured language.

BDD greatly enhances the business/development conversations a common and natural language is used that is inherently useful to developers but is framed in business terms. This reduces confusion which saves development time and results in better functionality being delivered which is more likely to meet the business need, work to the right level of quality and is more likely to be implemented in a maintainable and readable way. Or more simply put a higher return on investment. 

As the title of Kent Beck’s book “Extreme Programming Explained: Embrace Change” highlights, change is at the heart in Agile software development.   Agile techniques like XP and SCRUM focus on the people and how those people interact when they develop software rather than pre-scribing how this interaction will occur.  This is done through continually allowing the team to review and improve the way it works to deliver software better and faster.  In SCRUM the primary tool for instigating this change is the retrospective.  I have taken part in many retrospectives both as a facilitator and participant and have found varying degrees of success.  Issues include

·         Denial by the majority that a problem exists

·         Resistance to change as it is seen to ‘get in the way’ of delivering software

·         Lack of commitment to making the agreed changes happen

·         When changes have been implemented, the team slip-back into their old behaviours.

To address these issues I want borrow a tool from psychology called the Cycle of Change.  It is used primarily in helping people with addictions but can apply to any change process.  The cycle is illustrated below:

  It proposes two key things

·         there are several stages a person must go through before they successfully action and maintain lasting change (a stage cannot be missed).

·         that change is cyclic, it can involve several tries before lasting change is implemented.

It is important to note that the techniques to move people from one stage to another are different depending on the current stage they are in.  For example, offering solutions when a person in Pre-contemplation will not help whereas if they are in Determination this could be very productive.  It is therefore very important to identify what stage a person is in when they are confronted with change.

 Applying this to a team I have put together the following table that could help identify what stage the team is in and some approaches to dealing with the team to help them move to the next stage. 

I have found that retrospectives are very good at moving issues that are in Determination stage to the Action stage, but do not cope so well with moving teams through the other stages in the cycle.  This tends to driven by the team deciding what issues they wish to deal with at the end of each sprint.  These weaknesses could be addressed by ensuring that important issues are not forgotten about by highlighting them during subsequent retrospectives.  Suggestion include

·         Provide analysis to the team that demonstrates the need to change regarding issues where the team are in the Pre-contemplation or Contemplation stage

·         Summarise performance against metrics to highlight potential relapse and reaffirm success for at least six months after the change has been implemented.

However try to avoid forcing a team to make a change.  Change will not be effective if the team have not taken ownership for the change.

General Eisenhower said that "The plan is useless; it's the planning that's important." How I understand this quote is that the General knew that even with all the planning he participated in, some people would end up on the wrong beaches, there would be a loss of some critical supplies, and many soldiers would face life-threatening obstacles that were never imagined in the planning process. However, he knew he had created an ongoing planning process that was understood by his people that would allow them to recognize and address the most seemingly insurmountable problems. I think that if the General could successfully pull off this planning process to win a war, surely we can do this in our everyday jobs. 

I am writing this blog to explain my thoughts on why I think the Agile project planning process (Release plans) are more effective than Waterfall projects plans.  Here I make reference to PRINCE2 as a waterfall methodology as this is a process I am familiar with.  I note here that PRINCE2 is not necessarily a Waterfall methodology, rather a process for managing and planning any project, but it is often used to deliver projects with the Waterfall approach of design up front in mind.

I also wish to address three specific issues raised in the following article.

http://www.claretyconsulting.com/it/comments/agile-scrum-fails-to-get-to-grips-with-human-psychology/2006-08-17/

Namely, Agile

1. [is not based on] Commercial production decisions are based on rational expectations.
2. [that] Resource Management had vanished.
3. Each of these [Agile] organisations had differing development approaches [within the same IT department] and tools from project to project.

Anyone not familiar with Agile planning and is interested can get an excellent introduction to this topic by reading Agile Estimating and Planning by Mike Cohn.  Here I will stick to the Scrum estimating and planning process described in Agile Project Management with Scrum by Ken Schwaber (except for the use of story points and velocity which I think greatly enhance the Scrum planning process).

As with any project a Scrum project cannot start without a business case, funding and an aspirational set of requirements for the project.  In Scrum this set of requirements is called the Product Backlog which is an estimated and prioritised list of requirements that are written as goals (the what, not the how) and preferably with defined acceptance criteria.  These Product Backlog items are then grouped into four week iterations (or less) based on business value (high priority, low cost) and architectural importance that could be completed by a certain team size (which includes analysis, development, documentation and testing).  This grouping into iterations gives what is called the Initial Release Plan as it is the first estimate of what could be delivered by when.  As emphasised by Mary Poppendieck and Tom Poppendieck in Lean Software Development in the chapter on Waste, the art of gaining maximum benefit from software resources is to release it as early as possible and do it as often as possible.  Until software is being used it just costs a lot with no benefit, so I try to champion this view when forming the Initial Release plan. 

As part of this planning exercise the types and numbers of resource needs to be determined.  To do this and to support the business case a basic understanding of the technologies to be used also needs to be determined.  This will not only be driven by the business requirements, but also the IT department’s strategic requirements around, including tool selection and architectural guidelines. 

A waterfall project will also go through a similar planning exercise. In the PRINCE2 methodology this is done in two phases, Start-up Project and Initial Planning where the business case is defined or verified, a quality plan is produced and a high level plan that is broken down into a number of stages.  Again this will involve resource planning and certain decisions about the use of technology.

So far, these sound reasonably similar, but they are some key differences that should not be overlooked.

• The Scrum Release plan is broken down into fixed length iterations where a PRINCE2 Stage can be any length
• In Scrum each sprint targets producing something that is production quality, i.e. it could be picked up and implemented in the real world “tomorrow”.  PRINCE2 stages do not have stipulation and could produce artifacts that, on their own, hold no value in their completed state.
• The technical design in a Scrum project may be a lot more high level than that of a Waterfall project.  As suggested in Lean Software Development, Agile projects try to make decisions as late as possible.  This could also be the case in PRINCE2 but in many Waterfall projects the technology is often decided to quite a detailed level up front (as getting it right up front is the basis for Waterfall).  A SCRUM project may start with a choice of technologies to be used, that could all satisfy the business and technology requirements.

I think the key here is that both Waterfall and SCRUM project plans, if done properly, allow commercial production decisions to be based on rational expectations.  I hope this is enough to convince you that condition 1 is met by both methodologies at least at the start of the project.

Similarly, both involve an initial resource plan so item 2 is also addressed (at least at the start of the project).

I also think item 3 has been addressed in full, regarding organisations having differing development approaches within the same IT department and tools from project to project.  I would suggest that either there was a lack of strategic direction within the IT departments or it was felt within those departments that no form of strategic direction is required in an Agile organisation.  Personally I think this is a mistake, and a balance must always be struck between the business and technology requirements.  Ken Schwaber makes specific reference to this in Project Management with Scrum when discussing multiple Scrum projects. The first sprints are concerned with getting the technology and operational structure right for multi-team development whilst still demonstrating some business value.  In Lean Software Development in the diagram of the Agile value chain also describes the need for this initial architecture. 

The heart of a Scrum project is the Sprint.  A sprint starts with Sprint planning where the Product Owner (the person responsible for delivering business value) collaborates with the team (those tasked with delivering the Sprint) to agree which Product Backlog Items can be delivered in the Sprint.  (Note that this may be different from the Initial Release Plan due to further understanding of the requirements.)   Having agreed this, the Team identifies and estimates the tasks that are required to produce an increment of developed and tested code in the Sprint.  This list of tasks is called the Sprint Backlog.  The Sprint then starts and the team assesses the Sprint Backlog on a daily basis including revising time-to-go or creating unforeseen tasks or deleting redundant tasks.  This is communicated in the Sprint burndown chart which plots the time to go against the number of days in the Sprint.  This gives a clear indication to the team of how the team is progressing against the plan.  At the end of a Sprint the team demonstrates their increment of functionality to the major stakeholders and feedback is given. This may identify additional requirements or items where functionality that did not meet expectations or may result in the realisation that certain requirements have become redundant. The Product backlog is updated according to what was delivered which gives a new estimate to go.  It also gives an idea of the velocity the team is working at.  This is either done on a Product Burndown Chart which plots Sprints against effort remaining, and/or by calculating the velocity which I calculate as

Velocity (story points per unit man day/hour) = Story points in Sprint DIVIDED BY man hours/days dedicated to Sprint

Note, that for these purposes, the story point can be considered as the original man-time estimate made when the Product Backlog was created.  This information can be used to re-assess initial Release Plan to gain an improved estimate of what can be delivered when.  It will prompt questions such as

• can we go-live early, we better get the production hardware ready a month early?
• Oh my god we are never going to get this done in four months, maybe we should extend the go-live date now or reduce the scope of the project so that the majority of the business case is delivered on time?
• Should we change the resource profile?

The point is that the Release Plan is constantly evolving with each Sprint.  This is based on, what I believe to be the best metric in software development and that is amount of working, acceptable code delivered.  This again is a principle that supports the Agile Manifesto: Working software is the primary measure of progress.

In a waterfall project using PRINCE2 a stage plan will be drawn up during the previous phase.  Although this may be done in conjunction with the team, the project manager owns the plan.  The project manager then tracks the project against this plan getting regular updates (which can be daily) of time spent and time to go.  The project manager will try to deliver the stage within certain constraints called tolerances and if he/she fails to do so an exception plan will be created to reflect the new timeframe or cost and resource profile.  The exception plan can be for the Stage and where necessary the overall Project Plan. At the end of a phase the stage boundaries process is implemented to review overall project progress and the plan.

Again similarities can be drawn between Stages and Sprints but there are some clear differences that need to be understood:

• Sprints are of defined length (apart from in extremely exception circumstances where a sprint could be stopped) where as stages can vary in length due to an exception plan
• Sprints deliver an increment of shippable functionality whereas Stages may not.  A Stage may produce a functional requirements document or a technical design. 
• The project manager owns the Stage plan in PRINCE2 whereas the team own the Sprint Backlog

Again, if implemented properly both Scrum and PRINCE2 satisfy both item 1 (Commercial production decisions are based on rational expectations) and item 2 (Resource planning) throughout the project lifecycle.  They are both planning processes adapting to plans.

So why do I think Scrum release plans are better that Waterfall project plans?  This comes down to following factors:

• Waterfall encourages getting the plan right up front.  If a waterfall project is not going to plan, people being human, will sometimes attempt to keep things to the original plan if they feel their reputation is at stake.  This will lead to a “head in the sand” approach to delivery, where delivering to the original plan must be achieved at all costs, resulting in a lot of issues being pushed to the very last phase in the project often resulting in an unexpected delay in go-live right at the end of the project.  I think when this occurs Commercial decisions are being based on anything but rational expectations.  Conversely, if your stakeholders are groomed properly they have different expectations set and are more willing to accept hiccups and wins as they are seeing working code every four weeks in a Scrum project. I am not saying the “head in the sand” approach does not occur on Agile projects, I would say I have run two of these where through contract or stakeholder belligerence the scope is fixed as well as the timescale resulting in the very same issues as found on waterfall projects when the “head in the sand strategy” is adopted.
• In Scrum, the team own their own plan for the iteration whereas a project manager owns the plan in most (if not all) waterfall projects.  The distinction may seem arbitrary, but I am strongly in favour of the team owning the plan, even if I can feel a little in the dark sometimes about what is going on.  When I have managed both Scrum and non Scrum projects I have found that team members can give very poor estimates to go just because either they do not want to have to think about it or are scared to give bad news, often resulting in late notification of delays.  By placing ownership and responsibility for the Sprint Backlog within the team, the team tend become better time managers and are less worried about giving bad news.  I think the reasons for this is are that the team learns the value of monitoring their time to-go when they need to deliver something in four weeks and are more honest to the team where previously they would have to explain themselves to a perceived superior.  (I must confess, however, that getting the team to use the Sprint Backlog in the first place is a difficult exercise, as this is an additional responsibility that would have traditionally sat with the project manager and can be mistaken for an administration overhead by team members.)
• The Scrum release plan process targets the highest business value early, whereas Waterfall projects often have a fixed scope (a defined product in PRINCE2) so that this business focus can be lost.  Imagine a project that is going to take twice as long as originally expected.  The Agile project allows for high value code to be released on time with a second release later or with abandonment of lower priority requirements.  A Waterfall project plan does not have this flexibility and could be abandoned due to overspend without any business value being derived at all.
• The Scrum release plan process is based on a more effective metric – working code.  All elements of the delivery are being measured from the first sprint.  The analysis, development, test and documentation is all performed in a sprint.  If your going to get a surprise in Agile regarding estimates it is usually going to surface in the first two or three sprints usually well in advance of the release date.  Waterfall metrics tend to use measures derived from other projects and are therefore not attuned to the nuances of the specific project.  If functional design overruns then development is likely to overrun by the same % factor.  This may not always be the case.
• Reduction in number of non-critical artifacts.  Although PRINCE2 focuses on products and not tasks, Waterfall does imply a design up front process.  This means that documentation is created to inform future phases, but may have little use when the project is completed.  Agile favours working software over comprehensive documentation.  This does not mean that Agile projects do not produce documentation, but the documentation is minimised.  This means, from my experience anyway, that when coding from day 1 is the focus, this leads to more functionality for less money.
• Agile planning brings together the business and technology arms of a project in the same room to conduct its planning and to review the outputs regularly.  This gives both parties a feeling of control over the outcome of the project because they can debate what is wanted versus want can be delivered on a regular basis.  Waterfall projects can tend to show the business a plan up front and a set of requirements that the business is told it must (try to) stick to and show there commitment to this by signing the requirements off as a complete statement of their requirement.  In most cases divergence from the plan will occur, either through the business realising that what they asked for was not actually what they needed, or the project is just harder or easier to deliver than was expected so the project undergoes an often laborious change control process to the “contract” that was made at the beginning.  As the change control process is written in paper I find a lot of resentment builds up between parties as neither side feels they are being properly understood.  I find that face to face direct communication tends to resolve these issues but only if people are brave enough to assert themselves.