Recently I was sent an email asking me for some advice on using TFS with the scrum for team systems plug-in for managing testing tasks and defect tracking on an agile project. I thought the answers might be useful to others, so I have posted them here:
- Basically I want to use VSTS / TFS (with the Conchango template) to manage functional testing as well as unit testing. Our project test team will own the functional testing of the application releases both in the scope of a Sprint and the scope of System Test and User Acceptance Testing.
Generally speaking around managing specific testing, as I'm sure you already have set up, testing tasks can be entered as sprint backlog items assigned to the appropriate tester. Test cases/scripts (depending on your terminology I refer to the sql queries to be executed) can be stored in source control so a record is kept of them and version history can be tracked. Regarding unit testing, it has always been my understandin in agile that this is the developers responsibility as part of ensuring their code is ready to go to test. They can however store unit tests in source control and if desired I would suggest adding a field to the release note on the wiki to confim that developers have successfully carried out unit testing.
- How do you typically use the status types available: Decomposed, Deferred, Deleted, In Progress, Not Done? I’m trying to relate these values to what I’m used to: New, Re-Open, Assigned, Ready for Re-Test, Fixed, or Closed.
You're right in that the status types are simplistic, but you can relate them to the values you are used to:
- When a defect/bug is created it has a status of 'Not done' and it remains like this until work is started on it, at which time the status should be changed to 'in progress' - I suppose this is the equivalent of 'new', although this is not used because otherwise we would have to define how long is a bug new for and when does it get updated to not being new, who's job would that be etc.
- Re-open does not match up, however I've never encountered a scenario where I would need this. Generally if a defect is closed off and then reoccurs a new defect would be raised, and you could link to the previous one if you wanted to maintain a record that this has happened before. If it turns out the defect was never fixed properly, you can retrieve it from the done list and change the status back. The history tab would keep a record that it had changed status back and you could leave a comment if desired.
- A defect is always assigned to someone, therefore this is not a status of the defect. It is either assigned to someone 'not done' or assigned to them and 'in progress'
- Ready for re-test also is defined by the owner of the item. When a developer finishes fixing a defect, the status remains 'in progress' and they assign it to an appropriate tester. Any defect therefore assigned to a tester with a status of 'in progress' can be assumed to be ready for test. If a tester has a defect with a status of 'not done', it is likely a defect which has been raised but not assigned to a developer yet. This is best practice, but of course sometimes people forget to update this status field, and comments can be added to the history tab to help keep a record.
- When a defect is fixed and retesting has confirmed this, the status can be changed to 'done', at which point it is also closed. Fixed therefore also equates to closed. An item may also be closed if it is deleted (perhaps the defect has already been raised, or turns out not to have been a defect, etc). I also like to add a comment to the histroy tab to state I have retested the defect, in case the status is changed accidentally.
- An item may be deferred although this shouldn't happen regularly. Situations where it might be necessary include when an area of functionality is parked therefore defects raised against it also need to be parked, so the status is set to deferred so that when development resumes these issues are not forgotten about. Alternatively, a defect may not be resolvable at the current time due to circumstances outside of the team's control.
- Decompossed I must confess is new to me as I don't have it available on the version of TFS I'm using.
- I usually collect details such as how to reproduce the bug, comments / responses from developers, etc. In TFS, is there a difference between “description” (owned by testers) and “comments” (shared between testers / developers)?
Regarding capturing details, the description field is designed to hold much more detailed information than the comments field and is intended to give the developer enough information (perhaps combined with some screen shots attached) to fix the defect. This field is not necessarily owned by the tester, since anyone can raise a defect and therefore enter a description in this field. Furthermore, I may sometimes not know enough detail to fully populate this field, and the defect may be temporarily assigned to a BA to make a call on something. They will then update the description before passing this on to the appropriate developer. Alternatively, it may be assigned to a devloper for investigation, who may then update this field and pass it on to another developer for resolution. The history tab is used for much smaller comments, such as 'update the config file', 'ready for testing', and 'retested and passed in test', etc.
- Capturing the resolution in a defined field is very helpful for researching similar defects. Does TFS have the ability to do that?
Capturing the resolution is a good point and TFS does not provide a clear way to do this, although it can still be done if required. As above, detail about the defect and what the solution was can be added to the description field, or as a comment. In the 'area' field you can select the specific area of the project that the defect relates to, so that you can review the defect by area at a later date should anpther defect arise in the same area. Furthermore, you can link defects together, or to any sprint backlog or product backlog item, so there is always that relationship to refer to.
- Is there a way to automate escalation, notifications, etc.
TFS can notify you via email anytime one of your sprint backlog items (and defects are SBIs) is changed or assigned to you: In the 'Team Explorer' pane, right click on your project and select 'Project Alerts'. Check the first box; 'My work items are changed by others'. and enter an email address to send to.
- It seems like the tool is focused on completing everything within a sprint (including sprint backlog items of type “bug”) by the end of the sprint. Is there a robust way to manage / report on items that might hang around for a while (across multiple sprints or releases)?
you are absolutely correct that the tool is focussed on the sprint, as it should be, since it is intended that all issues are resolved within the sprint they are raised. This is the ideal though and it is recognised that this often is not the case, and each defect item has a sprint number so you can review by sprint the outstanding items. There probably isn't a system to manage these which is as 'robust' as you are used to, however I would argue it isn't required because other aspects of the agile methodolgy make it unnecessary to rely on a tool to do this for you.
Beyond answering these questions above I also want to make the point that most important are the principles of constant communication and participation; the daily scrum is an opportunity for big ticket issues and blocker to be raised, which can create an awareness for a subsequent meeting later in the say of required. It is also recommended to have regular defect and feedback meetings (currently we run them twice a week, although they can be daily if required), which the majority of the team should attend, during which outstanding defects are discussed along with progress on these. New defect and feedback are also discussed and a decision is made on priority, who to assign to and any clarification is requested/given. This facet of agile testing impacts all of the questions above in a number of ways
Report tracking on outstanding defects from previous sprints does not need to be as robust because defects are constantly talked about. You will quickly realise if a defect is still being discussed in the regular defect meetings after two weeks, a sprint, two sprints, etc
Escalation and notification is more efectively communicated by the formal daily scrum and by the constant communication between team members
When a bug is raised it is not done in isolation and then passed to a developer with no direct communication. There is two way communication and often collaboration on identifying, troubleshooting or pin-pointing the defect and developers are encouraged to have a conversation with the tester to understand the defect fully. The regular defect meetings also provide a forum for th
Generally tracking the number of outstanding items, the status of these items and recognising similar defects/relating these to similar defects is also supported in this non-structured way. It may sound somewat flaky in comparison to a traditional approach to testing, however when you take into account that we work sprint by sprint, it is expected that you have only the one sprint's worth of items to deal with (already acknowledged that in reality this is not exactly the case), therefore it is quite possible to maintain a bulk of this knowledge in the team's collective head. I generally know the defects I currently have open, those that are ready for retesting and the big ticket items still waiting to be resolved. For when I do need a reminder I find the reporting on TFS adequate enough to provide me the detail I need. Similarly, if a particular defect arises again, especially if it is a big one, then in my experience someone on the team will realise this and together we are able to find the previous solution.
Put most bluntly, quite a lot of the time, we don't need a formal tool to give us the information about the current state of things; we just know. This is because everyone is involved everyday in constant communication about the project.
There are a number of queries you may want to consider setting up under 'team queries' in the 'team explorer' pane. If you don't have permission to create a team query, then you can create them under 'My queries'. Currently I have set up among others the following:
All active defects (Only defects in-progress or not done)
All active defects - High/Medium/Low priority (one for each)
All defects raised today (useful for the defect meetings; I actually have this set up to show the last 3 days since our meetings are twice weekly, not daily)
It should be possible to set up queries to show this sprint's defects, defects from previous sprints, and any number of other views required. I am happy to help if you have any questions about how to do this.
James