Tag Archives: metrics

How to Measure Team Agility

Quote

Posted on by
Reply

This article has some good thoughts on how to measure the health of your agile teams.  I am personally looking at this currently as I am getting ready to spin up 8 scrum teams and need a way to assess the health of the teams to determine where coaching is needed.  Looking forward to trying out some of these tips.

If you have led, coached, or participated in an organizational transformation to Agile, it will be just a matter of time before someone asks “How do we know if we are truly Agile?” or “How do we know if our Agile teams are healthy and doing well?” Good questions often coming from leadership.

Read more here…

Agile Metrics

Posted on by
Reply

In my last post, Is Working Software Enough, I lamented about the desire of teams to optimize velocity and talked about how that can lead to sub-optimal performance. It has the tendency to push design off the team and onto a separate team which is usually not the best solution (I won’t say never because there are always exceptions). As I mentioned in that post, we would be better off measuring and optimizing value delivered…but how do we do that? Over the past week we have been talking about metrics here at the office as we have been starting to think about changes to our company and department goals for next year. So I thought I would bring up some of the thoughts that have come up that seem like better things to measure and work on optimizing than velocity.

Customer Satisfaction/Delight
If we are delivering valuable software to our customers frequently, they should be satisfied, correct? Given there are often many other aspects that go into customer satisfaction like sales, support, and training just to name a few, this could be a stretch. However, it is still something that most software product companies already measure that is a good indicator of the overall value delivered to customers. Also, there are often some lower level questions in these surveys that can tie back more closely to the software specifically. For example, one thing we have added to our satisfaction survey is a question about perceived quality of the software. Obviously this is a lagging indicator, but depending on how frequently you measure this it may not be lagging by much. Some people avoid lagging indicators, however I think they are important because in reality, there is no way to determine whether what you did was successful or not until after it is done. We can always come up with leading indicators that have historically predicted success, but that doesn’t mean they will always predict success.

Usage
An obvious measure of value could be revenue, or even better, profit. However, calculating this can often get very complicated especially if you are doing incremental enhancements. So if you are having trouble measuring revenue, another measure could simply be usage. If people are using the software, that is a good indicator it is providing them value and this can be much easier to track than revenue in many cases. If you are working on delivered software that would require users to upgrade, looking at the number of clients that choose to upgrade would potentially be another good indicator of value delivered with that release.

Focus Factor
So we’ve discussed two good lagging indicators. Now we get into some leading indicators. One thing that companies usually want to measure is efficiency. Focus factor has been around as a measure for some time, though the standard definition has some issues with it such as it was usually only done at an individual level and assumed that all time spent on a task was productive. Scott Downey at RapidScrum has re-defined this measurement so it does a much better job at measuring efficiency on a scrum team. Using Scott’s new definition, Focus Factor is a measure of how much effort during a sprint resulted in requested, completed, accepted working software. If you’re familiar with Scott, you will know he never uses hours. You always estimate in story points and you also report work in story points. So you end up with some number of story points reported during the sprint (work capacity) and velocity (the sum of the original story point estimate of completed accepted tickets). So a team’s Focus Factor would be their velocity divided by work capacity. An added benefit of this measure, since it is a percentage, it can be used across teams even if they have different story point scales.

What is a desirable focus factor? Well, explaining that isn’t really the focus of this post, so for now I’ll just tell you that 80% is the target for focus factor (this number comes from W. Edwards Deming). So anything too far away from 80% (lets say lower than 70% or higher than 90%) would be an indicator that there may be an issue. The team may be committing to too much or too little, or a number of other things could be going on.

Found Work
In Mike Cottmeyer’s recent post on The Real Reason We Estimate, he points out that many times, an estimating problem really isn’t an estimating problem at all, it’s a shared understanding problem. So if we buy into that (which I do), what is a metric we can look at as an indicator that there is a shared understanding problem? Scott again provides a key measurement here he calls Found Work. Basically, it is a measure of how much unexpected complexity a team discovered mid-sprint on work they had already committed to. So how well did the team understand the work going into the sprint. Scott measures this as a percentage of the original commitment, so he takes the work reported on a card minus the original estimate on the card (again, both estimates and reported work are in story points in this case) and divides that by the original commitment for the sprint (in story points). So this would give us a percentage and the closer to zero the number is the better the shared understanding.

Conclusion
So there we have four metrics, two lagging and two leading, which can help provide good indicators on how our scrum team is performing. All of these measures also encompass the dynamic between the product owner and the team. Both parties need to be doing a good job for these numbers to be good. If the team isn’t building valuable stuff then customer satisfaction and usage would be low. If the team is not working efficiently, then the focus factor will be out of the target zone. Found work being high indicates a low level of shared understanding either between the team and the product owner or among the team about the technology. Does anyone else have other metrics they like to track on their agile teams?

UPDATE: While I was working on this post, Mike Cottmeyer posted an article on his blog, Gaming the Numbers, that talks about this subject as well and there are some good thoughts and discussion in the comments thread I would encourage you to check out.

Is Working Software Enough?

Posted on by
4

For the past few weeks, maybe even months, I have started to feel like the dynamics between the Scrum team and myself, the product owner, were not working in the best interest of the product overall. The team seemed very concerned about achieving a high velocity. They showed reluctance to slow down and spend more time in “grooming” and planning. Previously, in an effort to optimize velocity and reach “hyper-productivity” the team had capped the amount of time the product owner could use for grooming at 5% of the sprint. However, I was starting to think that really wasn’t enough time. It was forcing a lot of important design decisions off onto myself and the UX designer without getting the opportunity to explore different options and get feedback from the development team. There may be situations where this setup works well, however I have a very smart and talented development team and I felt that spending more time grooming and brainstorming potential solutions to the problems on the backlog with the team prior to designing the solution would, in the end, result in a much better product. Not only that, but that additional grooming time would help increase the shared vision among the team which also seems to have been suffering. So I began to think about what the root of this problem was and why this reluctance might exist.

What about value?
One of the core values of the Agile Manifesto is…

Working Software over Comprehensive Documentation

Don’t get me wrong, I definitely agree that working software is much more valuable than comprehensive documentation. However, is working software really enough? What about valuable software? What about software that delights the customer?

If you go past the manifesto and dig into the Principles behind the Agile Manifesto, you see that the very first principle gets at this point as well…

Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.

So there we have it, the highest priority is to satisfy the customer with valuable software. I wonder, if this really is the highest priority, why didn’t it earn a spot on the actual manifesto? The fact that it has been left off the manifesto itself seems to have driven some potentially suboptimal practices. If you look at most of the mainstream materials out there on Scrum, for example, there are lots of resources talking about how to track and calculate velocity. But what is velocity? It is simply a measure of how fast the team can deliver working software. It helps the product owner determine how long it will take to get through the backlog. This is definitely important and useful information…but where in the manifesto or the principles do we value development speed? I can’t find it mentioned anywhere. So if we don’t value speed, why is it the primary thing we are measuring?

If our highest priority is “to satisfy the customer through early and continuous delivery of valuable software” shouldn’t we be measuring that? Shouldn’t we be measuring the amount of value delivered per sprint, or the effect of a sprint on customer satisfaction? Granted, these are much more difficult things to measure and track, but I have never even encountered the idea of attempting to track this mentioned in any mainstream agile materials. If we aren’t measuring and tracking it, how can we hope to optimize it? Instead, we are measuring and tracking velocity, so that is what we will naturally optimize. Scrum says it is the product owner’s responsibility to maximize the value of the product, but I have found few resources with suggestions on how to do that. I also haven’t found any mainstream materials that even recommend tracking value in a measurable way or pay attention to a metric such as value delivered per sprint or release. Instead we plaster velocity everywhere. We externalize the tracking of value and assume it is being optimized while we focus on optimizing velocity when in reality, we should all be focusing on optimizing value.

Thoughts from Agile2011
I was pleasantly surprised at how well many of the presentations I attended at Agile2011 tied into the value concept. I sat in on a few Lean Startup sessions and that movement seems to be much more in tune with this idea. Typically, they will complete one thing at a time, release it into production, and measure the results in order to validate their prior assumptions, then take the knowledge they have gained and make some more adjustments. So they definitely seem to put more value on the outcome (increased signup, conversion, or whatever they are trying to improve) than the output of working software. There was a great example of this recently on 37signal’s blog where they walk you through how they set out to increase the signup rate on their Highrise homepage in a three part post (Part 1, Part 2, Part 3). They had a goal of increasing the signup rate, so they would make a change, push it to production and do A/B testing to see the result. They would then take what they learned and make further tweaks, measure again, and keep repeating until they were satisfied with the results. This is definitely much more focused on delivering value than the typical agile project I have seen.

The Design Thinking camp also seems to be on the right track. I missed Mary Poppendieck’s presentation on this, but was able to attend a separate session on Design Thinking and reviewed Mary’s slides afterward. She is definitely in agreement that “the only valid measure of our success is their [product owner] success” implying that if the product owner is not tracking and prioritizing for optimal value, it really doesn’t matter what your velocity is. She also has a lot of good information on her blog like Don’t Separate Design from Implementation that talks about some of the problems with pushing the focus of design and value off the Scrum team.

I also sat in on a great presentation by Steve Denning about Creating Customer Delight that is summed up very well in his posts on Forbes.com entitled Agility is not Enough. He makes a very good argument that satisfying the customer with valuable software is not enough. We should be focused on delighting the customer. In fact, not only should the agile development team focus on delighting the customer, it should be a strategic objective for the entire company.

Next Steps?
Hearing all these great ideas at the conference helped me see the bigger picture and see what may really be at the root of the problem. Velocity is being measured and plastered all over the place so the team is very focused on optimizing velocity instead of optimizing value. Also, with many of the comments I heard from other attendees at Agile2011, it seems that this problem is much more widespread than just my team. I can definitely understand how this is a problem with the fact that what is claimed to be the “highest priority” principle associated with the Agile Manifesto is constantly glossed over and not emphasized in much, if any, of the mainstream materials. Not only that, but there is almost always pressure from above to “go faster” and now, with agile, we have this nice easy metric of velocity to measure how fast we are going.

So how do we get this intense focus off of speed and shifted onto value? I definitely don’t have the answers and am still working on ways to measure value and increase the visibility of that measure so it can be optimized. I’m also working to communicate that increased velocity is not the ultimate goal. Sometimes you have to slow down to speed up. However, I fear that communication can only go so far. Until there are measurements around value that are given as much visibility as velocity, the desire to optimize what we are measuring may just be too much. How does your team measure value and work on optimizing value?