Agile is critical to today’s business
@john_mathon: The purpose of agile is to learn fast. Yes, but… For a GOAL. Don’t agile in name only Don’t agile for agile sake @john_mathon: think about the #ROL RETURN ON LEARNING not the ROI for new agile initiatives.
Sometimes a tweet cannot get the full meaning across easily. I am elaborating on a couple comments I made at the TCS Innovation Forum where speakers talked about Agile. Agile impacts the way the whole organization delivers products.
In today’s cloudified world it is almost the only way software is delivered. We are listening to our customers and improving software rapidly, offering new experiments in interaction paradigms, testing out new products and services constantly. Agile is the only way this can be done. As a result the entire business is organized around the notion of incremental smaller deliveries.
Making the Agile machine run effectively and without hiccups is a tough job for the entire organization including CIO, CTO, CMO, CSO, … who need to be engaged.
The Agile process
Let’s return to a basic premise of Agile. Agile is about making fast delivery of software to get feedback from customers to improve the product fast which meets the needs of the customers. There are many parts to making this work well.
1) Providing user stories (examples of things customers would like to be able to do with the software or service) that are detailed enough but not too detailed
2) Understand your team and the stories enough that you can estimate the duration to complete a story with the team you will have
3) Structuring the right stories in an iteration that can be accomplished and deliver the most value to customers
4) Making sure that the stakeholders are cognizant of the reasons and limitations, history and content of the iteration so that they are in sync with the result that is expected
5) Performing the iteration with consistency and quality so that at the end of the iteration you accomplish the goal of the iteration
6) After the iteration the stakeholders agree and the customers (if a customer delivery was made) what the results were including negatives that were uncovered (badness)
7) Holding a retrospective with the team and all the appropriate people to determine what badness happened and what corrective measures could be taken to prevent similar badness from recurring
8) Implementing learned lessons from the last iteration
If Agile is done well
If executed well as I have seen and done it is a beautiful and inspiring thing to see happen. Customers see a steady stream of well thought out features that meet their demands quickly. The business owners understand what is going to be delivered on what time scale and what to expect from their engineering and delivery teams. The teams execute with high quality and learn from their mistakes rapidly so that they execute with higher and higher precision gaining the trust of the organization, investors and customers.
If Agile is done poorly
Customers complain they get features that don’t make any sense to them, Developers are late in developing and start scrimping on important quality aspects of delivery, the organization becomes disenchanted with engineering and delivery and people are fired and there is great upheaval. Other parts of the organization disrespect engineering and don’t trust or understand what they are doing or why.
Why does Agile get done poorly?
There could be failures in any part of the process I outlined above but there are places I’ve seen in my years of software engineering management that are more likely to be the cause. The critical pieces that many organizations miss implementing well in Agile are:
1) Story definition
The very first part is defining stories. Frequently I have seen stories that aren’t stories at all. They are 5 or 6 words that some group knows what they think it is but others may be clueless or have a very different idea.
The basic simplest requirement of a story is that it be phrased something like: “As an online order taker I can change the billing method in 20 seconds while the customer is on the phone.”
The idea is that the people understand who the feature will affect, what exactly the functionality desired is and metrics around the feature that make it acceptable. Instead stories can be very vague allowing the result to be highly unsatisfactory to everyone.
This is really not that hard but it is frequently, very frequently messed up. The most common situation is people are too vague in the story definition but it is also frequently the case the story definition is too specific forcing developers or architects along a path that actually makes no sense or could be done better.
If the story is too specific it can lead to developers feeling like they have no creative choices but also it is very likely that whoever specified things in such detail had a mental model of things that is only one way to do something and in many cases a very bad way to do it. So, the story should leave open things that aren’t critical to the function wanting to be achieved.
2) Estimation process
Frequently the estimation process is over-simplified. This isn’t always bad. Initially it is a big mistake. In the beginning or at major points of transition of team members or change in functionality the estimation process breaks and people make bad estimates. What happens is that estimation in the agile way is done so that people who have familiarity with what took X amount of work in a previous iteration will take some multiple of X. This is left purposely abstract not looking at X as a number of hours but simply looking at the difficulty and comparing it previous tasks accomplished. The velocity is then the number of these X type tasks can be done with the team.
The team can change. People leave or go on vacation, they have other responsibilities. The team may not have familiarity with the task or tools being used for doing this new task. Over time a team should converge on a velocity but frequently the convergence breaks when things change and the team needs to relearn its velocity. When that happens you have to be very systematic at understanding why your velocity changed. If your velocity doesn’t stabilize it means the team isn’t estimating well and that leads to distrust by the organization in the overall estimates putting the entire project at risk.
The estimation process is complicated and a person familiar with the technical aspects of this should be used to train and help the team get on track with good estimation practices or the team will not converge on a good velocity estimate and the iterations will be wildly off the mark all the time.
3) Engaging the rest of the organization in the agile process
It is critical that the stakeholders are aware of the reason that stories were selected, why the number and scope was selected and agree. There are 2 reasons I’ve seen this fail most often. The first is when Engineering is not good at explaining these things or doesn’t really involve the stakeholders. When they do this they are really operating in a waterfall methodology.
A key principle of Agile is to keep the stakeholders supportive they must be engaged and aware of the tradeoffs and why they exist. Otherwise, managers tend to assume that engineering is screwed up, lazy or other negative perceptions. It can also be Managers feel they are too busy to be engaged and refuse to provide time to understand. Then they get upset when things blow up and don’t happen at the pace they expect. This can be managements fault.
They may simply be guilty of poor management or they could be purposely trying to avoid complicity. Managers are smart. If they listen and understand and then things blow up they can be held complicit in the failure. If they were involved then they agreed to everything. Political management will avoid being drawn into the process so they can later lay the finger of blame without it backfiring on them.
4) Retrospectives and transparency
Another key aspect of Agile is the retrospective and the process of transparency. For the process to converge on a smooth machine that operates as people expect there has to be transparency and honesty. This is hard in many organizations. There is retribution for admitting any flaw.
If management doesn’t understand the process and its failures, why the failures occurred and what corrective actions were taken they will become disenchanted.
If engineering feels that management is overreactive and uses failures to punish individuals or the team then the morale of the team erodes. This is a really tricky thing and it is hard to be truly honest about what went wrong, to admit failure and it can be very hard to identify the reason accurately.
Frequently if the reason is too personal it can lead to firings or if it isn’t personal enough it doesn’t solve the problem. To me this is a big management cultural issue.
I prefer smaller organizations where we know we hire the best people. We therefore know that if an error is made anybody could have made it. It is simply something someone or some group of people didn’t understand and they will do better. Nobody assumes that it was an egregious error that indicates incompetence. It may be the case that people are incompetent and it may be the case some people don’t belong in a particular job or doing a particular thing. That is a management question as well.
The process of talking honestly about what went wrong, trying to fix it is critical to becoming a great company, to creating consistency and improvement. So, this is the learning part of agile and is arguably the single most important aspect. Yet it is also in some ways the hardest.
Agile involves the entire organization
I think that you can see from the above short description that Agile is a process that requires participation of the entire organization. It goes all the way to culture. If the culture doesn’t support transparency you aren’t going to do the retrospective process very well and you aren’t going to learn.
You may have some exceptional people who overcome the problems of others. That is frequently the case however, if there are one or two people who perform and everyone else is a dolt I suspect you really have a major transparency problem in your organization.
There can be problems in management who must be engaged in the process for it to work. There can be problems in engineering whose leadership tries to avoid communicating, communicates too much or can’t give the true picture of what’s going on.
Why does agile work better than waterfall?
Some will argue that agile and waterfall are similar ways of getting to the same endpoint. Some will argue that the overhead of the process of agile slows down the development or leads to things being done wrongly from an architectural perspective. This is wrong.
Agile takes many small steps. There is overhead in dealing with the stories, the retrospectives, the demos and meetings that seem required for each iteration therefore many people think that the agile process can slow things down. This is a major misunderstanding of the problem of software development.
Theoretically if you want to achieve goal X using waterfall you carefully analyze goal X and understand everything involved in implementing goal X. You understand the team, the work needed and the time. You specify it all out carefully, review it with experts and you are good to go. The problem with this is that frequently you arrive at goal X late or not at all because of the large commitment and changes in goal X by the time you achieve goal X the company may have decided they don’t even want goal X.
There are many reasons these projects failed and usually it was extremely damaging to individuals, groups and the whole organization when they failed. Agile leads to arriving at point Y. You did not know Y in advance. It is where you all agreed during the process to end up but you didn’t know Y when you started. It is unlikely point Y looks exactly like X. Agile says that because we are evaluating and changing what we are doing at each step we get to a different point than X, presumably a better point than X because of the constant adjustments.
We achieve Y with agile not X. The difference is that at each point along the way to Y we chose a path that let us stop with something useful so that the whole thing is not a waste. We also learned a lot about our customer, about the requirements, about our teams capabilities, what other ways to achieve X that are different maybe better than X. Agile says that the stepwise process leads to a better faster outcome than X (point Y) than if you tried to push in one giant push to X.
Summary
Agile is a critical part of the modern enterprise. So, everyone seems to assume it as a given. However, that doesn’t mean your organization really knows how to do agile well or that your organization has the right management or culture to make agile work well.
I can say that when agile is working it is brilliant and better than any waterfall process I ever did. When agile first started there was a lot of faddism and people would brag how short their iterations were. There were a lot of complaints from different parts of the organization.
All kinds of things were tried and differences emerged but over time we have come to understand this a lot better. I don’t think agile is well implemented in most corporations. I don’t think many people are schooled in agile well enough in engineering let alone in all the other parts of the management yet it is critical to the smooth and efficient functioning of the modern corporation.
CloudRamblings 
One thought on “A Simple Guide to Agile Software Development ”