TSSJS2007 - Venkat Subramaniam on Agile
A couple of days ago I assumed that a regurgitation of an agile talk on a blog as no value. Today I’m not so sure. There seems to be much less adoption of agile practices than what I had thought, and Venkat is a great speaker who has a wonderful ability to put things into context, which hopefully will help me do the same as I encounter others who have not yet decided to take a bite of the agile apple.
We are looking for a process that lightweight and sufficient. There is a point of diminishing return for everything we do, so sufficiency needs to be focused on. Machines are getting faster, but we are not really getting better in delivering software applications. Capers Jones has a study that finds that only 10% of software projects were successful: on time, within budget, feature complete, works (failure free). Is this reasonable? Venkat thinks it is, since few other activities have outside of politics have such a dismal failure rate and are put up with. Is software design engineering? The term came from the 60’s where people wanted to be focused on following principles and procedures as a path to success. Success does not come just by trying: we have to learn and make mistakes. This is why its extremely important to be open about what solutions are out there and to find out how things might benefit us. Is software like manufacturing? Its cheap to build software and deploy software, so this breaks the comparison. How much time does it take to produce 100 cell phones? This can be quantified. But how much time did it take to build the first cell phone: 37 years. Manufacturing is predictive, while product design is not. It is our job to take what customers what and figure out what they will be wanting: how what they want is evolving. Most software development is an innovative process rather than predictive manufacturing. Waterfall is a bad idea because it expects that phases are complete once you move beyond them.
The longer you wait between getting the requirements and getting feedback from letting users use the application (not just demoing it), the higher the chance that what you develop is not what they want. Unfortunately one of the constraints is that project planning does not take interaction between users and developers into account. Have you worked on projects where you are fixing bugs months after the deadline? Is this not hypocrisy that we say we meet the deadline, but the quality is bad? Can you rename notepad.exe to the project name, deliver it, and say you have met the deadline? Giving users what they want must be factored into meeting the deadline. It is not enough to deliver something on time, you must deliver what the customer wants on time. It is best to deliver things in much shorter cycles to account for the changing environments and requirements.
Notice in the agile manifesto (link) that it does not say what should be done or what shouldn’t be done, rather it focuses on priorities and principles. No matter what methodology we pick, any methodology that is dogmatic will fail. Venkat does not use any one methodology, but rather does a hybrid of methodologies for his projects. XP brings emphasis on what works, and tells us to do more of what works. For example code reviews are extremely important. But most people hate code reviews because people don’t want to have their code bashed, or there are other political reasons for not doing it. Venkat prefers a more tactical low profile task level code review. XP says, why not do code review all the time: which brings about the instant code review of pair programming. One thing that he feels is a high benefit is that he feels more energized after pair programming, than programming by himself all day. XP also emphasis short cycle, and de-emphasises documentation.
Consider the control variables of software cost time quality and scope. These things are not independent of one another. Do not let your customers control all of these, or they will be picking failure. If you change the time the quality is impacted.
Consider the values communication, simplicity, feedback and courage. Courage is about admitting that things stink and are overly complicated and redoing them. Courage is about taking responsibility and ownership of things. XP takes good commonsense principles to the extreme. If code review is good we will review code all the time: pair programming. If testing is good every body will test all the time. If design is good, well make it part of everybody’s daily business through refactoring. If integration is important then we’ll integrate and test several times a day. Venkat told an experience of someone who had a 3 minute continuous integration build environment where there was an audible shout every time there was a successful build, and a “you haven’t learned a thing” in an intimidating voice when there was a failed build, so that someone would know right away when things fail.
For using XP it is important to look at the size of the team, greater than 10 is more difficult, but doable. Also the culture and work environment is important. If you have no control over those factors he suggests you get another job. If the organization is not willing to succeed then we need to find an organization that is committed to being successful.
Scrum specifies things a little bit higher so that its principles work well on a management level. Venkat prefers a hybrid of scrum and XP when running teams, because he has found the greatest success with that combination.
Evolutionary Project Management (EVO): one of the oldest agile development processes from 1960s by Tom Gilb, published 1976. Short 5 days iterations, evolutionary requirements and design, measurable short list of project objectives.
RUP: skeptical, but it does have a good emphasis on incremental, however often times people place too much emphasis on documentation.
Crystal: No shoe fits all. If you are developing a game and it crashes once in a while someone might cuss. However if you build an application that manages casinos there needs to be more due diligence. Depending on what your impact of failure is your development approach must change. So he has stages from clear to red, where the hotter the crystal is the darker the color and the more emphasis on methodology and structure. Here he mentions the concept of domain driven design where he tries to bring the relevance of what people want directly to the software.
LD: emphasizes that everything is changeable, and that reversibility is extremely important. Needs determine technology, complete don’t construct.. these are some of the emphasis of LD.
ASD (adaptive software development): speculate collaborate and learn cycles with a non-prescriptive nature.
DSDM (popular in europe). One of things he likes is taking requirements prioritized as must have should have and want.
FDD (Feature Driven Development). Some of the things it proposes is controversial. It has short cycles, but expects requirements to be well captured and understood, which can be questioned. Expects class to be assigned to individuals: which goes against the idea of team ownership, a core principle of XP.
Ceremony: when you have to do things. usually this must not be bad, but it depends on what the ceremony is. When it is documentation then it can be bad, a stand up meeting is not that big of a deal. What are the ceremonies that give you the most value (I would say) and that work for your particular team.
He mentions that you should leave time for planning the next iteration and having a retrospective on what worked and what didn’t work. Agile exposes things very quickly so that you will fail faster.
Analogy: in the hospital someone might have heart failure and in this case the doctor does not say: you should have been exercising go run some laps. Therefore if you are working on a project adapt the methodology where it makes sense. If things are critical there is going to be less room for very aggressive XP practices of which the team has no familiarity.
Q: Now a days people try to leverage different tools to help project management and to facilitate communication. How do you think the importance of those tools help agile development.
A: there is a saying that a fool with a tool is a dangerous tool. It is the people that make the difference. Venkat has found that post-it notes are very effective. He also finds that face to face communication is very important. Next there are wikis that are a light weight way to document things so that you can see how things evolve. Tools have not given his teams the best benefit. Integration tools are the tools where he has gained the biggest benefit: unit testing and continuos integration tools. Venkat leans more towards face to face interaction over tools, that is where the emphasis is.
Q: Adapting scrum, and what he has noticed is that stand up meetings evolve into things that make the meeting go much longer, and that many use it as a forum for excuses. How do you manage that, and how do you take what you get from that and communicate that properly to project management who are not looking in burn charts but rather Microsoft project.
A: This is where the scrum master is an important role. First of all people must be on time. Secondly when people start talking the scrum master must politely keep the emphasis on things that are pertinent. If you ask me at the end of the day what my pain point is I’m not going to remember it. So there is a wall where they have a list of what hurts and what works so that during the day people change list list. In the scrum meeting they very quickly touch on each item.