coded ruminations

There are some people who respond very well to being told “here is a better way, you should try this”. Then there are people who react defensively, almost out of instinct. These types of people tend to put great weight in the status quo, and believe that what they are doing is good enough. A few even maintain that their way is better without doing due diligence on what you are proposing. Its like the book “Who Stole My Cheese?!!”. Why is it then that when we propose that people try Agile many use the words that imply the judgment on Agile being inherently better? This sets up the scenario where someone tries agile, fails, and then shoots the messenger.

Instead of making judgments for people, I think you come out ahead if you state the benefits and drawbacks of whatever current approach is being practiced, and also the benefits and drawbacks of an Agile approach you are suggesting. This works best from the bottom up, because you need to be able to have developers realize benefits greater than their current approach in order to realize business savings that you can then promote up the chain.

For example: say a business uses a waterfall approach (3 month releases with a 2–3 week up front design.. I find this to be typical). There are several ways that if you are critical of your current process you can gently migrate to an ever more effective one. Even if the end result (where the team decides to stop changing the process) does not look like Scrum, or XP, but is better than where the team started, isn’t this preferable to proposing sweeping changes promising the moon and stars and getting shot down? Here are some areas to look for improvement:

• introduce automated testing
• introduce a build server that runs those automated tests
• propose leaving design tasks for another day to take some time to prove out design discussions and ideas
• introduce code reviews that are pinpointed on topics everyone can agree on that are critical to code health. These will differ from team to team but might include design principle violations, formatting, framework contenders, etc.
• start a refactoring list, prioritize, and evaluate after the release to see if anything on the list would make the next set of features coming down the pipe easier to implement
• keep focus on YAGNI
• invite people to look over your shoulder (pair programming… but don’t call it that… instead say you could use a second set of eyes)
• shorten your cycle by a week, or advocate splitting a release into two releases
• suggest to remove artifacts that you create, that are never used after they are created
• be critical of meetings you are sent to. If you can’t get out of the meeting, suggest that only you go and take notes for others.
• promote unit tests that test units (this works best if code abides by design principles), and code that can be unit tested. This will help your design be more decoupled.

The alleged benefits of doing nothing is that you will continue to doing what it is you are doing, and whatever results you are getting now out of your process will continue. Many people consider known issues (over budget, late release, or 60 hour weeks to make a release) to be better than unknown issues. Try to understand why people feel this way and empathize with them.

The alleged benefits of any agile approach are:

• the smaller your feedback cycle the more feedback you will get
• what you create is able to adapt more easily, because its had to due to the increased feedback
• you have less things that you really don’t need, since what you focus on is feedback driven with less guesswork
• because you create less waste, and are able to change more easily, you incur less risk

Note that these benefits are not code specific. When you apply these benefits to code you arrive at tangible development benefits such as:

• the more often your code releases to version control the sooner you find out if you broke something, and the better you will remember what it was that might be the culprit, saving time and money
• the more often you test code you wrote the sooner you find out when someone else changed something that affects you, allowing you to react to it sooner, saving time and money

The degree to which you realize the above benefits is a factor of your team’s ability to embrace change on all levels. But if you force them to bend all at once, or too quickly, there is a good chance you will break something. Never put your live frog in boiling water; its best to turn up the heat slowly.

Again, I attended this talk for the same reasons as Venkat’s: to gain more breadth in how Agile is talked about and communicated. This talk links to some good resources:

• The design in software development is the source code
• Building Bridges
• Software Design

One of the key enablers for being able to gather metrics is its repeatability. Other Benefits of agile: much less time spent on requirements (traditionally up front, and throughout, vs just a little up front and more throughout). Developers spend more time coding, because they are not in endless requirement meetings up front. Vet architectural decisions early in the process. If you create your standard diagram describing the system, you are not going to prove out that system until implementation time, which comes much later. In agile you are proving out the architecture and its maintainability in every iteration.

While its a bad idea you ask people how long is it going to take to do this, you can ask them how complicated a task is relative to a task that they did previously. Business Analysts can put together user stories that place the focus on the user, rather than the system. The developers then consume these stories (which have much less detail than a use case), and then the developers decide on a high-level architecture (stuff that is hard to change later: what language you are using), and then sit down with the business analyst and rate the story cards on a complexity scale. Its either a 2,4,8,16. I’m asking how complicated stories are in relation to one another. You are not trying to slice things by hours or days. The project manager takes this list of numbers and multiplies them by a load factor (size of project, how many meetings you have) with the assumption that its going to be wrong, because we are going to change it as it goes along. Realize that you are going to get your first data point in two weeks, so that after several months you can hone the load factor number to the point where you will be fairly accurate as you go along. Q: the story doesn’t have enough details though A: Its good enough, and I’m not trying to get an exact estimate. Q: but what about technical requirements that influence the story? A: well you can have technical stories because because these are going to make things more complicated, and so they should be factored in.

The load factor is the magic part of all of this, and it may be wrong, this is a skill that the project manager needs to gain over time. Not only are you throwing the dart on the wall, but you are going to tweak it as soon as it makes sense to do so. There is magic in all estimation processes, and most people are asking each developer to apply their own magic. Its a better idea to have one person apply the magic and get better on this, so at least the error is spread evenly. Too keep from poisoning the well they use rock paper scissors at thought works to throw out their estimate, so that people are not influenced by others. YAGNI: you are not going to need it. Try to only do things that are needed to complete the story. It is better to keep related things that you swear you are going to need, on the back burner for when you are going to implement it. Chances are good that what you write now will need to be heavily modified when it comes to it later.

Agile is actually disciplined because it is driven by story cards. The cards have the domain defined completion criteria. You must have Specific completion criteria that reside in the domain, and it must be a hundred percent done to be done. Until its a hundred percent done its zero percent done.

Metrics: most of the metrics books are outdated. The way we write software now changes abruptly: we are writing more complicated things. Problems have gotten harder, although tools and languages have gotten better. Most text book metrics focus on thousand lines of code and other meaningless metrics. For example how do you measure productivity if someone finds something better that removes 20,000 lines of code? You need to validate your measurements continually. Otherwise you are measuring apples and oranges. Agile gives us consistent repeatable steps.

Testing is the best metric you can have. One thing you hear people talk about is trying to compare software to bridge building. The design is not nearly complete enough to be the design. The design in software development is the source code, and the building is pressing the compile button. Our manufacturing process is really cheap. If bridge builders could build little pieces of bridge and test them as they went along it would be a lot easier to build bridges. So over time they use civil engineering to vet their design. The engineering rigor towards software building is testing. Why bother with all of the plans when you can manufacture things so cheaply and just run it to see if it works. The guy who was the heaviest critic of the man who designed concrete reinforced bridges built the tacoma bridge that collapsed. The concrete bridges were proved to be better through testing, mathematics of the day ended up failing: they claimed that the concrete bridges would not hold up, and that the tacoma bridge would. Glenn Vanderburg has a blog entry devoted to this, in fact, Neal borrowed the bridge example from him. Both link to a facinating article written in 1992 on the topic of whether or not Software Design is Engineering, written by Jack W. Reeves.

Microsoft Project is the most abused piece of software ever. People usually create the gant chart and show the completion date. Then the manager says no I need it done sooner, and then they hack together a shorter timeline. We are wishy washy on completion: we make up percentage numbers as we go. And most managers as a result believe that we failed mathematics. Binary completion criteria for all tasks is the key factor of being able to do metrics. This gives you the ability to say to a manager we are 62.3 percent done. This also gives you the ability to say to people who are changing the scope: here is our list of things we are doing, what would you like to remove so that we can do this new thing. You also want tasks that are as atomic as possible. Things that take 5 days are not so good, things that take 2 hours or a day are great, things that take 8 days need to be split up.

Project managers at thoughtworks use spreadsheets and not Project. There was some discussion as to wether or not share Sparky’s spreadsheet. They decided that they would allow you to download Sparky’s spreadsheet, but to remove the formulas. The spreadsheet is the result of many many projects that Sparky has worked on. The first page is the high level overview for managers. The first box is the sanity check question: “Do you have enough development capacity remaining to complete the remaining Priority A cards. “Number of cards through narrative” means that the BA’s have filled in details and have determined what the completion criteria are.

The binary completion gives you a concept of done that does not have the fuzziness of I’m % so far done. Q: who manages this A: the project manager does this. The PM has 3 jobs: 1. keep client and business people out of the way of the developers 2. keep good snacks 3. keep everything going and on track and measure progress, move impediments letting people do their development work instead of emails, meetings, and tech support. Q: how do you handle bugs A: the failed tests get high priority cards for the next iteration.

Code coverage tells you not that things are correct, but rather that your code has been touched. You want to try to not have the users be the first people that are touching the code. You can download slides and examples from Neal’s site, which includes Sparky’s spreadsheet.

A couple of days ago I assumed that a regurgitation of an agile talk on a blog as no value. Today I’m not so sure. There seems to be much less adoption of agile practices than what I had thought, and Venkat is a great speaker who has a wonderful ability to put things into context, which hopefully will help me do the same as I encounter others who have not yet decided to take a bite of the agile apple.

We are looking for a process that lightweight and sufficient. There is a point of diminishing return for everything we do, so sufficiency needs to be focused on. Machines are getting faster, but we are not really getting better in delivering software applications. Capers Jones has a study that finds that only 10% of software projects were successful: on time, within budget, feature complete, works (failure free). Is this reasonable? Venkat thinks it is, since few other activities have outside of politics have such a dismal failure rate and are put up with. Is software design engineering? The term came from the 60’s where people wanted to be focused on following principles and procedures as a path to success. Success does not come just by trying: we have to learn and make mistakes. This is why its extremely important to be open about what solutions are out there and to find out how things might benefit us. Is software like manufacturing? Its cheap to build software and deploy software, so this breaks the comparison. How much time does it take to produce 100 cell phones? This can be quantified. But how much time did it take to build the first cell phone: 37 years. Manufacturing is predictive, while product design is not. It is our job to take what customers what and figure out what they will be wanting: how what they want is evolving. Most software development is an innovative process rather than predictive manufacturing. Waterfall is a bad idea because it expects that phases are complete once you move beyond them.

The longer you wait between getting the requirements and getting feedback from letting users use the application (not just demoing it), the higher the chance that what you develop is not what they want. Unfortunately one of the constraints is that project planning does not take interaction between users and developers into account. Have you worked on projects where you are fixing bugs months after the deadline? Is this not hypocrisy that we say we meet the deadline, but the quality is bad? Can you rename notepad.exe to the project name, deliver it, and say you have met the deadline? Giving users what they want must be factored into meeting the deadline. It is not enough to deliver something on time, you must deliver what the customer wants on time. It is best to deliver things in much shorter cycles to account for the changing environments and requirements.

Notice in the agile manifesto (link) that it does not say what should be done or what shouldn’t be done, rather it focuses on priorities and principles. No matter what methodology we pick, any methodology that is dogmatic will fail. Venkat does not use any one methodology, but rather does a hybrid of methodologies for his projects. XP brings emphasis on what works, and tells us to do more of what works. For example code reviews are extremely important. But most people hate code reviews because people don’t want to have their code bashed, or there are other political reasons for not doing it. Venkat prefers a more tactical low profile task level code review. XP says, why not do code review all the time: which brings about the instant code review of pair programming. One thing that he feels is a high benefit is that he feels more energized after pair programming, than programming by himself all day. XP also emphasis short cycle, and de-emphasises documentation.

Consider the control variables of software cost time quality and scope. These things are not independent of one another. Do not let your customers control all of these, or they will be picking failure. If you change the time the quality is impacted.

Consider the values communication, simplicity, feedback and courage. Courage is about admitting that things stink and are overly complicated and redoing them. Courage is about taking responsibility and ownership of things. XP takes good commonsense principles to the extreme. If code review is good we will review code all the time: pair programming. If testing is good every body will test all the time. If design is good, well make it part of everybody’s daily business through refactoring. If integration is important then we’ll integrate and test several times a day. Venkat told an experience of someone who had a 3 minute continuous integration build environment where there was an audible shout every time there was a successful build, and a “you haven’t learned a thing” in an intimidating voice when there was a failed build, so that someone would know right away when things fail.

For using XP it is important to look at the size of the team, greater than 10 is more difficult, but doable. Also the culture and work environment is important. If you have no control over those factors he suggests you get another job. If the organization is not willing to succeed then we need to find an organization that is committed to being successful.

Scrum specifies things a little bit higher so that its principles work well on a management level. Venkat prefers a hybrid of scrum and XP when running teams, because he has found the greatest success with that combination.

Evolutionary Project Management (EVO): one of the oldest agile development processes from 1960s by Tom Gilb, published 1976. Short 5 days iterations, evolutionary requirements and design, measurable short list of project objectives.

RUP: skeptical, but it does have a good emphasis on incremental, however often times people place too much emphasis on documentation.

Crystal: No shoe fits all. If you are developing a game and it crashes once in a while someone might cuss. However if you build an application that manages casinos there needs to be more due diligence. Depending on what your impact of failure is your development approach must change. So he has stages from clear to red, where the hotter the crystal is the darker the color and the more emphasis on methodology and structure. Here he mentions the concept of domain driven design where he tries to bring the relevance of what people want directly to the software.

LD: emphasizes that everything is changeable, and that reversibility is extremely important. Needs determine technology, complete don’t construct.. these are some of the emphasis of LD.

ASD (adaptive software development): speculate collaborate and learn cycles with a non-prescriptive nature.

DSDM (popular in europe). One of things he likes is taking requirements prioritized as must have should have and want.

FDD (Feature Driven Development). Some of the things it proposes is controversial. It has short cycles, but expects requirements to be well captured and understood, which can be questioned. Expects class to be assigned to individuals: which goes against the idea of team ownership, a core principle of XP.

Ceremony: when you have to do things. usually this must not be bad, but it depends on what the ceremony is. When it is documentation then it can be bad, a stand up meeting is not that big of a deal. What are the ceremonies that give you the most value (I would say) and that work for your particular team.

He mentions that you should leave time for planning the next iteration and having a retrospective on what worked and what didn’t work. Agile exposes things very quickly so that you will fail faster.

Analogy: in the hospital someone might have heart failure and in this case the doctor does not say: you should have been exercising go run some laps. Therefore if you are working on a project adapt the methodology where it makes sense. If things are critical there is going to be less room for very aggressive XP practices of which the team has no familiarity.

Q: Now a days people try to leverage different tools to help project management and to facilitate communication. How do you think the importance of those tools help agile development.
A: there is a saying that a fool with a tool is a dangerous tool. It is the people that make the difference. Venkat has found that post-it notes are very effective. He also finds that face to face communication is very important. Next there are wikis that are a light weight way to document things so that you can see how things evolve. Tools have not given his teams the best benefit. Integration tools are the tools where he has gained the biggest benefit: unit testing and continuos integration tools. Venkat leans more towards face to face interaction over tools, that is where the emphasis is.

Q: Adapting scrum, and what he has noticed is that stand up meetings evolve into things that make the meeting go much longer, and that many use it as a forum for excuses. How do you manage that, and how do you take what you get from that and communicate that properly to project management who are not looking in burn charts but rather Microsoft project.
A: This is where the scrum master is an important role. First of all people must be on time. Secondly when people start talking the scrum master must politely keep the emphasis on things that are pertinent. If you ask me at the end of the day what my pain point is I’m not going to remember it. So there is a wall where they have a list of what hurts and what works so that during the day people change list list. In the scrum meeting they very quickly touch on each item.

Venkat’s talk as not started yet, and the keynote questions are droning on. I’m attending the talk because for as much noise as Agile has made there has not been a proportionate level of either criticism or adoption. Instead my perception of what is happening is that most shops (> 50 percent) are happily chugging along ignoring pain points and doing business as usual. Perhaps this is because the current mode of operation is good enough, though others suggest that these people just don’t “get it”. I think that its just in peoples nature to resist change. At the very least I think that people should try, fail, figure out if they did everything as prescribed, if they did perhaps its not for them, and if not, try again. I’m hoping to get more ideas from this talk about how to position Agile to people who have not yet decided to investigate the alleged benefits of shortened transparent continuous feedback cycles. For all the arm waving that has been going on with regard to Agile I think there is room for a bit of pragmatism. In a way agile methodologies are too complex and can be simmered to a few principles, in my opinion. The engineering practices that arise from these principles however sound deceptively simple when presented in a nice powerpoint, but are in truth quite complicated to implement given the huge amount of variance in politics and skills present in teams. Anecdotal evidence for this is that there were 60 people at a Test Driven Development talk I listened in on yesterday, for a concept that people have had years of time to investigate, experiment, and try out (not that it will work for everyone, but that if they had they wouldn’t be in an intro to TDD talk). I was surprised that the talk was even scheduled, and didn’t expect that much interest. It goes to show that these topics are still being chewed on in the great master of the development community, and that most have not yet decided wether to swallow the concepts, or violently puke. Situation Normal …