These days there is a lot of hue and cry about a new model of development called "Agile". The name "Agile" doesn't give any clue as to the fundamental structure of this methodology unlike Waterfall Model or Spiral Model or Iterative Development Model. Instead the name implies two attractive positive qualities "Faster time to market" and "Friendlier to last minute or continuous changes" much loved by the marketing team. Also there is a hidden incentive for the CFO - shorter dev time, causing may be a higher burn rate, but possibly lesser overall cost and definitely immediate returns. In general people like to get things delivered fast and at the same time retain the capability to postpone decision making. The Agile keyword seems to provide both and hence definitely will behold attention of decision makers who need to deliver results fast.

One wonders - So what is the magic in Agile which helps to achieve this? Is there a catch or have we truly figured out the trick behind the "free lunch"?

In this quest, when one tries to understand the structure of Agile we see that there are many different flavors of the same thing which bear the name ‘Agile' unlike a traditional waterfall model of development or iterative development. In general Agile is a loose term referring to different methodologies - SCRUM, XP (Extreme Programming), DSDM (Dynamic Systems Development Method), Agile Up (out of which SCRUM and XP are the more popular ones atleast in the US) which exhibit certain common characteristics.

Some key Agile Methodology characteristics are (derived from Agile Manifesto)

• The individual is more important than process.
• Do not document and convey, just code and convey. A delivery in the form of document is of no use to the client and doesn't signify value or progress
• Sign only Time and Material contracts - Do away with Fixed bid contracts as what you are bidding for will definitely change completely.
• Do not plan, but just react intelligently and self organize
• Be open to change till the last moment
• Business and Dev team need to interact face to face on a daily basis.
• Keep delivering working software every few weeks

If you want to delve deeper into Agile, one needs to understand Agile blockbusters like SCRUM and XP as the techniques practiced in these methodologies mostly constitute the overall techniques of ‘Agile' and define ‘Agile'.

SCRUM

This methodology was invented in 1986 by Takeuchi and Nonaka and was later developed by Jeff Sutherland around 1993 for Easel corporation. It was then adapted for software industry in a formal way by Ken Schwaber in 1995. SCRUM derives its meaning from Rugby which is a way to restart the game.

I repeat here the abstract of Ken's paper (1995) which elucidates the motivation for a new process methodology.

"The stated, accepted philosophy for systems development is that the development process is a well understood approach that can be planned, estimated, and successfully completed. This has proven incorrect in practice. SCRUM assumes that the systems development process is an unpredictable, complicated process that can only be roughly described as an overall progression. SCRUM defines the systems development process as a loose set of activities that combines known, workable tools and techniques with the best that a development team can devise to build systems. Since these activities are loose, controls to manage the process and inherent risk are used. SCRUM is an enhancement of the commonly used iterative/incremental object-oriented development cycle."

If you listen to "The roots of SCRUM" presentation by Jeff, the initial implementer or SCRUM processes (1993), you would be pleasantly surprised when he confidently states that SCRUM would question and break the historical "Cost-Time-Features" iron triangle. This is the weapon which is currently used by many smaller software companies to enable them to break into impenetrable forts of large scale contract projects. This is the dream these boutique software shops with their power (extreme) programmers tries to realize for their clients. But I do not think this claim can be proved theoretically as you cant deliver all functionality in zero and zero cost. I think Jeff may be trying to indicate that it is possible to deliver with lesser time, more features and possibly at a lower costs.

What was thought as impossible to do in 12 months, the proponents of SCRUM cheerfully advocate can be achieved in 12 weeks with three intermediate deliveries!! But they won't sign a contract which promises well defined deliverables, but just a loose agreement and commitment to show rapid visible progress at short intervals. From the client's side he neednt wait for half year wondering what is happening to the money he is spending, but he sees solid results in a month!! If you don't see those rapid results which are promised in a month or two, the understanding is that the SCRUM experiment is a flop and you could annul the engagement and cut the loss. Please note that there is a thread of trust which underlies this engagement principle. This is something hard to accept, and as Jeff points out - like alcoholics who cannot resist their temptation, it is hard to resist the temptation of creating a fixed legally binding contract. One should leave that temptation and boldly adopt this iterative trust approach to make SCRUM work.

In terms of actual results, as per Jeff, Primavera which followed SCRUM trebled their productivity in 3 years and increased quality (reduced bug count) by 20 times in 3 years. So SCRUM is not only faster and flexible, but better in terms of product quality is a very compelling argument presented by Jeff. The most successful implementation of SCRUM was by Borland which exceeded industry productivity by an insane number.

Cool!! But how?? There should have been lot of good managers to manage and plan the project to make it that successful. Well No. The recipe is what Jeff calls Self Organization. He says "Do not manage people, let them be who they are and allow them to self organize and the software will emerge!". This guy has gone bonkers one would think.

This difference in management (or rather non-management) approach is neatly explained by Takeuchi and Nonaka (who coined the term SCRUM) in their seminal HBR article "The New New Product Development Game" where they contrast the traditional model of development and the SCRUM model. Instead of the traditional relay race approach they encourage to ‘follow the holistic or "rugby" approach where the team tries to go the distance as a unit passing the ball back and forth'. In a relay race, the baton is passed between specialized teams - Marketing, R&D, and Production. In Rugby kind of an approach a multidisciplinary team is formed which work together, constantly interact from start to finish and the process and product emerges as a result of this constant activity and self organization.

But then wouldn't it all lead to chaos and anarchy? This is where the SCRUM methodology or rather SCRUM techniques come in to create an atmosphere of controlled chaos, which seems to be an ideal environment to achieve fast, high quality, customer friendly software development.

In the next part in this Agile series, I will delve more into what constitutes those SCRUM techniques and methods which create the controlled chaos atmosphere.