Agile is dead. Long live Agile!
We may have failed to make Agile work properly. But, should we forget about it and start afresh?
“Agile has failed. Officially.”
Earlier this month, in his blog Developer Rants, Tamás Polgár wrote an article with the title ‘Agile has failed. Officially.’. Of course, this was a deliberately provocative headline, but Polgár’s blog postings (including this latest one) have a series of examples why the way we do Agile may be fundamentally flawed. Worth a follow, I suggest, if you are into this type of front line stories.
It did make me think, because I also see that whatever Agile is, the way it is often used and portrayed in a business environment seems imperfect or even, to use the strong phrase Polgár used:
Untold millions of dollars and other currencies have been wasted on the idea of Agile, and its ugly bastard kid, Scrum - Tamás Polgár, Developer Rants blog, December, 2nd, 2023
Strong language, indeed, but enough to warrant an article on Agile, I thought, and -accepting the issue- an exploration of whether (and how) it could be saved.
Therefore, I would like to address the following
What is Agile?
What are the problems with Agile, and are they fatal?
What can we do to save the good and leave the bad behind?
So, let’s start.
What is Agile?
It would help to start with the very beginning of Agile. And a good place for that would be the Agile Manifesto. The Agile Manifesto is a written proclamation that includes four values and 12 principles that are intended to improve ways of working within software development. It was originally created in February 2001 (yes, it’s that long ago) by 17 software engineers that called themselves The Agile Alliance. A good article on that is from project manager and strategy consultant Sarah Hoban, The Agile Manifesto Explained And How to Apply It. Allow me to summarise the key elements of the Agile Manifesto here, starting with the 4 Values.
The authors of the Agile Manifesto do say that “while there is value in the items on the right, [they] value the items on the left more”.
There is always a debate going on whether Agile is a Methodology or a Mindset, but it seems to me that based on the original Agile Manifesto, it is less about following a specific process or method, than it is about a way of thinking.
Additionally, over and above the 4 Values, the Agile Manifesto lists 12 supporting principles, represented very well here in this infographic from The Digital Project Manager.
What makes the Agile Manifesto still so relevant today, in my opinion, is its express focus on a people-centric way of working that resonates very well with my own experience of focusing on conversations as the very core of what makes up organisations.
There are many methodologies that are expressions of this Agile mindset. The most well known are:
Scrum
Extreme Programming (XP)
Lean Software Development
Kanban
Crystal
Feature-Driven Development (FDD)
Dynamic Systems Development Method (DSDM)
See this article from Planview that outlines these in a nutshell, if you are interested.
Agile vs waterfall
Usually, Agile is contrasted with the traditional Waterfall methodologies, see for instance this article in Forbes Magazine.
The key difference between the two methods can be compared via these 10 different dimensions:
Note that neither is good nor bad.
Waterfall projects work best for well-defined projects with clear business requirements, where there are clear deliverables with a clear set of agreed deadlines. Also, when long term commitments are required, for instance with expensive software development or legal and/or regulatory approval processes.
Agile projects tend to work best when the business requirements are much less clear at the onset, where there are multiple complex stakeholders that need to be involved or may be geographically dispersed, making continuous engagement difficult.
Not surprisingly, many organisations end up with a mixture of the two types of project, hybrid projects, They take the more specific direction and pre-determined implementation aspects of Waterfall and marry them with the iterative, short bursts of productivity parts of Agile. In this world, there are still large projects with stage gates and overall project plans, but the work is chunked up into Sprints that have a specific scope and timeline. When possible, each Sprint’s output is released, but sometimes large initiatives may take multiple Sprints until they’re ready for prime time. This is the type of project that I see most in my clients these days.
In any case, hybrid or not, agile ways-of-working seem to be firmly embedded in today’s ways of working. And, with that, companies are looking at whether that ‘new’ ways-of-working are good value for money, or not. But, whatever the case, it does seem that there are limited amounts of agile projects that really meet their lofty promises.
What are the problems with Agile, and are they fatal?
Let’s start with one of the most obvious issues with Agile that I have seen. As discussed, the Agile ways-of-working were developed for software product development, and in general has done a decent job in increasing its effectiveness. But, similar to its cousin Lean for manufacturing, the logic and thinking of Agile has expanded into (too) many other business areas.
It is not unheard of to hear people talk about agile budgeting or agile recruitment and so on and so forth. Instead of sticking to the values and principles documented in the Agile Manifesto, as discussed above, Agile was started to be used as a panacea methodology, and -I would argue- to avoid having to carefully plan and prepare for the work products required.
Indeed, when developing a new software product, it is relatively straightforward to get a list of business requirements of the customer, and create a prototype or Minimal Viable Product (MVP) through a series of Sprints (usually of around 2 weeks duration per Sprint), show it to the customers, get their feedback and modify it in the next Sprint iteration, or -if it’s a complete dud - ditch it without too much having been spent on it.
But in other contexts than new software product developments, the whole notion of prototype or MVP makes little sense. What, for instance, can be learned from a prototype budget? What, even, would that look like? A budget created on limited assumptions? Why? In this case, ensuring we plan carefully, take our time to gather all information and so on, makes much more sense.
But, even in the original software context, the agile way-of-working is often not delivering. And I suggest that there are a couple of key causes for that:
Developer Power
Laziness
Here we go.
1. Developer Power
CAUSE 1 - Agile’s relentless pace puts the power dynamic firmly in the hand of the developer, not the customer.
Why is that, you ask?
I suggest there are around 5 key categories of concern.
Let’s look at those in a bit more detail for them to make sense.
Cognitive Biases
I think we can distinguish two types of cognitive bias with the developer that puts the power of what the product eventually will look like firmly in the developer’s hand.
Confirmation Bias - By focusing on information that confirms their existing beliefs or assumptions and then, by default, ignoring contradictory evidence.
Availability Bias - By relying heavily on their recent or easily recalled information, which may not be representative of the broader context to meet the customers requirements.
Implicit Assumptions
Unspoken or unrecognised assumptions about user needs, technical requirements, or project constraints will lead to misalignment and misunderstandings within the developers team.
Stereotypes & Preconceptions
Stereotypical beliefs about certain user groups or stakeholders can heavily influence design decisions and limit inclusivity, accessibility and creative new solutions.
Anchoring
Fixating on initial information or requirements can limit the developers’ exploration of alternative solutions and will lead to tunnel vision.
Technical Debt
Unconscious constraints related to prioritising short-term gains over long-term sustainability (a typical trap that short, sharp Sprints throw up!) can lead to accumulation of technical debt1 and significantly increased development costs.
The main issue with these five categories is that they implicitly determine whatever prototype or MVP the developers present to the customers, but that the latter have no way of recognising these inherent limitations. Ergo, the possibility space gets severely limited without that being known by the business.
2. Laziness
CAUSE 2 - Organisations and businesses use agile principles as excuses to not do the basic careful planning, project management and preparation required, because it is easier.
Laziness is a deliberately provocative term, but it is a term to describe something I have seen a lot. Forgetting the core values and principles that were at the onset of the Agile way-of-thinking, the teams use an Agile way of working as an excuse to avoid doing the (sometimes tedious but always) time-consuming basic project management activities required to deliver a good outcome.
The teams then tend to say they do things in an agile way, whilst they are really doing things in a sloppy way. It is unforgivable and almost always (very) costly. After all, basic planning and preparation practices, trying to gather business requirements upfront, etc, have evolved because they work.
But, because doing things in an agile way is such an accepted method (and often very much misunderstood by senior management!), many programme directors know that they actually can get away with eventual mistakes by claiming that with enough agile sprints eventually the right solution will emerge.
Until it of course never does.
That’s what happens when delivering two (or three or four) week sprints becomes the primary -or even sole- focus, rather than delivering the actual required outcome!
So, is it fatal?
Because we see the issues with Agile emerge everywhere, the Agile community complains that the way it is being used is not real agile. Agile itself, of course, is not the problem, it is the way it is implemented that is the problem! It is for that reason that people are talking about Agile 2. The True Agile! They even trademarked the term Agile 2.
I can’t but help seeing the equivalence to people decrying that the Soviet Union didn’t succeed because it did not implement True Communism, or the pro-Brexit crowd in the United Kingdom pointing out that the problems that the country encounters are because it hasn’t implemented the True Brexit.
So, is True Agile’s promised land always just around the corner, just like the ideal communist world or an ideal Brexit Britain, or is it fatally flawed, as Polgár so expressly states in the article referred to above?
What can we do to save the good and leave the bad behind?
Well, the thing is that Agile can and does work, if used in the context and in the way it was intended, accepting that Good Ol’ Manifesto.
It is tempting to cynically dismiss the suggestion of Agile 2, that I mentioned above, as let’s just crank the handle again. And, in a way, it is. My, indeed rather cynical, view is that by labelling it Agile 2, the people behind the initiative want to build on the good name of Agile and suggest that this next iteration, indeed, will get it right.
Don’t people see how comically self-referential this suggestion is? How many agile iterations of Agile do we need to get Agile right? Two? Three? Ten?
I suggest to drop the pretence. Agile is what it is, it has its place and it has its problems.
So, how can we build on the good and avoid the bad?
Here are some practical suggestions.
Working Backwards
One of the challenges with the Agile model is that you go from iteration to iteration and, more often than not, end up with a non-optimal, disappointing, solution as explained above. One suggestion is to use a practice that is widely published as used in Amazon.
Work Backwards, they call it. And, as perhaps you would expect from a Jeff Bezos company, there is a detailed methodology behind it. But, for the purpose of this article, I want to practically represent it as follows:
Rather than starting with Iteration 1, then 2 then 3, we start with an agreement with our customers what the end point should look like. Unconstraint by what our developers may think. In a joint conversation between all stakeholders, we then ask ourselves the question what set of business requirements we need to start Iteration 1 with, to achieve that Vision.
Then, we follow the Agile approach to tighten up our development towards our future state. Thus we build on the potential benefits of Agile, whilst avoiding the pitfalls we described above.
Go Slow To Go Fast
The second suggestion is what I would call Go Slow To Go Fast. This is to avoid that the short Sprint cycles themselves will be the main driver for the development, rather than the desired outcome.
The idea is that instead of pushing for a quick Minimal Viable Product (the standard Agile outcome of the first cycle) we take our time to ensure that the first iteration represents the customer requirements well. Once we have that confirmed, and the solution is not the usual rather clunky MVP but a more polished solution, we use subsequent cycles to make the solution better and better. The key point is here that we stay in the first cycle until we are satisfied we meet the requirements as well as we can.
The way I often explain this to clients is that only once we are satisfied we have that level of satisfaction, we press the acceleration button. Note that a pivotal leadership role here is to ensure we stay in that first cycle long enough. Because there is a real desire to define the first iteration as good enough, as minimal viable. But we do not want minimal viable, we want an Elegant Viable Product. Perhaps counter-intuitively, slowing things down is exactly the right thing to do.
The best way to adopt these suggestions would be to combine them.
Work backwards to determine the starter requirements set for your project
Then take your time for iteration 1 to ensure you know you can meet those requirements, prior to accelerating your next sprint cycles
Doing things this way, gives the business the control over the outcome that the developer-driven Agile approach simply does not give.
So, is Agile dead?
Clearly, Agile is not dead. It is happily alive and will live on for a long while. Too many businesses are locked into it and too many offer courses for things like scrums and the like. With great marketing, he says sarcastically.
But, the purpose of this article is to make people think about what Agile is, and that for many (?most) businesses and projects it simply does not deliver on its promise.
First, for new software development, as indicated, Agile can work very well. And that’s good.
Secondly, let’s not forget that Agile is not a methodology, but a way of thinking about projects, and in particular, software development projects.
Please, avoid believing that Agile is an alternative for these good old programme and project management practices. Most work we do requires the rigour of those practices. Agile won’t absolve you from doing things right!
There are some practical suggestions, though, that can ensure that Agile practices can be usefully and effectively incorporated in our regular programme and project approaches.
Introduce the Working Backwards way-of-thinking which will give us a real headstart and avoids many of the typical Agile pitfalls
Be brave. Not to try to accelerate as fast as you can, but quite the opposite. Dare to go slow to go fast. Ensure that you know you have the basic requirements well covered before accelerating your development.
Combine those two practices to ensure the business stays in control of the project you are embarking on and to optimise the potential for a successful outcome.
So, the title of this article perhaps was somewhat provocative, but there is a real intention in it. Agile, as many execute it, may well (need to) be dead, but when used sensibly, it can be a very valuable part of our approaches to programme and project management.
Are you a proponent of Agile?
Do you manage to control the downsides of the Agile approach?
Did you know the values and principles as documented in the Agile Manifesto? And, now you do know them, do they make sense?
Technical debt is a metaphor used in software development to describe the cumulative consequences of choosing a quick and easy solution now instead of using a better approach that would take longer. Just as financial debt accumulates interest over time, technical debt can accrue interest in the form of additional time and effort required to address issues or limitations introduced by earlier shortcuts or suboptimal decisions.