Nothing lasts forever and there`s always the next big thing. Has Agile won or is it dead already?
Angela Prentner Smith

Something I’ve been pondering for a while is what’s the next big thing in project delivery? Agile has been around for a while now, and businesses are still struggling with its adoption. I asked the IIBA (International Institute of Business Analysis) LinkedIn group what they thought, and I got about 40 differing opinions. I did some research and again I got a typical mix of views.
 
‘Agile is dead already.’
‘Agile was never meant to be used for what it is now.’
‘Agile is the only way.’
‘Businesses need to be agile, but that doesn’t mean they need to adopt a specific ‘agile methodology.’
 
The conclusion I’ve come to is that actually all of those opinions are right. Most importantly - there isn’t an ‘Agile’ silver bullet that you can implement into your business and make everything better. There isn’t a methodology that you can stringently adopt that will mean all your projects get delivered effortlessly. Indeed the very premise of agile, as a word, a concept and a manifesto, is about being adaptable and responding to change. A strict project or development methodology breaks that very premise. 
 
The Agile Manifesto
 
The Manifesto for Agile Software Development was just that. A manifesto for software development. The problem with that? It was never written to extend beyond software development. As those of us with war wounds from projects know - a large business project never starts and ends with just software development. Agile was developed to improve software development methodology. The original 17 founders of the agile manifesto, are still trying to do that. Some have moved on from the original manifesto and are looking at new and better ways of developing software. 
 
Death by Popularity
 
Yesterday I attended a conference on family friendly working. The word agile was used frequently to refer to flexible working in terms of hours and locations. I also heard agile used in reference to hotdesking. Whilst these applications of the word aren’t actually wrong - agile just means nimble and flexible - they do muddy the water when we have projects of work using Agile methodology. The popularity of the word agile and its potential overuse just adds to the confusion. 
 
There’s also the Agile Marketing movement, and more generally discussions on agile cultures.  Again both meritted uses of the word and potentially successful ethoses. However they may further add to the confusion when business leaders, technologists and marketeers come together to deliver a project - whose agile is this anyway?
 
Design vs Development
 
I’ve personally been stuck in the middle of this argument. What a design sprint entails in an agile world vs a front end development sprint don't deliver the same things, don’t have the same rates of execution, the same stakeholders and beyond. Sprint - that simple concept within the agile nomenclature can cause masses of confusion, and honestly they don’t really align. Design sprints need to test features with users quickly, and so does development, but front end to back end development don’t follow the same creative cycle of design and they have different gestation times.  Sprints are just time periods during which specific work has to be completed and made ready for review, but you can’t get the same amount of development done as design in the same timeframe.
 
Design methodologies
 
Service design, user and customer experience design, user interface design and all the rest could be described as methodologies themselves. But once the design has been done, and technology is required, the agile software development lifecycles begin. These should still iterate, test and learn with users, and business counterparts. They needn’t be at odds, but what is needed is clarity between the design methods and the project approach or development lifecycle. De-risking design and de-risking development can both be achieved through iteration - a key premise of agile, but not necessarily within the same sprint cycles.
 
Beyond Agile
 
I’ve read a bit on DevOps and think this might be where to hedge your bets on the future of software development at scale. Scaling is certainly one of the key criticisms of agile project methodology. What works to build a small app doesn’t necessarily scale to an enterprise wide programme. That said traditional waterfall methodology is flawed, as Mike Gill discussed in Seeing the Light and Go Agile or Go Home. 
 
Where that leaves us
 
So businesses need to be agile - that is flexible around the needs of their teams, their customers and logistically in line with the digital age. But that needn’t be confused with agile software methodology or design methodologies. The most important point in developing a project approach is:

1. That there is an approach that is communicated and understood by all participants
2. That it is not delivered in silos
3. That it is designed around the needs of the customer and that co-creation is looked for
4. There are common goals
5. It is open to change when things don’t work.

 
Meaning it matters less whether we call it agile or anything else and more that everyone knows what’s happening, is working together and willing to be progressive rather than dogmatic about the methodology.
 
I do believe agile is better than waterfall, but I don’t think that’s the last word on the subject.  There will be the next wave of software development methodology and beyond that businesses will be more agile as they are enable by technology and respond to the demands of the workforce and their customers.

We have launched an innovative, industry led solution to embedding people centred project disciplines into businesses to aid 'agile adoption' and 'digital transformation'.  We'd  love to share that with you.

Michael Gill, Agile Business Analyst, discusses the key tenets of the Agile Manifesto for software development and what they could mean for business.

 It's nearly fifteen years since seventeen IT professionals met at a ski resort in Utah, to explore an alternative to documentation driven, heavyweight software development processes. What emerged was the Agile Manifesto espousing four values and twelve principles. Here we look at what this might mean for an organisation either seeking to transition to or already embarked on an agile methodology.

Before I do that, however, let's think about what's happened since the Manifesto's launch, at least in technology terms. The internet was still in its infancy. Some technology experts and organisations weren't convinced of its longevity. Mobile phones were pretty much just that, a mobile phone, and desktop computing was dominated by Windows software. Not even the much vaunted 'Windows XP' had been released. And the terms 'social media' and 'Google' hadn't been invented. 

Since then there has been a revolution. More fundamental than perhaps the early years of the industrial revolution. The internet is part of the very fabric of all our lives. Smartphones, or more accurately, pocket computers are integral to the way we communicate and gather information. There are over 3.5 billion Google searches every day1, and to many the concept of a life without Facebook or Twitter is incomprehensible. New devices such as tablets or Kindles are part of our everyday fabric, and for many of us, Windows remains a distant memory as other players have established themselves in the market.

We live in a world where someone can wake up on a Monday morning with a great idea and weeks later millions of people across the world are running her software.

Yet despite this transformation, many organisations are stuck in the past as far as their development methodologies go. It's as if the last fifteen years never happened.

So what does the Agile Manifesto give us and what do the values actually mean? 

Here's the opening statement along with the four core values in their entirety:

"We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more."

Let's explore how this might play out by examining each of the values in turn.

1. Individuals and interactions over processes and tools

For many organisations, this value can be one of the most challenging. The emphasis on people and communication over rigidly sticking to a process is an uncomfortable place to be. 

By putting communications at the heart of everything you do, there is a greater likelihood of solutions being found to problems, issues aired and confronted, conflicts resolved and consequently forward momentum.

Where adherence to a process dominates, on the other hand, issues and conflicts are more likely to be hidden and glossed over. More emphasis is placed on conformity rather than achieving the end goal of building software. 'We all know our place' might be another way of putting it. 

Many financial institutions, for example, have a very hierarchical, even feudal management structure. This is one reason why these sorts of organisations struggle to adopt and benefit from agile. 

Emphasis is placed on the process, so we change from one form of box ticking in waterfall to another form in agile without ever truly understanding the philosophical shift necessary to realise that structure and hierarchy just aren't that relevant anymore.

The truly self-organising team can be a real challenge to the status quo. The idea that a group of individuals plan and innovate outside what is the norm can be seen as a threat. And many times the diktat will be 'you can be agile, but only so far.'

Adopting agile takes courage, honesty and a willingness to break with the past. The emphasis on people and communications is vital to making this a success.

2. Working software over comprehensive documentation

Hand in hand with an emphasis on the process is the production of documentation. After all, that's how we know the process is working, isn't it? 

What we need to concentrate on, however, is the reason we're developing in the first place - working software. 

I was working on a regulatory project some years back and we were legally obliged to deliver to a fixed deadline. Despite this, the project was starved of resources and instead the focus was on a bright shiny new system for another aspect of the business. Two years in, they were due to go live and informed the new incoming head of IT that there was a problem with the new system. 'Is it a problem in testing?' he naively asked, 'no,' came the reply 'we're still in analysis.' But at least there was lots of documentation.

Ask the question what does the documentation contribute to the goal of producing working software? Does it provide intrinsic value? Is anyone going to read it? If so, who and why?

Other factors to consider are the location of the team(s). Co-located teams with good communication will negate the need for reams of documents. Multiple teams in different locations and possibly time zones might require a degree of more formal documentation though this can still be done with a light touch.

As for the working software itself, this is production quality code, able to be released when it makes sense to do so. Make this the focus and eliminate unnecessary and costly documentation cottage industries. 

3. Customer collaboration over contract negotiation

You would think that collaboration as a concept would be a no-brainer, after all without it, pretty much most things involving two or more people won't have much chance of success. The central aspect of this value though is in the involvement of the business. 

Embedded, involved stakeholders are pivotal to the success of an agile project. Able to make quick decisions, constant involvement in re-planning, (re)prioritisation, guiding the team as they witness the growth of the (working) software week by week. Sometimes it's uncomfortable for them to be put on the spot, but rather a poor decision than no decision at all. 
Their involvement in ceremonies such as the 'stand up', and 'retrospective' fosters a greater degree of honesty. Helping to write or writing acceptance criteria, provides them with a real stake in the development and an investment in the project's success.

Of course collaboration extends to all the team members, some used to handing over documentation down the lines to other disciplines (contract negotiation), can struggle initially with regular interaction, but as they witness the organic growth of the system, they wonder how they ever did things differently. 

If you have to deal with third parties, then things can get a little trickier. Almost invariably you will be saddled with contract negotiation, and many companies make money on the number of change requests raised. 

If you can, you need to negotiate contracts where you can buy the third party's time for use as you see fit, harnessing them into your sprints for example or if they don't work in an agile manner letting them develop up front with incremental deliveries. This is a subject in its own right, but when negotiating with third parties ensure you have flexibility so you are not constrained by prehistoric working practices and philosophies. 

4. Responding to change over following a plan

Eisenhower once said, "In preparing for battle I have always found that plans are useless, but planning is indispensable." Who am I to argue with the great General? A detailed Gantt chart is a complete waste of time and money. 

I used to love creating long, complex project plans. Structuring detailed lists of tasks, assigning resources, creating weekly time-sheets, capturing actuals, it was like producing a work of art. It took a degree of courage for me to realise it had pretty much zero value. 

The reason? As soon as the plan was 'finished' it was out of date. Often, even before it had been distributed to the willing or unwilling participants. And keeping it up to date (yes I was one of the few statistically, who updated their plans) was a huge maintenance overhead. 

And of course, I've been on the receiving end of so many unrealistic plans. I remember one time when regardless of progress, the plan end date always remained the same. Any slippage was accounted for by shaving off time from subsequent 'not started' tasks, regardless of the effort necessary to complete them. Problems only surfaced when there was no time left to shave, we were reaching the end date but had lots more still to do. But up until then it was a success,  management was happy as things were 'going to plan'.

Unfortunately this is the problem. Plans become a method of appeasing layers of management, never reflect the reality on the ground and everyone's left scratching their heads when nothing's delivered.

So my advice, forget your Gantt charts, with the exception of a simple, easy to read milestone plan, that's about all they're fit for. 

Plans fail because of no control over scope and change. Traditionally waterfall has struggled with the very concept of change, after all, the requirements need to be locked down early so that everyone knows what they're building. Change has to be negotiated through a typically convoluted change management or change request process. And there is often fierce resistance to any change to the plan (the plan that doesn't reflect reality), as that will jeopardise the delivery. 

Agile, on the other hand, recognises that change is inevitable and accommodates it. It's part of the regular planning activity (note the word 'planning', not 'plan', Eisenhower would be proud). Change is put entirely in the hands of the business where it should be. Remove some piece of equivalently sized scope to accommodate the change, or accept a longer delivery time. Nothing new or revelatory here, but it leads to tight control over scope, and that control leads to working software within desired time frames.

What can Agile do for my business?

Keeping focussed on the left side of the values provides the opportunity to transform the way an organisation works and delivers software, leading to genuine change, and change for the better. 

And it hasn't stopped at software development. The Manifesto has evolved into a philosophy for business management in general, with executives latching on to agile practices such as its heavy emphasis on flexibility and collaboration. Which, when done properly, can be transformational for any organisation.

Similarly, other business disciplines are taking the spirit of agile and creating their own models and values, again derived from the original Manifesto, for example ‘agile marketing', 'agile UX' and so the list keeps growing...

You can find the full manifesto text, values and principles at agilemanifesto.org. I'll take a look at the twelve principles in a future post.

1 http://www.internetlivestats.com/google-search-statistics/

Michael Gill, Agile Business Analyst, on the origins of Waterfall methodology

There were rumours of course, that decades before, some fool called Copernicus had spouted the same nonsense on his death-bed, but nobody's taken him seriously. This Galileo, however, well he's a different story. To say that the Earth orbits round the Sun and is not the centre of the universe is clearly heresy. Fortunately the Church knows how to deal with him; place under permanent house arrest and force him to recant. 

And as for that so-called genius Sir Isaac Newton, who claims to have mathematically proved Galileo and Copernicus were right all along, what does he know?

Ever tried to climb up a Waterfall?

Let's transport ourselves forward in time, from the heady heights of astronomy into the heavenly world of software development. It's 2001 and the launch of the Agile manifesto. Living in more enlightened times, no one's threatening to burn this group of self-appointed industry experts or have them imprisoned. But it's right to be sceptical. After all why change what we've been doing for years? We have the Waterfall methodology.

You gather requirements, analyse, design, code, test, release. Alright, so quite often good communications are lacking. It doesn't facilitate change particularly well. It's hard to accurately plan and poor old test always seem to have their time squeezed, but after all it's only test. The finished product rarely matches the original business vision and delivery is frequently over budget and late. Better not tell the shareholders.

But should it have ever seen the light of day?

What if the adoption of Waterfall was built on shaky foundations? What if it's sequential model was never the intention?

With ever improving technology, software engineering as a new industry, had to find ways to manage the development of larger and larger systems.

As a contribution to the debate around this problem, in 1970 Dr. Winston Royce, now credited with Waterfall's introduction, issued the white paper 'Managing the development of large software systems'1

Dr. Royce lists each stage of the process:

• System requirements
• Software requirements
• Analysis
• Program Design
• Coding
• Testing
• Operations

We can all recognise this familiar model.

'I believe in this concept,' writes Dr. Royce, then immediately goes onto say 'but the implementation above is risky and invites failure.' He argues that by leaving the testing phase to the end, there is little or no opportunity for change should an error be discovered in a previous step. He concludes his premise by stating: 

'Either the requirements must be modified, or a substantial change in the design is required. In effect the development process has returned to the origin and one can expect up to a 100-percent overrun in schedule and/or costs.'

To address this issue Dr. Royce proposes a series of elaborations iterating around the process, to capture errors and to avoid a calamity during the testing phase. The end result is a complex process diagram, which, when compared to the the basic waterfall model, so the story goes, led a Pentagon official to adopt the simple route instead of Dr. Royce's complex elaboration. And we've been living with the consequences ever since.

An earlier example of a fledgling Waterfall process in action appears in the reissued study 'Production of Large Computer Programs'2 by Herbert D. Benington, originally published in 1956. The paper describes the broadly successive steps undertaken to develop a large system and the per instruction costs around its production.

If it's broken, fix it

Reflecting in 1983 what he would have done differently, Benington concludes:

'I would have worked to test and evolve a system. I estimate that this evolving approach would have reduced our overall software development costs by 50 percent.'

Both Dr. Royce and Herbert Benington understood the flaws in the sequential model, not from any academic standpoint but from hard experience.
​
In 1992 Pope John Paul II officially recognised that Galileo had been right all along; 350 years after the event. It would be a pity to think that the software industry would take as long to recognise the broken software development methodology being used to this day, instead of promoting the practice of iteration and early testing as espoused by those heretics who launched the Agile manifesto in 2001.

1 http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf
2 http://csse.usc.edu/TECHRPTS/1983/usccse83-501/usccse83-501.pdf