Getting the Most out of Agile. (Even) More Rules, or Back to Basics?

Banner blog post Getting the Most out of Agile

The Agile movement is not anti-methodology, in fact, many of us want to restore credibility to the word methodology. We want to restore a balance. We embrace modeling, but not in order to file some diagram in a dusty corporate repository. We embrace documentation, but not hundreds of pages of never-maintained and rarely-used tomes. We plan, but recognize the limits of planning in a turbulent environment. Those who would brand proponents of XP or SCRUM or any of the other Agile Methodologies as “hackers” are ignorant of both the methodologies and the original definition of the term hacker. – Jim Highsmith[1]

Redefining “Agile”?

Agile is a service-management approach that rests on 4 pillars, themselves supported by 12 guiding principles. Unlike Scrum, for instance, it’s not a framework, and it doesn’t come with hard-and-fast requirements. That can be a good thing—no one wants to be burdened by some restrictive set of rules that ignore what they’re trying to get done. But the absence of hard-and-fast rules has also led to a proliferation of ‘flavors’ of Agile, some of which start to feel as though they’re Agile in name only (AINO).

Now, it’s not that the principles are outdated, or that they were never any use in the first place: it’s rather that practitioners, but especially their managers, have been too agile in their interpretation of Agile. The idea has somehow arisen that Agile is so flexible as a method that it can accommodate the imposition of a lot of new rules (or even “Laws”).

The problem with introducing new rules is that it gets us away from the elegant simplicity, with its orientation towards both quality and efficiency, that is the hallmark of the method. What’s called for, rather, is a return to first principles. That way, once practitioners are clear that what they are doing respects those principles, they can continue to allow themselves some local latitude, and keep reaping the benefits of a healthy flexibility.

The importance of the organic

So how do we get back to basics? Well, here’s a question for all the Agilists out there: What is the key characteristic of software development that “agile” is getting at? To illustrate: writing on the History page of the Agile Manifesto site, Jim Highsmith recalls that participants at an early gathering during the lead-up to the meeting that produced the manifesto were considering the use of the words “light” and “lightweight” to capture the core principle of Agile. 

The point here is that we’re talking about the best figurative term to get at what’s most fundamentally behind Agile. No one thinks that the methods Agile has largely replaced actually weigh anything. Saying that they are the “heavy” things that the new “light” methods have replaced is saying that they are cumbersome, unwieldy, and so on, and thus unlikely to lead to the efficient creation of the best deliverables. By contrast, then, invoking “lightness” is actually invoking, well…agility. 

Trying to capture the key elements of a method invariably entails reaching for the best descriptive term that says concisely, whether figuratively or literally, what that method is about. 

“Agile” is a great way to capture this key characteristic. But because of the name, some confusion has crept in about what it refers to. It is actually about a proposed way of producing in software-development (and later, other) teams. But as we shall see from what follows, the term seems to have been taken to refer, too, to its own methodology, which is then seen as being open to the imposition of new rules and “Laws”. The thing is, though, that there’s no evidence in the Manifesto for this kind of flexibility.  

What’s needed instead, as part of a return to basics, is the recognition of a characteristic that’s already present in the original manifesto: the organic.[2]

Endless tweaking

To see the advantages of such a term, let’s look at what sorts of thing have gone wrong, or “awry”, as the authors of an article in Harvard Business Review put it a couple of years ago. 

The authors begin by citing the four key principles set out in the original manifesto, and proceed to show various ways in which these have not been respected in practice, despite the best intentions. 

They pluck a couple of illustrative examples from their broad data set to show how the first Agile principle, “Individuals and interactions over processes and tools”, has not always been followed: 

For example, in common practice, processes and tools have become the driver of work, not individuals and interactions. In one large Fortune 100 company, the head of digital products said to us, “we’re not allowed to question the Agile process.” In another Fortune 500 organization, product managers and engineers communicate exclusively through their tools, which are used primarily for the former to issue commands to the latter. 

For each of the principles, the authors give one or two examples from their research findings that serve to highlight the breaching over the observing in day-to-day practice. To illustrate how documentation ends up winning out over working software, they point to a case where mini-requirements were written up as tickets and put into a queue, with the result that engineers felt like “short-order cooks in a diner”—working fast and tirelessly to deliver items without being able to offer their input. When it came to responding to change over following a plan, one team’s “attempts to iterate often focused on low-value or strategically unimportant features”—the problem in that case being that de-emphasizing planning meant that strategic objectives were lost sight of, and there was in effect “no plan at all.” 

Laws, rules, steps, and guidelines

The remedy the authors propose involves, they say, “balancing both tactical and adaptive performance”—that is, focusing neither “too much on process and micromanagement” nor on “avoiding long-term goals, timelines, or cross-functional collaboration”. This prescription already feels less high-level, less over-arching, than the authors seem to think it is. And that impression is only confirmed by what follows: a series of six detailed prescriptions that they say teams should keep in mind as they develop their deliverables.

Development, we read, should be: 

  • collaborative (no hand-off)
  • guided by quick experiments to maximize “speed to truth” 
  • customer-centric (which has three further sub-prescriptions) 
  • time-boxed 
  • collaborative (organized to emphasize collaboration)
  • amenable to questioning 

There’s certainly not much to argue with here. Who would maintain, for instance, that teams should not collaborate, or that they should never question their processes? 

So one preliminary thought on these prescriptions is that they are not wrongwhich is another way of saying that they don’t actually deliver that much in the way of insights. That is, they haven’t really diagnosed a central problem, but have contented themselves with offering methodological tweaks around the edges, so to speak. 

A quick detour here to look at one further attempt, in Understanding Fake Agile,[3] to offer a diagnosis and a set of fixes—“quick” because the attempt barks up the wrong tree from the get-go, by appealing to “three laws”: 

  • the “Law of the Customer” 
  • the “Law of the Small Team” 
  • the “Law of the Network” 

Suffice it to say that the appeal to “Laws”, capitalized as though to convince us of the Weightiness of each, flies in the face of everything the Agile Manifesto stood for: where laws require strict, to-the-letter observance, the Manifesto simply calls for adherence to general principles. 

The larger point here is this: as different in other ways as these attempts at diagnosis and correction are, they actually share the same basic structure: look at the original method, offer any number of local, aspectual tweaks, and say Hey presto—just follow these six guidelines, or these five rules, or these three Laws, or these seven steps, to get “back” to Agile the way it was meant to be.[4] 

But in fact, none of the pieces makes an attempt to get to any kind of account of why “Agile goes awry”, despite the title of the first piece. What each does do overall is look at a number of ways in which the uses the method is put to are likely to lead to undesirable results 

The importance of explanation

Let’s look at the four pillars of the Agile Manifesto: 

  • Individuals and interactions over processes and tools 
  • Working software over comprehensive documentation 
  • Customer collaboration over contract negotiation 
  • Responding to change over following a plan 

Now, if we had to come up with a single characteristic for all the items on the left, and an opposing hallmark for those on the right, what would we say? 

One characteristic on the left is, of course, agility, and that in turn would give us to believe that the items on the right represent, among other things, some form of rigidity. 

And that’s true as far as it goes. That said, however, we do not yet have a single explanatorily based characteristic for the items on either side. And the point here is that this is precisely what we need if we are to offer a single explanation for how things have “gone awry.” 

One heading above gave a clue: “the importance of the organic.” The term refers in this case to human activity that is free to develop within a category we can call human creativity and self-development. To schematize: in any unconstrained, organic production process, people—developers, for instance—will ideally develop and enhance their creativity and produce excellent deliverables along the way (and not only  as an end in itself). But constraints there are, to be sure, and they turn up in the principles above only on the right side: documentation, plans, contracts, tools, and processes. Note that none of these entities necessarily restricts or hampers the creativity of producers. It is more correct to say that, if anything does the restricting or hampering, it is a contract, a tool, a process, and so on. 

As for the items on the left: each is either a person or a human activity—with one exception: “working software.” It might seem that software belongs in the right column, but the fact is that, to a developer in the act of developing, software products are an extension of their creative capacities, much like a spear to hunter-gatherers or a plough to the first farmers. The phrasing used—“working software” (almost like “hard-working software”)—lends further support to the view that the deliverables in question could fit easily into the left column.  

The left column, then, represents an organic creativity that can be developed freely without the need for material advantage as an extra motivator. On the right are means to control, to intensify production, and to restrain. Again, this is an overall characterization: as the authors of the Manifesto emphasize, “while there is value in the items on the right, we value the items on the left more”. That is, the items on the right have more-benign functions, but the features I have just listed are what bind them. 

Once we recognize that we are dealing, overall, with a distinction between creativity on the one hand and, on the other, restrictions on it, it becomes clear that local tweaks—rules, guidelines, or Laws—represent the wrong kind of fix. 

Organically channeled production

They are, to put it bluntly, enemies of production. Production makes them uncomfortable. You never know where you are with production; production is the unforeseeable. You never know what’s going to come out. And they themselves don’t want to produce. They want to play the apparatchik and exercise control over other people.”  – Bertolt Brecht[5] 

“Creativity” can mean different things to different people, depending among other things on their relation to the producer. If it can be channeled so that it fits nicely into an apparatus that is focused on profit-making, for instance, that is one thing. If, by contrast, it is free of those constraints, then what is produced cannot so readily be monetized, marshalled, weaponized, what have you.

This is the kind of “production” that Bertolt Brecht had in mind in one of the quotes that opened this piece. But he was speaking loosely, in fact: an employer, for instance, may not know where production “is going to come out”, but the producer themselves, the software developer, is not simply flailing around with no objectives in mind. They are driven, but not necessarily by the need for personal gain: a creativity that has its own set of rules bound up in it—that is organic production in a nutshell. The organic, then, is the terrain on which rules—internally generated, not externally imposed—meet flexibility. It is the organic that stops the agile from turning into a methodological free-for-all. 

The truth is that nothing is wrong with Agile. To take the case of the software developer: of course there are internal, organic rules, but they are there to support creativity. The outside rules—restrictions—are of a different order entirely. And the authors of the Agile Manifesto never said there should be no restrictive rules. It is just that they place greater value on the entities on the left side of their list, all of which are more tightly bound up with organically guided, creative production. Emphasizing that aspect, rather than micro-managing with tweaks, is the best way to get back to basics. 

[1] Jim Highsmith, History: the Agile Manifesto

[2] One Agile coaching company, Organic Agile, styles its coaching services as “organic.” They seem to be onto something, though they have not pointed out that the organic is already bound up in the original schema presented in the Agile Manifesto. 

[3] Steve Denning, Understanding Fake Agile 

[4] In a slightly clever approach, the authors of “How to mess up your agile transformation in seven easy (mis)steps” suggest seven things to do—text—but phrase them as things not to do: “Misstep 1: Not having alignment on the aspiration and value of an agile transformation” can simply be read as “Ensure alignment…”—and we thus have more of the tweaking around the edges we have seen with the other pieces we’ve looked at: treat Agile as a strategic priority, put culture first, invest in talent, and so on. Again, nothing much to argue with—who would suggest not investing in talent, for instance—and not much in the way of keen insights. See more

[5] Walter Benjamin, Conversations with Brecht