The problem with growth: why everything is failing now

The rows over Scrum vs. SAFe or Design Thinking vs. Lean UX miss one important point — and it’s killing companies.

When the Agile Manifesto was inked in 2001, it was supposed to spark a revolution, and it did: by 2023, 71% of US companies were using Agile. The simple list of commitments to collaboration and adaptiveness branched into frameworks such as Scrum and Kanban.

“Agile” was about having a responsive mindset, not about what process you followed, but it became about which process you followed.

Agile was designed for engineering teams but spread to whole companies. Scaled frameworks emerged to coordinate Scrum teams across whole companies, with a sprawling training and certification industry. In 2022, the enterprise Agile transformation industry was predicted to reach $142 billion by 2032.

When asked their goals for these transformations, 83% of managers said “fast deliveries to customers”

and that’s where the trouble started.

Source: www.scrumatscale.com

Product teams for end-to-end delivery

Scrum@Scale cites Harvard research that the optimal team size is four or five people, therefore a Scrum of Scrums Team should be four or five teams of four or five people:

“As a dynamic group, the teams composing the Scrum of Scrums are responsible for a fully integrated set of potentially shippable increments of product at the end of every Sprint. Optimally, they carry out all of the functions required to release value directly to customers.”

The Scrum Guide agrees that “Scrum Teams are cross-functional, meaning the members have all the skills necessary to create value each Sprint”, and also describes the team as “small”.

Whether or not the guides specify that the “unit of value” must be a product/feature, it has been widely interpreted as such: in practice, most product teams consist of a product manager, product marketing manager, engineers, and, occasionally, a designer.

The expectation is that those five people should be able to do everything that is required to bring that product to market, and that every team member will eventually learn how to do everything.

A 2021 study by the University of Gothenburg said that “tech workers over 35 are considered old in the industry,” but I only know one person who can design and code to anything like professional standard, and she’s even older than I am!

Learning takes time.

I have an M-shaped profile — deep skills in several areas honed over three decades in a variety of industries. I have had a little bit of training in scripting and designing and many other business skill areas, but no sensible company would ask me to design a logo for their flagship product, or push code directly to production. There are only two or three skill areas where they would expect me to deliver professional work, which have usually been the roles I have been hired for after a job was created when management agreed that there was nobody already there who could do it well enough and/or had the capacity to do it.

Companies, it turns out, need specialists.

Specialists in a generalist world

The most popular frameworks were developed by engineers who evidently didn’t spend much time considering anything that isn’t software development.

Perhaps that’s not surprising: in the previous two companies I worked for, engineers had little contact with the rest of the company.

At the last, software development was mostly outsourced to teams in Manila, and the UK-based product owner would have daily standups at 7am before a day of meetings with other managers. In other words, the developers had no contact whatsoever with “the business”.

At the company before, developers worked with business analysts who would meet with product marketing managers who would coordinate with every other business function. I had been there fourteen years before I ever even met a developer.

It is logical, therefore, that every non-engineering function is described with an airy wave of the hand — “supporting functions go over there” — with the same abstract regard for “everything else” that enterprise companies have had for software development in the past.

Perhaps the only framework to really consider it is SAFe, the widely-reviled rebranding of RUP which does, at least, acknowledge “the business” but frames it in a baffling and overly-prescriptive set of processes.

Where, in the other frameworks, user research is either absent altogether or assumed to be the work of the five team members, SAFe at least has the familiar “double diamond” explicitly in the framework, so we’ve gone from

just shove a few lines of code out there and see who bites

to

make some attempt to figure out user needs and run a few prototypes before launching it.

None of these frameworks consider the actual things required to run a successful large business for very long.

The things we have forgotten

Strategy guru Roger Martin recently complained that most business schools no longer teach the foundations of business strategy, as defined by Michael Porter. Porter’s teachings can be summarised with a question:

What can we profitably offer which customers need and that competitors will find hard to imitate?

Those three core elements of strategy require three sources of knowledge:

1. What customers need — a deep enquiry into the motivations, habits and behaviours of (potential) customers, segmented to discern which would be the most lucrative to target
2. What else customers might use — a sound understanding of the competitive landscape including comparators (substitute products that fulfil the same need — like how McDonald’s competes with both Burger King and a homemade sandwich to fulfil the need of hunger), and the capabilities of each rival
3. What we can profitably offer — a strong knowledge of your own capabilities and resources so that you can choose to offer something that is easy and inexpensive for you to offer but would require heavy investment for a competitor to offer the same, dissuading them from doing so.

Porter created two widely-used strategy tools to work these things out: the Five Forces model.

and the Value Chain model.

Notice how functions are included which are missing altogether from most Agile framework diagrams.

To answer the core questions that inform the Five Forces analysis, an important business function has traditionally been present in companies, but has all but disappeared in the startup world:

Market Research.

Given that almost every strategic decision will be based on data about user needs and market dynamics, most companies in the past agreed that this work was too important to be conducted by an untrained generalist. Market research is a highly professionalised industry with nationally-accredited qualifications on a par with accountancy: product managers in large companies don’t do their own corporate taxes, either.

Consumer research, product design and data analytics sit in the lateral ‘technology development’ channel in Porter’s model, just as recruitment and training flow along the HR channel.

In traditional companies, market researchers conducted the bulk of consumer research, which was why it was initially OK for designers without those specialist research qualifications to conduct some evaluative user research: it was low stakes stuff.

When innovation labs sprung up in the wake of The Lean Startup, they were small teams with a limited budget who could freely experiment outside of the usual checks and safeguards because the work they were doing was outside of the products on which the business depended for income. It was OK to fail, and most new products did.

It was only later when that innovation-lab model became a substitute for the traditional value chain (and Lean UX a replacement for market research), and with it, teams conducting work that they are not qualified to do and making business-critical decisions based on the results.

The problem with growth

When you start a company, one of three things will happen:

1. You will fail within the first year. This is the fate of most businesses.
2. You find a stable fit as a boutique company with under 40 employees. According to Daniel Priestley, once you have more than 40 employees, you have to grow rapidly or you will buckle under your own weight.
3. You become a large company. Large companies operate very differently to small companies and, according to Forbes, most companies that grow very rapidly fail.

Look back at the Scrum team diagram at the top of this page and the Value Chain model further down the page. They look very different, don’t they?

When you have only five people in your company, it’s OK to make a few mistakes and be a little bit scrappy — nobody has any particular expectations and, for these fledgling companies, those first few customers don’t mind very much if it’s all a little rough around the edges. You don’t have to worry about big, expensive market research projects because you know your customers by name.

Even so, the disproportionately large number of startup founders with MBAs (Stanford alone has launched over 1500) have learned about the importance of good quality customer data, and about “the business” — the winning combination among top unicorns, it seems, is where the founders are engineers but their first hires are “business” people.

When you have five thousand people in your company, customers expect polished performance and security. If you screw up in a five-person startup, you will have already accepted the risk of starting a company. If you screw up in a five-thousand-person company, the mail guy might lose his house.

Venture Capitalists aren’t so concerned with screw-ups if the money is rolling in. Since the failure rate is so high, they push startups for growth-at-all-costs, often crushing them in the process. A few years back, Fortune noted that two-thirds of the fastest-growing companies had failed just five years later, because they didn’t have any system in place for selling new products to new customers.

Hypergrowth companies are rarely profitable for a very long time, and by 2021, “ran out of cash” replaced “no market need” as the number one reason for failure.

When you align teams around products rather than segments, it makes it harder to nix the duds. Imagine if Boots No 7 has segment-based skincare product teams for women aged 15–25, 25–35, etc. I don’t know their structure, but they seem quite happy to cannibalise their own products within each segment. In segment-based teams, a single product manager and product marketing manager would look after all skincare products for women aged 25–35, and so on, developing a very deep understanding of that segment.

If you’re aligned around segments, you only need to do needfinding research project once, and can cover subsequent research with regular lightweight testing, since you already understand the needs of that segment for any other products, unless enough time has elapsed that those needs have substantially changed.

If each individual product has a team, that’s a very expensive way to organise people — especially given that over 90% of products fail. Every time you want to introduce a new product, you have to restructure your teams.

Since maintaining teams is so expensive and so many products fail, the idea is to ship as quickly as possible, to “get value to customers” without any real sense of what that value is.

Product-based teams don’t value good quality user research because nobody has ever told them that they should.

Twenty years ago, “waterfall” project management meant development cycles that could take years. A six-month market research project would establish the user needs, which would be developed into a comprehensive list of requirements, and those customer needs would not be revisited until the product was launched and the satisfaction research would take place, which would be the post-hoc attempt at usability testing.

Now, waterfall has become a myth, and it is the needfinding part that’s neglected, with UX focus on evaluative research (often, also, after the product has been built).

With the approach to user testing being “let’s launch the product and see if anyone buys it”, the thing that nobody wanted or needed is shipped in virtually unusable form, and then begins a frantic rush to try to tweak it into a format that works. Six-year cycles became one year, then, eventually, two weeks. Everyone is rushing around with no idea where they’re headed.

That frantic approach can hide a multitude of sins. AirBNB and Uber are hailed as innovations when they didn’t really invent anything: AirBNB started as a company to let out a spare room (my sister did that back in the 90s), which then quietly pivoted to be a holiday cottage rental company (an idea that The Beatles associated with old people). Uber started out as a ride-sharing company (which originated in the 1940s) and then quietly pivoted to be an unlicensed taxi firm: if your startup idea doesn’t work, just scrabble around for an older idea and call it new, after removing any safety checks that might have made it expensive.

Through frantic rushing around, Cyberpunk 2077 managed to claw its way back from disaster, but Mass Effect: Andromeda was not so lucky. Both were multi-year projects where each new part could only be developed once the previous part was done (i.e. waterfall) but with the rudderless, chaotic approach of a modern startup. EA takes things further with The Sims — sticking to the most recent game for a full decade and chucking out new content every few weeks, which is usually buggy, but eh, no worries, we’ll patch it later.

I’ve even seen this attitude on job adverts: “must be willing to ship something imperfect to refine through post-launch patches”. The point of a Definition of Done is to agree, in advance, that the thing ought to bloody work!

No checks, no balances

Risk management, like market research, is largely forgotten.

Last month, CrowdStrike’s outage is estimated to have cost firms $5 billion. Crowdstrike workers wear many hats — there’s no one user research function, nor a testing one, it’s just part of someone else’s job.

On July 19, a bug in CrowdStrike’s cloud-based testing system — specifically, the part that runs validation checks on new updates prior to release — ended up allowing the software to be pushed out “despite containing problematic content data.” To fix it, 8.5 million individual devices had to be manually reset. CrowdStrike has since introduced the kind of testing procedures that should have been present all along, including more control for customers over software updates.

Just a few days later, Microsoft Azure was brought down by a DDoS attack, which it then admitted had been exacerbated by “an error in the implementation” of the security protocols that were meant to defend it. Sean Wright, head of application security at Featurespace, said that the incident “highlights the importance of testing software thoroughly”.

Delta Airlines lashed out at both companies, saying that their negligence had cost it $500 million:

“If you’re going to have priority access to the Delta ecosystem in terms of technology, you’ve got to test this stuff,” Bastian said. “You can’t come into a mission critical 24/7 operation and tell us we have a bug. It doesn’t work.”

When risk registers and IT governance are no longer routinely part of the process, it just doesn’t get done properly. I saw a comment on LinkedIn the other day that, after laying off all the software architects, companies are scrabbling around trying to hire them again.

Enshittification — and plain bad service

Cory Doctorow coined the term “enshittification” in 2023, which the American Dialect Society deemed its Word of the Year.

To understand enshittification, we first need to understand customer centricity. According to Wharton’s Peter Fader, “customer centricity” isn’t simply being nice to customers, but understanding their desires and habits so thoroughly that you can work out what’s valuable to them, calculate how much they’re likely to spend, and orient your business around the most profitable customers.

Enshittification is corporate centricity — trapping the customer into a monopolistic system and then taking Porter’s Five Forces model and shafting everyone in that diagram.

Doctorow uses the example of Facebook: it locked end users in via network effects (you’re there because your friends are there), then locked in advertisers by selling user data for cheap ads, then locked in publishers with “recommended” content that was shown to users whether or not they wanted to see it, then squeezed those users and advertisers and publishers to return the money to shareholders. Now everyone has a miserable time — you don’t see your friends’ posts, advertisers pay more for ads that are viewed less, and publishers have their content held hostage: pay up, or we won’t show your content to anyone.

The word “content” says it all. Content. Not art, not education or thoughtful discussion, just vapid filler. In 2016, 64% of web traffic was non-human. In 2023, we talked about “sludge” or vapid sensationalist filler (the type responsible for last week’s riots). In 2024, a new word, “slop”, has been coined for vapid filler that has been generated by AI: much of the internet has been written by bots and is read by bots.

You couldn’t make it up.

Half the time, the makers of these “services” aren’t intentionally being malicious, they just genuinely don’t understand the needs of their users or how businesses are supposed to work.

A few days ago, The New Yorker ran a story about Spotify, and how its new user interface is attracting ire.

Diving deep into a particular artist’s discography requires scrolling through “Popular” tracks, “Artist Picks,” and “Popular Releases.” On Spotify, “the entire concept of an album feels more like a hindrance than anything.” Music on the app is most easily consumed in a disorganized cascade; every song becomes audio “content” separated from a musician’s larger body of work. In short, Spotify does not seem to care about your relationship to “your” music anymore; for long-term users, this has felt like a slow-motion bait and switch.

When the UI makes it harder to access albums, over time, they become “less important as units of online listening”. If Spotify succeeds at turning us all into passive listeners, then it doesn’t really matter which content the platform licenses. As design professor Jarrett Fuller put it, “It’s about ‘How do you get through as much music as you can so you keep paying for it?’”

Everything is about “engagement”, but not so much “enjoyment”.

Milton Friedman has a lot to answer for

Of course, we shouldn’t be surprised by this — for at least two hundred years, a pendulum has swung back and forth between “profit at all costs” and “hey guys, maybe we should stop being such assholes”. Adam Smith’s factory management theories gave way to Keynesian safety nets, which in turn led to Reaganomics.

In 1970, economist Milton Friedman declared that “the social responsibility of a business is to increase its profits”. By 1984, R Edward Freeman had countered this with Stakeholder Theory — the idea that a business is only sustainable long term if every stakeholder feels like they’re getting a good deal.

Exemplifying this approach are B Corp companies — for-profit companies who meet certain social and societal standards. Examples include Ben & Jerry’s ice-cream, Patagonia apparel, and Charlie Bighams foods.

These companies run counter to the prevailing economic trend, that of laissez-faire capitalism, which spread under the tenure of Chairman of the Federal Reserve, Alan Greenspan. Greenspan was a disciple of Atlas Shrugged author Ayn Rand (darling of both US Conservatives and the Church of Satan) whose philosophy is basically, “screw you guys, I’m going home”.

Under Greenspan, loans were irresponsibly foisted upon people who could ill afford them until the whole system collapsed in 2008, after which the government bailed out the banks and implemented austerity measures on everyone else because, well, screw you guys.

Under this system, wages have been suppressed and America’s 1% have siphoned $50 trillion from the bottom 90%, trapping people into jobs where they work more than almost everyone else in the world despite having lower productivity than almost everyone else because, after more than about 50 hours of work, you’re just too exhausted to do a good job. (Most European countries have settled on around 35 hours for optimal performance.)

So, how is all this killing companies?

Companies don’t lay off tens of thousands of staff because they’re doing just great.

Eventbrite just laid off 11% of its staff in addition to the 8% it laid off last year. It posted $85 million in revenue this quarter, an increase, but posted a 6% decline in ticket sales. In other words, revenue was not the most important indicator.

Peloton laid off 400 people following worse-than-expected figures since the world “returned to normality” after a pandemic-related home exercise boom.

Dell, like many other companies, is laying off workers to rush to AI in a move that most people would call “really stupid”.

I once sat in a user interview where a very eager engineer showed off a new AI tool that hallucinated in the first attempt in the demo, to the bemusement of the customer. It is not good enough, and companies are entrusting it with their futures.

The poor performance of AI is disappointing to companies that saw it as a panacea to their problem of hiring workers — the thing they’d do almost anything to avoid doing — because tech companies have not yet realised, unlike sports teams or almost anyone else, that the only real value they have to anyone, anywhere, is tied up in the people they employ.

We call them “dependencies” or “resources”, but what we mean is “people”. We know Shaquille O’Neal is valuable, we know that Tom Cruise is valuable, but we seem shocked when we find out that the only value that our company holds to consumers is in the quality of people who work there.

When you’re exhausted and scared of losing your job, it’s hard to work to an optimal standard.

An inevitable result is the decline in customer service: almost half of customers report that they feel customer service has got worse in the past three years and Qualtrics estimates the cost to companies of bad customer experience as around $3.7 trillion.

So, what can we do?

Traditional waterfall approaches did not consider that user needs might change, or that it was impossible to have all of the real requirements upfront.

Agile frameworks don’t typically consider that any business that survives beyond a year will have more than 5 or even 50 employees, and that the very reason specialist roles exist is because generalists are not good enough at any of those roles to do it well enough to meet the expectations of enterprise customers.

None of the frameworks, new or old, are fit for purpose in the modern age. We need something else, something not seen before, which acknowledges both the work that Porter et al did in mapping out corporate dynamics and acknowledges that plans will (and need to) change at very short notice.

We need a system which acknowledges: this is what we need to know about customers and competitors; this is what we need to understand about our own capabilities, and this is what we need to understand about what it takes to develop something wholly new and genuinely useful within this space.

If Scrum only worked for as long as there were waterfall systems in place to support it, we need to replace both with something that both acknowledges and improves that reality.

Something that provides a win-win-win for all the stakeholders in the equation, rather than ruthlessly exploiting everyone and making them miserable.

Something that acknowledges the value of the humans in the equation, and the systems that they need to have in place to do their best work fearlessly.

We don’t seem to have this right now, but I’m far from the first to call for it.

We changed the world before. We can do it again.