What is a Lean Enterprise?
This is my definition of a Lean Enterprise:
“A Lean Enterprise is an organisation that allows the business to continuously learn new and better ways to deliver value by validating hypotheses using a rigorous scientific approach.”
There is a lot to it, so let’s examine.
A Lean Enterprise enables the business to identify valuable ideas fast. A Lean Enterprise is different from more traditional organisations (e.g. pathological and/or bureaucratic) because the whole organisation is tuned towards delivering business value. The products to build (portfolio management) are chosen based on gathered facts and data, rather than more ego-driven approaches, i.e. the Highest Paid Person Opinion (HYPPO) principle or gut feeling. In the majority of large enterprises, only about 25% of projects is decided based on some financial data.
In a Lean Enterprise, each business value hypothesis is quickly validated in production and the business adapts its strategy based on the feedback received by the end users.
The business grows through a series of incremental, calculated experiments, each taking a product (and the company at large) from one target condition to the next.
The key question is: how do we enable the business to operate in such environment?
The answer is: culture, people, processes and tools.
- The Organisation must have a culture geared towards enabling the business to quickly identify valuable ideas to invest on through the scientific method. The side effects of reaching this outcome are that as a consequence, the organisation lives the right culture of leadership / followership, trust, respect and mission to deliver value. There is no room for ego-driven decisions, conjectures, bias and so on.
- The Organisation must be structured in self-organising teams, empowered to take decisions within the culture and general direction boundaries and ultimately responsible for the organisation success
- Technology must build the road that the business will use to rapidly validate hypotheses and adapt its strategy based on the observation of data and numbers. To do so, apart from being deeply rooted in the aspects just mentioned above, technology must use the right tools.
- A Lean Enterprise doesn’t distinguish between the business and technology: there simply are people with different skills, all geared towards the delivery of business value faster, better and cheaper.
The Four Pillars of Lean Enterprise Technology execution
While in India on a business trip, my friend John Ferguson Smart and I came up with a vision that represents the four pillars of Lean Enterprise Technology execution.
We believe these are:
- Behaviour Driven Development (BDD)
- Test Automation
Moreover, we identified four outcomes when looking at the four pillars in a clock-wise flow.
The Outcomes derived from the Four Pillars of Lean Enterprise Technology Execution
When looking at the Four Pillars of Lean Enterprise Technology Execution in clockwise order, we can see four outcomes:
- When combining BDD and DevOps we deliver value sooner.
- When combining DevOps and Test Automation we can deliver Quality at speed.
- When combining Test Automation and Agility we can adapt with confidence and finally
- When combining Agility with BDD we can discover value effectively
When combining the four pillars together we believe we have built the road for the business to continuously validate their business value hypotheses, therefore laying the foundation for a Lean Enterprise operating model.
Let’s now look at the four pillars in a bit more detail so that we establish a common language.
Behaviour Driven Development (BDD)
Plain and simply, BDD is a requirements discovery technique. Whoever tells you otherwise doesn’t know BDD. Many will say that BDD is a specific tool (like Cucumber JVM or JBehave or Selenium), or that BDD is Given/When/Then.
BDD is an operating model that stimulates and requires the business, technology and people with testing experience (Los Tres Amigos) to collaborate, interact and discuss in order to define the right thing to build, through executable acceptance criteria.
To implement BDD, organisations might choose different tools and there are techniques (like the Gherkin format Given/When/Then) that allow the capturing of requirements using a format very close to the natural language, while at the same time allowing the automated validation of those requirements.
BDD should be used before starting to code. If you’re using Scrum, BDD should be used as a design tool before entering Spring Planning. If you are using the Improvement Kata, BDD should be used in the execution cycle, when defining the expectation of each PDCA cycle.
BDD is what allows teams to build the right thing, rather than verifying that the thing they’ve built is right. Because of what BDD is, this operating model leads to very healthy practices when it comes to requirements engineering, e.g. the delivery of business value through an incremental series of Minimal Viable Products (MVPs) or what I call Minimal Enjoyable Products (MEPs).
DevOps, like Agile, is a term that has been and is being abused by various practitioners, particularly those who are new to it, and those who are trying to make a profit out of it.
It’s enough to search for the word DevOps online to find all sorts of interpretations. There are even companies that provide DevOps courses and certifications, as if DevOps was an exact science that could be taught with right or wrong rules and precise execution models.
DevOps is a term that defines the close collaboration between Development and Operations in order to provide the business with the avenue required to validate hypotheses @ speed @ high quality. It does so by automating whatever is possible. This means Continuous Delivery but also Infrastructure Provisioning, Configuration Management and Monitoring. It means attention to operation details when developers write code and the employment of development techniques and practices when managing operations.
Continuous Delivery is the ability to automatically deliver technology artefacts into production at the fastest pace (in some organisations, like Amazon, this means thousands or releases into production every day) with the highest quality. This is only possible if:
- Infrastructure is managed as a commodity, i.e. automated provisioning, tear up and down and configuration
- There is an Automated Delivery Pipeline, through which artefacts flow from left to right
- Every artifact flowing through the Delivery Pipeline is automatically validated to be what the business wants (thanks to a BDD operating model)
- All environments are functionally and architecturally identical
- There are no manual barriers between a line of code being committed to an SCM tool and the same line of code being deployed to production
- The artefacts flowing from left to right in the Delivery Pipeline are continuously and automatically tested
- The Continuous Delivery pipeline offers quality gates to safeguard operational risk. These can be automated (when the organisation is mature) or manual (such as pressing a button to deploy an artefact to the next environment), depending also on the organisation’s risk appetite and the MVP’s risk profile.
- Each artefact is physically decoupled from any other artefact and the only mean of communication between components is via (RESTful) interfaces.
- Each change should have the concept of a “Limited Blast Radius”, meaning that if something goes wrong, generally is not catastrophic. If we apply the Spotify model (more on this later) with self-empowered and independent teams, generally the team responsible for the failed release can recover pretty quickly thus the damage is limited.
- Additionally, the underlying architecture and tools should allow for 100% uptime whenever possible using techniques such as green/blue deployments and A/B testing
From the above is clear that the following are enablers for a Continuous Delivery operating model:
- Cloud (whether internal or external). Please note that internal Clouds are subject to finite capacity and therefore introduce a potential impediment to Continuous Delivery.
- Configuration Management, i.e. the ability to automatically set up hardware with the software and infrastructure artefacts required to run an application. Puppet, Ansible, Salt Stack and Chef are all examples of configuration management tools
- Product Teams, each one responsible for one or more components. Each team lives and breathes the “You build it you run it” culture. Each team is fully responsible for the whole MVP lifecycle, from defining it through BDD to implementing, deploying and maintaining it.
- Microservices or Containers. A Microservice architecture is the perfect landing place for an independent component to run and communicate with other components through interfaces, using widely open and known protocols (e.g. HTTP)
- A Continuous Integration tool, like Jenkins, that runs the Delivery Pipeline
- A Release Management operating model that allows artefacts to automatically flow from left to right, without human intervention. This doesn’t mean that artefacts will be deployed into production blindly; a Delivery Pipeline should still provide quality gates that guarantee the quality of what a team is delivering.
- A log management tool, like Splunk, that is capable of aggregating logs from multiple running instances of a component and present them as coming from a single instance
- The adoption of the 12 factors in each component’s architecture.
- The adoption of BDD as requirements discovering technique
- The adoption of Testing Automation (TDD) as a way to automatically validate that the component is right (what the business wants)
- Modern monitoring tools that can detect when a component instance is down and isolate it from the live service and conversely detect when an instance becomes available and add it to the pool of component instances offered into production. Monitoring tools should also automatically detect when any of the infrastructure required to run the component has issues and either take automatic steps to correct the fault or notify the operations team proactively (The Circuit Breaker Pattern). Monitoring tools should also be used to gather facts and data about each production deployment so as to offer the business the tools to validate their business value hypotheses and adapt their strategy.
Testing automation is the ability to automatically run one or more test suites to provide various feedbacks, e.g. whether the code quality is good or whether the team is building the right product. Generally we use TDD as a technique to run automated test suites, whether these are unit, integration, performance or BDD tests.
Agility means the ability to quickly adapt and act depending on gathered facts and data and to remove obstacles that stand in the way of reaching our goals. These can be around our way of working, feedback from production users once a component is deployed to production, a change in regulation or market condition and so on. Agility is a word, Agile is not. Agile is an adjective. So one might seek Agility and become Agile, but one cannot do Agile. Agile is not a thing.
Most refer to Agile as the mind-set resulting from adopting the Agile Manifesto. I personally think that the Agile Manifesto is a great document for developers wanting to move away from a Waterfall-like culture and way of working but it’s widely inadequate for a Lean Enterprise. I wrote more about why I think this is the case in one of my previous articles.
So a Lean Enterprise is Agile because it can adapt quickly to continuously improve its way of working and be competitive. This means that if you’re seeking Agility, by building a Lean Enterprise you will get that for free.
Which methodology should we be using to evolve organisations into Lean Enterprises?
Now that we have defined the four Lean Enterprise pillars with regards to technology we can define how these fit with the rest of a Lean Enterprise organisation.
From the Theory of Constraints, we learn that a chain is only as strong as its weakest link and that a system is only as fast as its slowest part. This means that in order to accelerate we need to elevate the weakest link in our chain. With regards to a Lean Enterprise, we need to identify the biggest bottleneck and break it; then move to the next one and so on, being careful not to slip into inertia, i.e. the feeling that we are now OK.
If we see a Lean Enterprise as a system and we are able to map the flow of value from left to right, we can identify the bottlenecks and resolve them.
The question is: how do we remove them? This is where a methodology generally comes in. Some will decide to use Scrum, others some sort of “Scaled Agile” methodology, e.g. SAFE. I believe that all of these so called “Agile” methodologies are just marketing tools and they contributed to the death of Agile as a principle and a mind-set. There is plenty of material that illustrates in depth what I’m saying. Here are some examples:
- Agile is Dead
- The Death of Agile
- Why Agile doesn’t scale and what you can do about it
- The Land that Scrum Forgot (thank you Peter Merel for the link)
The Improvement / Coaching Kata as a tool to grow Lean Enterprise Organisations
The Improvement and Coaching Kata (ICK) is an organisational approach to delivering complex changes. Apart from applying the scientific method to the delivery of incrementally more effective ways of working through a series of PDCA cycles, the ICK allows organisations to grow managers as leaders and embed a culture of leadership and followership, permeated by the right approach to feedback.
The ICK allows an organisation to mathematically link any activity up and down the organisation.
With the ICK, Leaders are also Followers, depending at which level they sit within the organisation.
The ICK is based on a flow that is very simple to understand but difficult to master:
- Understand the direction
- Grasp the Current Condition
- Establish the Next Target Condition
- Iterate Toward the Next Target Condition
Steps 1-3 are the Planning Phase, Step 4 is the Execution phase.
If we use the ICK across the entire organisation, then we can mathematically link all activities up and down the organisation, from the CEO to the receptionist.
The fundamental concept when using the ICK across the entire organisation is that one’s direction (or challenge) is the Next Target Condition from the level above as illustrated in the following picture.
Now I believe that if we use the ICK across all levels of the organisation, we can transform pathological and bureaucratic organisations is generative ones, i.e. a series of Lean Enterprises.
The other advantage of successfully implementing the ICK across an organisation is that it leads to a Lean Enterprise culture where the whole organisation is tuned towards allowing the business to quickly identify that 5-10% of ideas that actually generate value and invest on them and discard (or pivot) the other 90%.
This is not just a technology problem, which is why I find Lean Enterprise transformations so fascinating. Being able to separate the winning ideas from the ones producing no value or negative value is the differentiator between the companies that will be successful tomorrow and those who will go out of business.
Nokia and Kodak are examples of companies that, although market leaders in their golden age, failed to innovate and cannibalise themselves, didn’t listen to data and therefore didn’t adapt. The moment the iPhone reached the market, Nokia was dead and similarly the moment Kodak decided to keep its digital cameras on the shelf to protect the film industry they condemned themselves.
It’s companies like Amazon, Google, Netflix and Spotify that keep cannibalising themselves (by that I mean disrupting their core business to continuously improve). They all have one thing in common: they are Lean Enterprises.
How do we organise people in a Lean Enterprise?
In pathological or bureaucratic organisations, there’s typically a hierarchical structure. We will have departments which in turn will have “thematic” themes, e.g. products. Depending on the size of a thematic team, this might be decomposed in Product teams. In organisations that have adopted an agile structure, such as Scrum, XP, Kanban and so on, each product team might be divided into cross-functional teams, e.g. Scrum Teams.
Such organisations are typically driven by an ego-driven culture, where the person at the top is better than anyone else and commands all decisions (the HYPPO principle we have discussed earlier). Success is determined by the number of people if one’s line management and how much one earns.
In such organisations, budget management is centralised and typically rolled out in a Waterfall-like structure, where the COO and CFO allocate money depending on the CEO strategy and pots of money trickle down the organisation down to Product team level, where the Product Team Lead (generally an SVP / Director level) needs to decide how to allocate them. Being a Waterfall approach, all decisions about how much money each team gets need to be made up-front. This results in all budget-receiving candidate teams to go through the budget planning death-march, i.e. they’ve been asked to transform from engineers in magicians and instantly develop the gift of predicting, to the very date and to the very last person in the team, all projects they will be working on, how many people they’ll need, when it will get done, what value it’ll deliver and so on. This is known as “capacity planning” or “budget cycle” and when rolled out in a Waterfall-like style, it’s likely to have a 100% failure rate. By that I mean that 100% of the projects will not deliver exactly what was budgeted, at exact the dates that they’ve advertising to get the money, with exactly the money that were asked and delivering exactly the value that they had promised.
A Waterfall-like budget-cycle also leads to the wrong behaviour: middle-management, i.e. the people asking for budget, will need to demonstrate that they needed that money otherwise they risk that in the next budget-cycle they won’t be given the same amount. So it’s not atypical seeing a “spending frenzy” activity towards the end of the budget-cycle, which can materialise in training courses, new vanity projects, new equipment, new software and so on.
So what do we need?
A Lean Enterprise manages money differently: yes, there is a direction that senior executives provide, e.g. “We will become the best digital bank in the world” and there is a pot of money available but the money doesn’t get allocated all up-front, based on crystal-ball plans. Money gets allocated to either experiment hypotheses to find the best 5-10% ideas or, once these are found, money is spent in making of these ideas the best possible products. This way of working requires the whole organisation, from the CEO to the receptionist, to be aligned within the same strategy, to understand the direction and how each layer in the organisation can contribute towards the strategy execution. This is why the ICK described above is such a powerful tool: it allows such alignment and chain of activities to be mathematically linked between various levels in the organisation.
So rather than a Product Team Lead asking for £10,546,780 to rollout 4 projects with 47 people, such and such hardware (down to CPU cycles details), such and such software and so on, the responsibility starts with the business that needs to think differently at the way they deliver value.
People in the business will have hypothesis to generate value, e.g. “I think that if we deliver this functionality in the next two-three months, we can increase our customer base by 30%”. Then it’s the job of the Technology team to allow the business to validate this hypothesis in the shortest possible time, possibly by implementing the four pillars of Lean Enterprise Execution as described above. Once the functionality (MVP) is in production then technology and the business should work together to gather data on how that MVP is actually performing and compare what actually happened with the expectations. Then one of three things will happen:
- The hypothesis has been validated, therefore this enters the list of ideas to invest on
- This was a really bad idea, let’s just kill it
- The idea is pivoted into an alternative MVP that might generate some value
In such operating model (the Lean Enterprise operating model), money is allocated at two different stages: to allow the business to validate business value hypothesis and to seriously invest in the winning ideas.
To allow the business to identify highly valuable ideas through the scientific method (Plan-Do-Check-Act), the organisation must have the right structure and the right culture: I believe that such structure is very similar to the Spotify model, of which I show a representation below (the image is free for you to reuse. I’d be nice if you could attribute it to me if you’re going to use it).
With this model, an organisation is broken down into a series of Lean Enterprises, where each one is fully empowered to choose how to deliver value within the organisation’s boundaries, i.e. culture, guidelines and principles. A Tribe is a “Product Team” as described above and it’s comprised of a number of “Squads”, each fully independent and fully empowered to deliver value. Each squad will work with the business to validate a business value hypothesis (MVP) and through the scientific method will iterate through continuous delivery cycles until one of the three outcomes described above happens. This structure means a number of important things:
- A Squad is fully empowered to take any decision that is necessary to support the business in delivering value. This means that they are empowered to decide which technology, architecture, methodology and so on to choose in order to achieve their goal
- The Four pillars of Lean Enterprise technology execution are enablers for such structure
- DevOps practices are required in order to accelerate the Time-To-Market cycle, through a continuous inspect and adapt cycle
- There must be a culture of automated risk management (not avoided risk management), where comprehensive test suites and an outcome-driven approach to regulations must permeate teams activities
- Since for this structure to be productive, speed is essential (enabled by DevOps), the whole organisation must be tuned towards allowing a rapid cycle of MVPs from development to production. This touches on areas such as Access, Release, Risk and Portfolio management.
- Finally, the business must have the right culture, which is about deciding on the portfolio based on facts and data gathered by the MVP cycles into production, rather than ego, power and position within the company.
Across Squads there are a couple of structures which might vary depending on the organisation:
- Chapters. This are cross-Squad teams of similar skills, e.g. web or Java developers, DBAs, middleware developers and so on. Typically, one of the people in a Chapter will be the line manager for the Chapter but not in the traditional term. This person will act more as a servant-leader and a coach
- Guilds: A Guild is a community of practice / interest across Tribes and its purpose is to continuously define and share ways to improve.
This kind of structure has got an effect also on Performance Management. As the unit of business value delivery is now a Squad rather than a single person, targets and performance must be set at the Squad level. This demands a heavy involvement of HR in the design of such organisation and in the definition of what good looks like.
Finally, this structure requires the proper architecture. Each Squad should produce a mini “Product” (MVP) that is completely independent from any other. The only way two products should interact is via RESTful APIs. Therefore the correct architectures to support this organisational structure are Microservices and Containerisation.
The moving from a Monolith to a Microservice architecture, Jeff Bezoz addressed his team with an email that contained the following content:
- All teams will henceforth expose their data and functionality through service interfaces.
- Teams must communicate with each other through these interfaces.
- There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
- It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter. Bezos doesn’t care.
- All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.
- Anyone who doesn’t do this will be fired.
- Thank you; have a nice day!
Today Amazon releases some 10k times /day into production. Google, Netflix and Spotify all have similar metrics.
To know more on the Spotify model, you can watch the following Youtube videos: