Case Study: Using Agile to Improve Productivity by 240%

Table of Contents

Introduction

This is a case study of a two-year project with an actual client. Over the period, we iterated through numerous improvement evolutions, experimenting with various concepts and processes. At the two-year mark, we measured the following improvements:

Productivity increase of 240%
Reduce Product Release Efforts by 89%
Reduce Lead time by 73%
Reduce Rework Rate by 74%

This article is a detailed exploration of our journey with the client.

Client Profile

Industry: Fintech | Artificial Intelligence
Product Portfolio: Financial Crime (AML Suite)
Target clients: Large Financial Services Clients
Headcount: 150 – 750
Locations: US, Asia, Europe
Age: 12 years
State: Rapid growth

The client had seen a sudden success over the preceding three years, as their maturing product suite had started garnering adoption by the large banks, and they needed to start scaling operations rapidly to keep up with demand.

They were in the middle of negotiating a $50m deal with one of the largest banks in Asia ($400bn market cap) to adopt their Anti-Money-Laundering platform; however, they had significant functionality requirements, which would have to be developed rapidly to achieve very aggressive timelines.

The company had also increasingly been experiencing various problems in its operations, such as a reduction in quality, inability to meet delivery commitments, and reduction in throughput and overall productivity. Employee engagement was also trending downward, and the first signs of decreasing customer satisfaction were starting to show, threatening their reputation in the market.

Our Engagement Model

Our engagement model follows an eleven-step iterative approach, supported and enabled by our supporting services.

Engagement Process

Establish the “Reason for Change”: Before embarking on a change project with a client, there has to be a real business reason for the change. Adopting Agile is not an end in itself. It is a means to an end – something you do to achieve a specific result.
Assess the system & Co-create the change strategy.
- Assess the system: Rather than performing an assessment from an external perspective and presenting the results to the client with an implementation plan, we perform a collaborative evaluation, working with our clients to uncover the problems jointly and to form a shared understanding of the status quo. Clients will only initiate a change strategy if it makes sense to them, and for it to make sense, we need to have a shared understanding of what is going on.
- Co-create the change strategy: Combining our expertise in Lean & Agile Methodologies, training, mentoring, and coaching practices, and the client’s understanding of their business, we co-create the change strategy, identifying the high-level initiatives and who would lead the effort. We co-create the desired outcomes. This also supports “buy-in” from the client. With their buy-in, the change strategy has a higher probability of success.
Establish Psychological Safety: Change is complex, and effective change requires honest, possibly difficult discussions, so establishing a safe space supports this initiative.
Decide who to include: As we are using an iterative approach, we work with teams of people on defined improvement experiments. Deciding who would be part of the change process is also pivotal because they will be part of designing the improvement experiments. Just as with the change strategy, including the people who are part of the change, designing the change is a critical element supporting the success of each experiment.
Start with now: It is essential to understand the current state of things because any change we implement should consider all the lessons learned to date and incorporate and improve the existing processes.
Define a goal: We co-create tangible goals with the selected team that is part of the change. This ensures we leverage their institutional knowledge and achieve buy-in for the upcoming experiment. The goal is expressed as a “research hypothesis,” including a measurable outcome, leading and lagging indicators, and when, where, and how long the experiment would run. Good experiments are short, low-risk, low-cost, and deliver high-quality information or results.
Make the change: Once everyone knows the goal and metrics, the experiment is implemented and monitored. Early signals are used to pivot or persevere.
Measure the impact and learn: At the end of the experiment, the results and effects are measured, and we use the results in a learning cycle.
Persevere: Some experiments are discarded, and others are adopted depending on the specific results.
Update systems: The adopted experiments update the “organizational system,” including processes, policies, rules, roles, responsibilities, templates, standards, etc.
Cultural change: As the organization achieves multiple rounds of successful experiments and the difference is rolled out to more and more teams, the culture will change over time.

Supporting Services

Assess: We use various tools and processes to help organizations and teams “make sense” of their environment to drive effective change.
Systems coaching: We coach organizational systems, considering all system parts, including people, leaders, processes, organizational structure, relationships, etc., to achieve long-term, lasting impacts.
Facilitation: We use skilled and professional facilitators to create meaningful interactions between groups of leaders and delivery teams.
Training: We deliver role and context-specific, tailored training programs to teams, leaders, and non-tech business units, as well as certifications in several Agile disciplines and frameworks.
Team Coaching: We coach leadership or delivery teams on how to function better, learn better, and have more fulfilling and productive work engagements.
Individual Coaching: We offer professional coaching to individuals to help them achieve their personal goals and become experts in their specific roles in the organization.
Change Management: We drive change initiatives across the organization more broadly to ensure there is institutional support for transformation projects.
Leadership Development: We work with leaders, enabling them to support their change and adopt Agile working methods in their specific context as leaders of the organization and teams.

The Reason for Change

The client was building a healthy pipeline of large deals; however, they were starting to experience problems delivering to their existing clients, and other organizational issues were starting to pop up. We identified six specific symptoms that supported the reason for change as follows:

Product Quality: The quality of the product seemed to be deteriorating over time, with an increasing number of defects released into production and the levels of rework increasing across the entire value stream.
Delivery Predictability: Committed delivery dates were getting missed more frequently, and the gaps between expected and actual delivery dates increased.
Speed: The pace of individual features moving through the value stream slowed significantly over time.
Productivity: A decreasing amount of team capacity was spent on actual “productive” work, as teams were “fighting fires,” engaged with rework, or replanning missed deadlines.
Customer satisfaction: There were early signs of decreasing customer satisfaction, which threatened their pipeline of deals, as it became increasingly risky to use their existing customer base as client references.
Employee engagement: As pressure on the teams increased, the general well-being of employees was decreasing and trending downward, with talent disengaging and staff turnover increasing.

Assessing the System

Cause and Effect Mapping

We used a process of cause and effect mapping with the leadership and delivery teams to understand how the business problems interacted and to better understand the dynamics and root causes.

Looking at the resulting cause and effect diagram, we could draw several conclusions as follows:

Quality: was central to the cause and effect dynamics, in a poor state and also in rapid decline
Market responsiveness and Innovation: seemed to be at the root of what happened. The company was so dynamic and market responsive that it struggled to keep up with the expectations being set, and the tendency to innovate and be market responsive was continually increasing as more and more opportunities were opening up.
Continuous Improvement: Was not being leveraged and was, in fact, in decline. However, turning this around would significantly impact the entire system.

Agility Insights Diagnostic

We performed an Agility Insights Diagnostic through interviews and surveys throughout the organization to get a big-picture systemic view of the company’s functioning.

Using the assessment framework, we picked up three strong themes, and they are:

Leaders could not access high-quality information, inhibiting their understanding of what was happening. As a result, the company could not create a shared understanding, directly impacting their ability to act appropriately (agility).
The company lacked many of the capabilities required to deliver on its objectives, the talent pool lacked the motivation to get engaged in the work, and as a result, the overall “creativity” in the delivery space was severely lacking.
Due to a combination of the above (lack of information and capabilities), they could not drive the successful implementation of their goals.

Continuous Delivery Assessment

We performed a Continuous Delivery Assessment using the DevOps Health Radar, which showed an organization quite early in adopting continuous delivery practices. The delivery teams were aware that they had to implement better engineering practices. However, no capacity was assigned to these activities, thus unable to implement their desired improvements.

Continuous Exploration: they were very good at identifying market opportunities and developing new product hypotheses. However, these were not being transmitted to the engineering teams in a way that empowered them to deliver on these. They were also lacking the competency to prioritize work, and delivery was done in “big batch” mode (large products and specifications were being driven through to delivery teams) while everything was urgent at the same time.
Continuous Integration: The engineering team was functioning in a manual and fragmented fashion, organized into functional silos (front-end, back-end, data, research, development, etc.). There is a very low throughput of work generally and a lack of engineering standards implemented.
Continuous Deployment, Release on Demand: Stage, deploy, and release was essentially the same activity and quality (through manual testing) was done “after the fact,” resulting in a high rate of rework and defects released into the client’s production environments.

Co-designed Change Strategy

Working with leaders and teams, we co-designed the change strategy.

Objectives

We set up seven change objectives to drive the strategy, and they were:

Make Optimal Economic Decisions: ensuring that the highest “economic” value work items were prioritized for delivery teams, using a scientific, unambiguous prioritization technique.
Improve Delivery Quality: implement quality practices across the value stream (from Continuous Exploration to Release on Demand) to reduce rework and improve throughput and productivity.
Faster Sustainable Value Delivery: using Lean and Agile principles to manage workload, limit work in progress, eliminate waste, address bottlenecks, and implement automation.
Experimentation and fast feedback: actively continuously improve their product and work methods – a previously neglected area.
Retain top performers: they were losing their top performers due to the dynamics, and a specific plan was put in place to retain top talent.
Open, honest communication meant creating a psychologically safe space throughout the organization to encourage a “fail fast” mentality and a Lean-Agile mindset.
Lead by example: meant that the organization’s leadership should take charge of the transformation rather than expecting delivery teams to change, while the supporting organization maintained the status quo.

Work Streams

Around each objective was created a workstream, and work streams were clustered and assigned to work groups – four in total.

Planning and dependency mapping

The working groups took ownership of planning out the implementation of the change program and mapping the interdependencies between activities from different working groups.

Change Plan

With the leadership, we developed a change plan, split into four work streams. Each workstream had an assigned working group and iteratively developed a rollout schedule.

Leadership Track: We created and delivered a leadership development program, starting on the executive level and then rolling it out to departments and teams. We also coached and mentored leaders in specific disciplines and delivered ad-hoc training programs as required.
People and culture track: We set up an internal coaching community of practice, where we trained internal coaches, shared lessons learned, and collaborated on delivering the overall change program. We also centrally designed standardized training programs and delivered these in a coordinated fashion.
Relentless Improvement track: We set up an iterative process improvement practice. Teams would regularly reflect, typically through facilitated Kaizen workshops, and identify sets of new improvement experiments. We would co-create the improvement hypotheses, set leading and lagging indicators, and assist teams in monitoring these improvement experiments through their delivery. At the end of each experiment, we would analyze the results, successful experiments would be adopted into the company’s ways of working, and others would be discarded after extracting lessons learned.
Flow track: there was a specific track dedicated to improving the flow of value through the organization, including breaking down organizational silos, working on the reduction of release costs, quality initiatives, and overall delivery of value and productivity, using the same iterative improvement cycle as mentioned in the “relentless improvement” track.

Establish Psychological Safety

After designing the change strategy, the next phase is establishing psychological safety, which forms the foundation of any change program. Upon initiation, this process is repeated with each team and working group, including the leadership team.

Psychological safety is established through a facilitated process, where the team establishes their individual and team goals, rules of engagement, “working agreements,” and team protocols.

Select Pilot Teams

The change program took an “opt-in” approach, inviting individuals to join without expectation. Once the team was formed, they performed a self-assessment to understand their internal group dynamics and take stock of their practices, behaviors, and the organization supporting them.

Start with now

As part of “Forming” a new team and establishing their collective goals, they must understand their starting position – what we refer to as “start with now.”.

Value Stream Mapping – 95% rework rate

The teams typically mapped their value streams through a facilitated collaborative process improvement program. Then they estimated the average lead time, cycle time, wait time, and rework rates, so they could hone in on specific areas they would like to focus on to run their process improvement experiments.

Root Cause Analysis

Once specific focus areas are identified, teams use tools like cause & effect mapping, 5 Whys, and Ishikawa diagrams to understand the underlying reasons to make sense of the symptoms they are detecting.

Identified 74% Waste

Identifying significant waste areas within organizations is common, especially at the early stages of a change program. This is often the result of leadership “measuring the wrong things.”. The below example is a case in point.

Release activities: in these teams, the organization did not measure the cost of release activities. Due to the frequency of their releases, the complexity of the product, and the manual nature of the work, they invested a significant amount of their teams’ productive capacity just into “shipping products” to customers.
Defects/rework: due to a high rate of defects and rework slipping into the release process, a significant amount of time was spent, under pressure, to fix defects trying to ship products. To exacerbate this problem, defects were often being fixed in their staging environments and not in their development environments, and often these fixes would not be rolled back. Thus the same defects would often re-enter the release process multiple times. Or, “fixes,” then rolled back to development environments, would create new defects, and thus the new fixes would have to be rolled out to existing clients upon the next release batch. This had a compounding effect on the complexity of the release processes.
Buffers: there was significant pressure on clients to meet strict delivery deadlines, and these timelines would be passed onto development teams. To ensure they would make their timelines, it was common practice to add buffers into their time estimate (the official policy was to add a 25% buffer to every estimate). These buffers typically got consumed by work and exceeded. Thus, in effect, this was “lost capacity” (or a “commitment tax”) which the organization had no direct control over. We often ask our clients to do a mental exercise – if you take your entire engineering capacity (let’s assume you have 100 engineers), what is the cost of a 20% buffer of your engineering department? One hundred engineers x $75,000 annual salary equals a $1.5m “commitment tax” per annum.
Recruitment: There is a continuous recruitment flow in fast-growing companies or companies with staff turnover (which is all companies). Even with highly optimized recruitment processes, some engineering capacity would be invested in recruiting new engineers.

After performing all the calculations, this company, in this scenario, could only dedicate 26% of their entire engineering team’s capacity to productive work (delivering value to customers), and 74% of capacity was going to waste.

Define the Goal, Make the Change

Working iteratively with teams and leaders, we identified hundreds of change hypotheses over two years. Many of them were adopted, and many were not. Below is a summary of some successful initiatives that were ultimately adopted after proving their value.

Implementing Agile Requirements Management

Product Requirements Documents and Silo Mentality

One of the first changes we implemented was a scalable Agile Requirements Model. Up to that time, the company developed traditional “PRDs” (Product Requirements Documents), which were very large and detailed specification documents describing big pieces of functionality that had to be delivered simultaneously. The “PRDs” were typically developed over extended periods without technical input and then “shipped” to the engineering department for delivery.

Due to the engineering team’s constant pressure, the “easy way out” would be to reject the PRDs and send them back to product management, requesting various clarifications and elaborations. The engineering department would eventually set up an “audit committee” to “audit” (critique) PRDs being sent to them. This indicates how deeply the “silo” mentality had become embedded in the company’s culture. It was nearly impossible to get any requirements through to the engineering team, as they were constantly rejecting new requirements while trying to get to grips with the work they had in progress.

Facilitating Collaborative Agile Requirements Management

Starting with a single team, we created a “horizontal” structure, organizing the team along the value stream with a much tighter connection between the Product and Engineering functions. The Product Manager would start engaging with the team’s lead engineers much earlier in the definition process, getting their inputs on the product design, not only from a technical perspective but from a functional perspective too.

Requirements were developed iteratively over time, and by the team, the functionality was ready to be built; many of the team had already developed a deep understanding of the work coming their way – thus, they were not solely relying on documentation to “make sense” of the development work.

The documentation focused on documenting the most important parts of the product only, such as the user flows, the UX designs, business rules, and acceptance criteria. “Quality” team members were part of the process from the start rather than being sent code and test cases to test on the last day of the sprint.

Using the combined creativity of the Product Owner and the team, they collectively delivered a higher quality, more valuable product at a more predictable cadence.

Delivering Learning and Development Programs

Throughout the transformation project, we compiled 48 course modules organized into five learning and development tracks. Different stakeholder groups were taken through various tracks over time. All training sessions were recorded and added to the company’s internal training library.

The training was then supported with hands-on coaching and facilitation at all levels of the organization.

Reducing Delivery Batch Sizes

Rather than designing, documenting, and delivering products or features as “big batches” of work, the product team was trained to define their requirements in smaller “vertical” slices. This meant the team could deliver value in smaller pieces but, most importantly, on a more frequent cadence. The “Lead Time” for a piece of value was reduced from 4 months to 4-7 days. Typically the product team had to wait for approximately four months before they could see the delivery of the first piece of “business value” as one large chunk of work. They could see an additional piece of business value multiple times per sprint using small batch delivery. This also enables them to provide rapid feedback and quickly respond to inputs, making multiple small pivots while delivering one feature.

It also gave everyone more flexibility to adjust scope and priority. It would be very common for a user story that was part of the original requirement to be de-prioritized to make way for something more valuable or to meet accelerated timelines. Or for additional functionality not initially identified as a requirement or for the teams to adjust technical designs as needed rapidly.

Implementing Synchronized Cadence across Agile Teams

Initially, all teams ran their sprints on a separate cadence. This meant that when multiple teams were working on the same product, different parts of the development would be delivered out of sync, which slowed down the overall throughput. For example, let’s say Team 1 and Team 2 were delivering two parts of the same feature, which are interdependent. The complete functionality (combining the work of both teams) would only be “testable” once both teams have delivered. When bugs were picked up, the work would need to re-enter the teams’ sprints, and one team would already be part-way through their sprint.

Working with multiple Agile teams, one of the first changes we made was to synchronize the cadence across multiple teams to enable quicker integration and testing of combined work. It also set the company overall on a common rhythm.

We also asked the leadership to synchronize their cadence with the delivery teams to embed this mindset further.

Improving Quality Practices

The team iteratively implemented improved quality practices. From starting with the requirements management practices mentioned above (using Behaviour Driven Development) to express acceptance criteria, quality personnel were also included in the planning process during backlog grooming. Test cases were developed alongside the documents, and the team started implementing Test Driven Development. The team also started increasing the test coverage of their existing code base iteratively, setting regular coverage targets. An “Ops” person was added to the team and shifted from manual deployment processes to developing various automation scripts and self-provisioning tools to enable the team to deploy multiple times per sprint.

Implementing a Scalable Requirements Model

We implemented a scalable requirements model, enabling:

multiple teams to work on the same product simultaneously
tracking of work across multiple teams
“higher level” (bigger items) estimation using Agile relative estimation techniques
collaborative planning between multiple teams

The scalable requirements model served teams, the company on a program level, and the Lean Portfolio Level.

Implementing Long-lived Agile Teams

The company moved from the “shared resource pool” mentality to dedicated Agile teams with permanent members. Constantly shifting people between teams created disruptions and bottlenecks, but creating fixed Agile teams, with permanent members enabled a faster flow of work by eliminating bottlenecks and better team cohesion.

Encouraging T-Shaped Talent

From a “siloed” structure to a “value stream” based structure, talent was encouraged to cross-skill into adjacent skillsets.

Implementing a Program Management function

Agility at scale case study 1 - Program Level Synchronized Delivery

We established a program management function that would coordinate and track the work of multiple teams, managing dependencies, and inter-team collaboration. This function could also control “bigger batch” items spanning multiple sprints and teams. They interfaced with the Product Management function as well as the Lean Portfolio Management function.

Implementing Agile Planning Processes

To get all the benefits of Collaborative Agile Planning, but across multiple teams, we implemented a regular “program level” planning event that brought all teams together to collaborate around the work they would jointly deliver in the upcoming period.

This collaborative planning event was initially scheduled on a regular (quarterly) cadence. Over time we shifted to a “rolling wave” planning schedule to enable the company to be more responsive to shifting market depends and customer priorities.

Implementing Waste Elimination Processes

There was an ongoing focus on Waste Elimination, and the teams collaborated regularly to find ways to optimize processes. Previously the company had built up a very cumbersome control structure to counter-act the siloed / command and control style of the company. Moving over to more collaborative working methods enabled them to systematically optimize processes by removing unnecessary documentation and meetings and simplifying processes.

Typically, we use Value Stream maps to identify rework areas; however, this graphic shows the problem’s size more easily. At the start of the project, the client was not measuring rework rates during the development flow; they were merely focusing on the number of defects going into production. As you can see on this graphic, the “defects to production” rate was 30% at implementation time. However, the levels of rework were much higher “upstream” from there.

Addressing Resource Bottlenecks

As teams grow and talent shifts between roles and teams, bottlenecks appear. There is always a resource bottleneck somewhere in an organization, and through continuous improvement, companies should always seek to identify these bottlenecks and handle them accordingly.

As the chart shows, there was a severe bottleneck within the “Ops” function at one stage during the project. Multiple teams were routing their products for “shipment” to the ops team simultaneously, and this team was severely understaffed. By implementing a couple of relatively simple strategies and making a minimal resource investment (10%), they technically managed to unblock about 60% of their development capacity.

Investment in Continuous Delivery Pipeline

The company continually invested in its continuous delivery pipeline, encouraged by the data and metrics the teams produced. This greatly contributed to the overall increase in productivity they experienced over time.

Results

After completing every experiment, we analyzed the data, comparing it to the original improvement hypothesis to evaluate the results. Some improvements were adopted, and others were discarded according to the results. Some of the results of these experiments are captured below.

Reduce Rework by 74%

The fact that the company was organized in silos exacerbated the problem, as “departments” were only incentivized to ship their code to the next “department” in the queue. They were being measured on a silo-by-silo throughput rate but not accounting for the rework. The net result was that work was trickling through to customers incredibly slowly, but no one could tell exactly what this was. If you enquire with any of the “departments,” according to their perspective, the root cause of the problem would be upstream from them, but no one took a holistic view.

Reduce Lead Time by 73%

This, combined with the fact that the teams were working in “big batch” mode, meaning that it took about 30 weeks for work to travel from “concept” to “cash” and at a very slow trickle.

The teams reduced the rework rate by breaking down organizational silos, building stable cross-functional teams, and experimenting with different techniques to eliminate waste. They also reduced the lead time from 30 weeks to 8 weeks.

Reduce Product Release Efforts by 89%

The client did not actively consider the relative amount of time or effort required to release their product to customers. This “blindness” leads to sub-optimal decision-making and an inefficient resource utilization ratio. Up to 45% of all productive capacity was invested in release activities. By unblocking bottlenecks, implementing automation processes, establishing cross-functional teams, and coaching teams through various stages of maturity, this was later reduced to approximately 5%.

This number can be further reduced with continuing investment in automation and waste elimination.

Note that the optimal “batch size” or “release frequency” is not a static number but changes over time, depending on the “transaction cost” (the time and effort of a release) and the “holding cost” – the cost of not releasing the product (such as unreleased business value, or accumulating risk, etc.).

Increase Productivity by 240%

If you combine all the individual efforts, the company achieved an astonishing 350% improvement in productive capacity. In addition to the topics already covered (defects, rework, release activities), there were additional effects that had a significant impact:

The Impact of Planning Buffers on Productivity

Due to the company’s low tolerance for missed deadlines, delivery teams had a policy of adding a buffer (25%-30%) to all planning estimates. This buffer can be considered a “tax” on productive capacity. It’s a significant amount of resources that the company is paying for but has no direct control over. What makes all this worse is that buffers tend to be consumed anyway by defects, bloat, and gold-plating regardless. It also reduces the ability of the company to learn, as a high proportion of their lessons are swept under a “buffer rug.”

By making leadership more comfortable with flexible target dates, they could reduce buffers, which gave them more control over a higher proportion of their real engineering resource.

The Impact of Recruitment on Engineering Capacity

Due to a combination of rapid growth and a relatively high rate of staff turnover, the company was constantly hiring. Senior members of the engineering team invested a significant amount in this process. The company invested in streamlining the recruitment process, and by reducing the high rate of staff turnover, this reduced the load on the engineering team, unlocking more capacity.

Value Delivery

Although we realized an overall productivity improvement of 350%, using the modified Cost of Delay prioritization method, the client was not only able to deliver products faster – they were able to prioritize the most valuable products to be delivered first.

Cultural Change

Continuous experimenting, learning, and improvement formed new cultural norms over time. Ultimately, an organization’s culture is the engine that fuels its overall behavior and drives its success.

We regularly repeated the Agility Insights Assessment across teams and various levels of the organization. This helped identify focus areas and track the impacts of the various improvement initiatives.

Initially (A), the company found it difficult to make sense of its environment due to a lack of good information. This drastically improved, enabling the company to have a higher level of awareness, make better decisions, and determine the results of their decisions in shorter timeframes.

(B) The company also struggled with implementing its strategies due to a lack of internal capability. Continuous investment in systems and people improved this capability significantly over time.

Summary

Over two years, we conducted hundreds of improvement experiments and made incremental changes with long-term impacts. The project set the foundation enabling the company to continue scaling up as they grow. We also established the internal competencies that enable them to continue the journey without external assistance, and they are constantly improving their performance.

We remain in close contact with this client and consult with them regularly on specialist projects.

agility at scale