From Chaos to Success: Key Takeaways from ‘The Phoenix Project’ for Improving Software Delivery
Image generated by DALL·E 2
Do you feel like you’re constantly putting out fires on your software development project? Are you struggling to meet unrealistic deadlines while dealing with a never-ending stream of urgent emails or tickets? If so, you may be working on a “Phoenix Project”. These are high-stakes, high-pressure situations where everything seems to be going wrong, and it can feel like your project is on the brink of collapse.
But fear not! There is a way to unleash the phoenix from the flames and build a stable, continuously evolving software project where coding makes fun again. The secret to success involves following key principles of DevOps, a set of practices emphasizing collaboration, automation, and a focus on delivering value to the end user.
One great way to learn these principles is by reading the book “The Phoenix Project” by Gene Kim, Kevin Behr, and George Spafford. In this post, I’d like to share my key takeaways from the book and some crucial learnings I’ve gained throughout my journey as a software developer and project manager in several small to enterprise projects working for start-ups, as a freelancer, and now at &. So, whether you’re a product owner looking to boost your company’s software delivery or a software developer seeking to enhance your skills, grab a cup of coffee (or tea!) and join me as we explore my take on “The Phoenix Project.”
What is the Phoenix project, and why is it essential to adopt a DevOps approach?
The Phoenix Project is a novel that tells the story of an IT organization that transforms from a company with a chaotic and struggling IT landscape to a well-oiled machine using the principles of Lean Manufacturing. The Phoenix Project emphasizes the importance of aligning business strategy with IT operations and automation. By embracing DevOps principles, organizations can create a streamlined and efficient development process that delivers high-quality software products that meet customers’ needs. In today’s fast-paced, digital world, where software development and delivery can make or break a business, “The Phoenix Project” serves as a valuable guide for product owners and software delivery teams seeking to transform their organizations and improve product delivery.
Agile is dead — build a code factory!
Agile is dead — isn’t it? — At least the word agile is overused, and everybody thinks differently about what Agile actually is. Rather than trying to explain the meaning of agile, you should apply clear imaginable principles to your product team. The book compares software development to a plant factory and suggests that software development should be approached similarly. Build a code factory! This means dividing the development process into different work centers, including business analysis, design, development, testing, quality assurance, deployment, and operations which does not mean building silos in separate teams. Work centers should help you structure your team and improve your product team’s efficiency. By doing this, you can ensure that each stage of development is properly executed, leading to more efficient and effective software delivery. It also helps to improve collaboration and communication among team members, which is essential for any successful project.
Reduce handovers
When defining your work centers, it’s important to group people together who usually work together — obvious but unfortunately still not the state of the art in every company. People who often pass over tasks should work together whenever possible to reduce handovers between work centers and ensure optimal product delivery. This enables faster knowledge transfer and reduces the chances of mistakes due to miscommunication or missing documentation. So, it’s important to focus on reducing handovers between work centers and encourage collaboration among team members.
While work centers are well-defined groups, it’s essential to remember that they are part of a larger product team, and collaboration between different work centers is crucial for the project’s success. The product delivery pipeline of your project is only as strong as the weakest link, and all work centers need to work together towards the common goal — delivering a great product!
Software development is virtual work — visualize it, make it real
Software development is a virtual process, unlike, for example, the tangible work of a mechanical engineer, and for many people, hard to grasp. Therefore, it’s essential to be clear and precise about the type of work you’re doing in order that you can plan and prioritize tasks. Visualizing your work can make it more tangible and real. One effective way to do this is by using a Kanban board to keep track of tasks and their status. Ensuring all work is represented on the Kanban board is important. No work should be done without a representative task on the Kanban board.
When all tasks are visualized, you can adequately track them and see if they get stuck. The least productive tasks are those which move the Kanban board backward. To prevent this, you should define a “Definition of Done” for each stage of the development process and implement quality gates to reduce pushback of tasks. By implementing these practices, you can better visualize and manage your software development process.
Reduce work in progress
A quote from another must-read DevOps book:
“Queue theory in math tells us that as utilization approaches 100%, lead times approach infinity-in other words, once you get to very high levels of utilization, it takes teams exponentially longer to get anything done.”
— Accelerate: Building and Scaling High Performing Technology Organizations by Nicole Forsgren PhD, Jez Humble, Gene Kim
So find your bottlenecks and optimize your production pipeline to not run at a utilization rate of 100%. You can easily spot bottlenecks on your Kanban board — Your biggest bottleneck is the work center with the most “work in progress”. Why? The reason is simple — the more tasks are in progress at a particular work center, the slower the pace of work. Each task requires a certain amount of time and attention, and if there are too many tasks in progress, the work center may become overwhelmed and unable to complete any of them efficiently. Most of the tasks will be waiting in an idle state anyway, as there are not enough people or resources to process these tasks.
Therefore, measuring the time tasks spend in an idle state is essential. Idle time is unproductive lost time, and it’s important to minimize it as much as possible. For example, if a developer has to wait 1–2 days for a merge request review, she will start working on another task and forget about the context of the first task. Once the review is done, she has to stop her current task, regain the context of the old task, fix the review comments, and submit them again for a second review. Meanwhile, the reviewer may have already moved on to another task, causing further delays in the pipeline. This can lead to an unnecessary back-and-forth between tasks and increase idle time. By reducing handovers between work centers and keeping tasks moving efficiently through the pipeline, you can increase productivity and reduce idle time.
Types of work and the spiral of death
To effectively manage your team’s workload and the amount of work in progress, it’s important to understand the different types of work involved in a software development project. The answer is not as complex as it seems and is quite simple.
So, can you name the different types of work we are doing in software development?
“Yes, I think I can,” I say. “At the plant, I gave you one category, which was business projects, like Phoenix,” I say. “Later, I realized that I didn’t mention internal IT projects. A week after that, I realized that changes are another category of work. But it was only after the Phoenix fiasco that I saw the last one, because of how it prevented all other work from getting completed, and that’s the last category, isn’t it? Firefighting. Unplanned work.” — “Precisely!” I hear Erik say.
— The Phoenix Project
If you break that down from running an IT department to managing a single software project, there are four types of work:
- features (create new business value)
- changes (e.g. comply with regulations or update to a new API version)
- (technical) improvements (e.g., reduce technical dept, improve test setup)
- bugs (🔥)
Knowing these types of work helps to improve utilization and increases the quality of the software product, stability, and pace of the team. Focusing solely on one type of work, such as feature development, will increase the backlog of all other types of work, slowing down the general project. For instance, ignoring technical debt will lead to bugs, slowing down feature development and changes. To not miss important deadlines, you will eventually decide to take shortcuts that add technical debt, which leads to even more bugs,… — welcome to the spiral of death where the team’s productivity and the software’s quality decline over time. Therefore, it’s essential to give attention to all types of work and balance them properly.
Allocating time for specific types of work can be an effective way to manage your team’s workload. For example, you could spend 60% of your time on feature development, 10% on changes, 20% on technical improvements, and 10% on fixing bugs. While this may sound simple, the hard part is sticking to your plan. Do you know the — WE HAVE AN EMERGENCY — guy who sets everything on fire if he discovers a bug? If you have such a person in your team, it’s even more important to have a structured process where you properly prioritize your bugs just as you prioritize the rest of your work.
Sometimes, a bug may require immediate attention, such as a failing login that affects all customers. On the other hand, a missing frontend input validation for a rarely used feature may be less critical and can be addressed later. By prioritizing your work effectively, you can ensure that your team focuses their time and energy on the most critical tasks and ultimately achieve better results.
Automate repetitive tasks
Automating repetitive tasks is an essential part of modern software development. Human beings are not very good at performing repetitive tasks. When we have to do something more than once, the excitement and attention to detail tend to decrease, making us prone to mistakes. In contrast, computers excel at performing the same task over and over again without altering the output or getting tired, angry, or bored. This is why automating tasks such as testing, building, and releasing software is crucial. For instance, automated tests can help you find bugs reliably on every run, and building a continuous integration (CI) pipeline can streamline the process of executing tests. Similarly, creating a continuous delivery (CD) pipeline can automate the process of releasing software. By automating repetitive tasks, you can free up time and resources to concentrate on adding new value to the project instead of being bogged down with error-prone and boring repetitive work.
Fail fast, release often
Once you have automated repetitive tasks, you can focus on failing fast, getting feedback, and bringing value to your customers. But why is it so important to fail fast? Not knowing it is the worst thing that can happen if you are on the wrong track. Imagine going on a hiking trip and starting in the wrong direction. After six hours of walking, you realize you went in the wrong direction — your day is over. However, if you had asked for directions right away, you would have had a wonderful day in the mountains. The same applies to software development. Fail fast means getting feedback fast and adapting fast, leading to quicker success. One way to fail or succeed fast is to bring small, incremental changes to your customers to evaluate instead of investing months in development and releasing a big feature only to find out that your customers don’t like it.
By failing fast, releasing often, and getting feedback, you can bring value to your customers even faster. Releases should be a routine, unexciting event, rather than a long-awaited milestone that takes months to prepare for. We should be releasing code multiple times a day — at least ten deploys a day (according to the book). This sounds impossible? Well, maybe releasing 10 times a day is hard because you have to change not only your technical setup but also the way your team or company is working. However, keep in mind that any increase in your release frequency improves the status quo. We can’t wait months to deliver new features to our customers, especially when we don’t know if they’ll work.
It’s important to note that failing fast does not mean shipping untested products or letting your customers test the application. Rather, it’s about ensuring that you are building a feature that is useful and will be used by your customers. It’s also about testing whether you meet the expectations and fulfill all requirements correctly. So, while failing fast is important, it’s equally important to do so in a way that maintains the quality of your product and meets your customers’ needs.
What’s next?
There is still so much more to learn from the book, and I highly recommend reading it. However, simply gaining knowledge about these improvements is just the first step. Applying them in real-life projects can be quite challenging, but pushing through these challenges and implementing these strategies is important, as the rewards will undoubtedly pay off. Any improvement is better than none, or as Sensei Mike Rother says:
“…that it almost doesn’t matter what you improve, as long as you’re improving something. Why? Because if not improving, entropy guarantees that you are actually getting worse, which ensures that there is no path to zero errors, zero work-related accidents, and zero loss.” Erik continues, “Sensei Rother calls this the Improvement Kata, he continues. “He used the word kata, because he understood that repetition creates habits, and habits are what enable mastery.””
— The Phoenix Project
It’s time to build
At &, we are dedicated to delivering high-quality software with our cloud-native development teams. We constantly strive to improve our daily work by applying well-known DevOps principles and other best practices. We’re happy to help you deliver your software projects fast. If you have any questions about our approach or how we can help you, please feel free to reach out to me.