Waterfall-sh is a short story inspired by the book called Cloud Native transformation by Pini Reznik, Jamie Dobson, Michelle Gienow.

The book talks about the challenges that large enterprises go thru when they do Cloud Native Transformation. It’s a good read. Check it out!

Many companies consider themselves Agile, but they only adopt like one practice from the Agile manifesto, such as daily standups or scrum meetings. In the book, those companies are called Agile-ish. I think there is a spectrum between Waterfall and Agile organizations. On the wrong side of the spectrum are an organization, which I call Waterfall-ish, the companies who have all the symptoms of Waterfall organization – hierarchical, permission seeking organization, except that they lie to themselves that they are on the path to Cloud Native transformation. 

The story

Waterfall-ish adopts the new tech, uses Kubernetes, and hires many new people to do the transformation. But the organization hasn’t changed. Hierarchy is still there. The decision chain is still the same. Organizational processes are the same old.

They are actually even worse now; they just hired many new people and many new managers, which they need to manage. Hierarchy is even bigger now, and the decision chain is even longer and slower.

Ok, we have a bunch of new people, new managers. What should the company do? 

The plan

Here the struggle begins. High-level people and the new managers try to come up with a plan. There are many opinions. Some want a modern experience for mobile. Others – a better website. Newly hired people want to get rid of legacy code. After all, we are doing Cloud Native transformation, aren’t we? A lot of people argue. Nobody can agree on what to do. There are a lot of meetings scheduled. The whole process starts to feel like you’re trying to develop a Five-year plan for the Soviet Union’s national economy. 

Two months go by, people finally agree on a project. Now it’s time to do agile, go build, iterate, release, etc., right?

Build, iterate, release?

You would think so. You wanted to quickly create some proof of concept, build an API, mock some data, add a database, iterate.

Nope.

You are forgetting about the Architect. You didn’t even notice where he came from. But he did, and you have a lot of meetings scheduled. It’s time to talk about the Architecture, the entities, the relationships, where we add events, what they are, and how to model data. You know the important stuff.

Now your new system looks like a spaceship. You’ve added a bunch of queues, a bunch of events, a bunch of boxes, a bunch of relationships. Everything is clean. It’s enormous, but it might work. 

Finally, we’ve done with this nuisance, can we start building already?

Building a spaceship

You start work. Oh, and by the way, your engineer team is not that huge. It’s like you and this other person. Now the Architect made you use his new framework to connect to different services. He hasn’t really tested it yet, as he doesn’t have a lot of time. You know he is busy. He has a lot of meetings to go to. But it’s ok. It’s probably better than a random open-source framework from the internet, right?

You take 9 months to build out this spaceship. It’s finally ready for a trial run. Testing in a development environment shows something is wrong. You are randomly missing data. You spend a week debugging. Finally, you find that the new framework actually has a bug, which causes random loss of data. You ask around, turn out other people have already migrated away from this new framework to an even newer one, which is conveniently utterly incompatible with the previous one.

Oh well, you spend one more month rewriting all the apps. Finally. It seems like we are getting close to the launch. We are in the last mile. Fixing the last bugs.

The launch

Nope. There is some stuff going on in the background. Some high-level dude just remembered you and the other engineer exist. Turns out, that five-year plan is not set in stone either. Your manager and manager’s manager and product team and their managers decided it’s time to switch focus.

We are an Agile organization, aren’t we? We change our minds quickly.

DevOps

Cool. You are now assigned to the DevOps team to work on some CI/CD pipelines. Because they want to speed up their delivery. That’s fine. DevOps sounds cool, doesn’t it?

You notice that you get a lot of messages from other developers about broken Jenkin nodes. The disk seems to be full. You ask around, then one of your teammates takes care of it. You ask your manager, shouldn’t we monitor this? He says, “Nah, not in our plan; we need to finish this CI/CD piece and then work on storage.”. You work on some of those pipelines, add some cool stuff like CVE security scanning, play with the new tools. It’s fine. Everything is fine.

The culture

After a while, you notice, some of your teammates seem to be stalled. This particular teammate appears to be working on the same task for 2 weeks in a row now. You look at it, it seems doable in a day or two. You wonder, what’s going on?

Then you have an epiphany, you work in a Waterfall-ish organization. The faster you do the work, the more work you will get assigned. Then you remember the talks and promises when they hired you, the transformation initiative, the squads, the fast iterative cycles, the tech. 

Everything is a lie. You have another epiphany, that teammate was like you – active, involved, wanted to make a change, improve the status quo. Slowly, but surely you start to slow down your progress. What’s the point anyway? And you become just like your teammate.

Summary

Hey, hey, I hope you like the story, and I hope you don’t have similar experiences. The story is inspired by the Cloud Native Transformation book and a lot of conversations between my friends.

Let’s identify a couple of anti-patterns from the story below and figure out how to transform them into patterns.

Anti-patterns

Long decision-making cycles, constant change of priorities; teams have no authority.

Typical in huge hierarchical, permission seeking organizations. There are a lot of people in the decision chain. Nobody decides what to do. 

Solution: 

Look at what others do. For example, look at Amazon’s two-pizza teams or at Netflix’s freedom and responsibility culture. Read books. Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations is a good one. Cloud Native transformation is another. Basically, you will have to work on your hierarchy and let go of control and give teams/squads more responsibility.

Not invented here syndrome; Teams are forced to use tools or frameworks for political reasons.

Again very typical in hierarchical organizations, Staff Engineer or Architect creates a framework rest of engineering people are forced to use, no matter how badly untested, non-documented, with horrible user-unfriendly APIs. It delivers a lot of fake progress, as typically it’s seen as “We have the framework in place, now rest should be easy.”.

Solutions:

9 times out of 10, you don’t need to be reinventing the wheel. Just use a well maintained open-source system. CNCF has a good set of projects. If you still think you need to build a custom framework, don’t force anyone to use it. Let it happen naturally. It’s up to the engineer who created the project to gather the following and market it to others. Developers like well-documented code with understandable and straightforward APIs. Usually, useful & good frameworks will get adopted automatically. Read How big technical changes happen at Slack. 

I’ve noticed that in CNCF user studies, many companies have migrated from their own custom-made systems into the CNCF hosted project. That typically simplifies and improves their systems by a lot. Definitely look into that. If you are using any open source project, be a good citizen, dedicate some time to help that open source project somehow.

Hire good people, box them into specific design or architecture.

Software Engineering is a creative process. People need space to think and create. Don’t force your talent to go in one direction or another. Let them explore. 

Solution:

Adopt Agile. Iterative architecture is often a way better alternative than an Architecture designed in a Waterfall way. Systems can be messy, and that’s ok. The Architect’s job is to consult when needed, not force-feed his architecture.

The end

Thanks for reading till the end. Do you have any thoughts? Can’t wait to hear them in the comments 🙂