3 new DevOps lessons learned

“In a closed society where everybody’s guilty, the only crime is getting caught. In a world of thieves, the only final sin is stupidity.”

― Hunter S. Thompson, Fear and Loathing in Las Vegas

IT workers sometimes live in fear and too often loathe their jobs. Look no further than last year’s testimony by the former Equifax CEO as a reason why. He blamed a single IT worker for the company’s unprecedented data breach. That explains the fear, and his $90 million exit package may explain some of the loathing.

But there is light and hope on the horizon, and that was on full display at the DevOps Enterprise Summit (DOES) this year. Here are three of my takeaways:

[ You're probably recruiting for DevOps engineers right now. But should you be? Read The great DevOps engineer title debate. ]

1. Lawyers can be IT’s ally in the battle to modernize

Anne Bradley, Nike’s chief privacy officer, and Courtney Kissler, Nike’s VP of digital platform engineering, spoke about their journey to kill legacy systems. In a charming, hilarious, and brutally honest session, Courtney and Anne laid out how they used potential violations of data regulations as an impetus to shut down aging legacy systems that were inherently violations of those new regulations.

Anne spoke about “negative ROI” and spelled out a very simple expected value ROI on fines from regulatory agencies. Courtney used that analysis to advocate for killing those legacy systems and did so with an “Irish wake,” a cake, and poems.

When teams want to remove older systems, they can, and should, rely on an external support organization such as legal to help them make the case. In this example, it wasn’t just about saving time and money for Nike; the legacy system was putting them at risk for hundreds of millions of dollars in fines. Thus, removing the legacy system was a pretty easy decision.

This approach is counterintuitive to most technical people. Often, technical people look at a legacy system and create a list of reasons why it should go away but are met with pushback from other non-technical people.

Engineers need to look for allies in their efforts to modernize their applications. To help with “addition through subtraction,” a company’s lawyers can be a powerful asset.

[ Read our related article: How to explain cloud-native apps in plain English. ] 

2. Make databases part of your DevOps transformation

Dr. Nicole Forsgren and Jez Humble from DevOps Research and Assessment (DORA) presented their latest DevOps Research and Assessment State of DevOps Report. An interesting addition: They highlighted the value of including the database in your DevOps transformation.

DORA has spent years gathering this data, building on it each year. For the 2018 report, they found addressing databases in DevOps initiatives is positively correlated to IT performance.

Furthermore, they found databases to be non-differentiated, meaning that no matter where you are on your DevOps journey, you can benefit from addressing the database.

I’m reminded of an old saying: “The best time to plant a tree is 20 years ago. The second-best time is today.” (For more details on the findings, listen to the Accenture SolutionsIQ podcast recorded at DOES this year.)

3. Software release approval process needs a rethink

Forsgren's “Accelerate” book signing proved popular at the event. Two things that I find interesting about this book: None of the authors (Forsgren, Humble, and Gene Kim) work at a company that sells technology; the authors use years of data to prove their findings. This is very different from other books that speak from anecdotes and experience.

Using that data, they have made some counterintuitive discoveries. The most mind-boggling finding is on page 78-79, concerning the approval process for software releases. Of course, our industry has believed for years that we need to have a very deliberate, albeit slow, approval process. We wanted humans to verify that everything was ready for the release.

The researchers asked organizations about the following approval process scenarios:

• All production changes must be approved by an external body (such as a manager or change advisory board)
• Only high-risk changes, such as database changes, require approval
• We rely on peer review to manage changes
• We have no change approval process

The findings were absolutely the opposite of conventional wisdom. The data clearly shows that “teams that required approval by an external body achieved lower performance.” That’s right: Instead of improving performance (things like MTTR and success of releases), it decreased performance. This is about as significant as when we found out that asbestos causes lung cancer – a thing we believed to be safe is actually very dangerous.

This year’s DOES conference created more noise in the DevOps community around the importance of the battle for efficiency and modernization, the significance of the database, continued focus around CI/CD, and the awareness that the approval process needs to be streamlined. It’s great to see the attention around these issues growing, and I hope to see this continue in the future.

[ Is MVP part of your DevOps work? Read Minimum Viable Product disconnect: How to avoid a dangerous trap, by Robert Reeves. ]