One of the better ways to get started with CI/CD: Dig into why you should get started with CI/CD at all.
“When getting started with CI/CD systems, they should be viewed as a force multiplier for your product teams,” says Josh Komoroske, senior DevOps engineer at StackRox. “Investing in setting up CI/CD system properly will pay dividends in engineering and testing time as it has the potential to vastly lower the time to market for new features.”
CI/CD: A primer for leaders
CI stands for continuous integration, and it involves building, testing, and merging developers’ various code changes into a shared repository such as GitHub or a container registry, usually multiple times per day.
CD, on the other hand, refers to either continuous delivery or continuous deployment, which are sometimes used interchangeably. Either way, this typically refers to later stages of the software pipeline, and especially to how new code moves into production.
(For more on CI/CD basics, check out What is CI/CD? We’ll revisit that primer a bit later on in this article. Read Getting started with CI/CD: 4 success factors, for more advice.)
Komoroske notes that there are multiple measurable gains to be had from a well-conceived CI/CD pipeline. Take testing as a major example.
“CI/CD can lower the amount of time it takes to test, and conversely, allows the testing of more code. By running a test suite in CI, a developer can get feedback on their feature faster,” Komoroske explains. “In the meantime, it frees them up to do more development work instead of running tests. You can also run tests on every single commit instead of once at the end of a release cycle. This greatly improves code quality and reduces bugs.”
CI/CD can also shrink feedback loops and speed up features, enhancements, and fixes: “CI/CD should lower the iteration time between development and feedback. Proper CI can give feedback to developers moments after they push code,” Komoroske says. “It also lowers the amount of time it takes to deploy new features. Proper CD can enable the deployment of new features moments after they are reviewed and approved.”
These all sound like wins from a modern IT and business standpoint. But Komoroske did mention a prerequisite to these benefits: Investing in setting up CI/CD system properly.
4 keys to setting up a CI/CD pipeline
We’re here to explore, with help from Komoroske and other experts, what goes into that “proper setup.” This isn’t a step-by-step technical guide to setting up the perfect CI/CD pipeline (in fact, some experts would say that such a software development utopia doesn’t exist). Rather, it’s a leadership-oriented overview of some of the key principles and concerns that should guide the setup of a CI/CD pipeline based on the specific needs of your organization, with a particular eye toward teams that are starting from scratch. Let’s dig into four main areas:
1. Embrace key principles: Simplicity, repeatability, flexibility
“Fundamentally, succeeding with CI/CD requires transitioning from a process that is very manual and slow in many organizations to one that is much more automated and rapid,” Red Hat technology evangelist Gordon Haff noted. “Some companies can end up going from one or two releases per year to a release of one or more per week or even per day.”
But if you want to realize CI/CD’s promises – starting with faster and more frequent deployments, but also including improved testing and code quality and reduced operational burdens – then choose design principles for your pipeline and processes carefully.
Let’s start with simplicity: It’s a must. Too much overhead – in terms of implementation, operations, optimization, and other contexts – can immediately eat up any of the benefits and then some.
“Remember the golden rule: YAGNI – you ain’t gonna need it,” says Nate Berent-Spillson, senior delivery director at Nexient. “The key to building your first CI/CD pipeline is to stick to the essentials based on only your immediate need.”
We’ll talk tools in a moment, but suffice it to say that you want to avoid what Berent-Spillson once described to us as “tool salad” (not to be confused with one of the healthy recipes you pledged to consume more of in 2020). If you’re experimenting or otherwise just starting out, Berent-Spillson recommends getting going with open source and free tools; you can add commercial products as needed later.
[ Related read: Getting started with CI/CD: 6 pitfalls to avoid ]
Simplicity begets all manner of other principles and benefits, such as widespread buy-in and cultural acclimation. In fact, it’s a defining characteristic of successful CI/CD.
“Your goal is to create an automated process that takes code from your source repository through compilation, testing, and deployment, and notifies the right people when it’s done,” Berent-Spillison. “If you do only that, you’ll have done quite a lot.”
Simplicity can also guide your long-term strategy: Build your pipeline incrementally – start with just CI and work up to CD, for example – and order key steps in the process with simplicity as a guiding logic.
“When getting started, run the high-impact, low-cost things first, like unit tests,” Komoroske advises. “From there, move on to low-impact, low-cost things like code linting and formatting. Run the high-cost things last, like integration tests or test deployments.”
Repeatability is a related guiding principle. Since automation is intrinsic to CI/CD, it’s important that steps in the pipeline are easily re-run, especially given the importance of testing (and re-testing) in a CI/CD pipeline.
“CI/CD allows humans to focus on what they do best and lets the automation take care of the repeatable minutiae,” Komoroske says. “Repeatability is key, so developers should structure builds so that they are as deterministic as possible and have all the information required to be re-run.”
While your pipeline should be well-defined and understood, avoid rigidity where possible, whether it’s a product of dogmatic thinking, a lack of foresight, or other reasons. Flexibility is important, especially when you take the long view. Komoroske recommends, for example, enabling simultaneous deployments.
“If there are multiple developers building features at the same time, it’s not unreasonable to have them all deploying test deployments,” Komoroske says. “Structure things to allow for this possibility.”
2. Understand the building blocks of a CI/CD pipeline
Dissuade yourself – early in your CI/CD exploration – of the notion of an ideal pipeline that works for every team or organization. Your path to a well-oiled, productive CI/CD pipeline will be paved by some fundamentals, but the actual route differs from team to team. This becomes especially important to keep in mind when you start considering tools.
The “What is CI/CD?” primer mentioned at the outset has a good simple visual of the fundamental building blocks of a pipeline. The CI phase covers build, test, and merge. The CD phase depends on whether your “D” stands for “delivery” (repository) or “deployment” (production); In some organizations, the terms “delivery” and “deployment” might be used interchangeably, and it’s not worth losing sleep over these semantics.
“Remember that CI/CD is really a process, often visualized as a pipeline, that involves adding a high degree of ongoing automation and continuous monitoring to app development,” the article notes. “Case-by-case, what the terms refer to depends on how much automation has been built into the CI/CD pipeline. Many enterprises start by adding CI, and then work their way towards automating delivery and deployment down the road, for instance as part of cloud-native apps.”
[ Trying to articulate the advantages of cloud-native to others? Read also: How to explain cloud-native apps in plain English. ]
At some point, you’ll map tools to these key building blocks. There’s a vast menu of choices out there.
“There are many tools that can automate the tasks in CI/CD pipelines,” says Tsvi Korren, field CTO at Aqua Security. “From open source to cloud providers, these tools include several key features: connections to repositories where code and other artifacts are stored; a procedural language or UI that outlines the steps to integrate the artifacts into an application; connections to testing tools and staging environments; and connection to DevOps automation tools that can provision and configure the infrastructure for application deployments.”
Again, remember Berent-Spillson’s “YAGNI” and the virtue of simplicity here. Too many tools cobbled together might turn into an IT equivalent of Frankenstein’s monster.
Korren shares a relatively simple illustrative scenario of translating the basic components of a CI/CD pipeline to recognizable tools or platforms; keep in mind, this is just a single illustration of what this might look like.
“For example, a pipeline based on open source can include Nexus as a repository, Jenkins as a CI platform, and Kubernetes for CD of containers,” Korren says.
[ Kubernetes terminology, demystified: Get our Kubernetes glossary cheat sheet for IT and business leaders. ]
Then the process could look something like this: “Developers will commit their code, which triggers a build of an image by the CI. The resulting image is pushed to the registry and is used in automated testing and staging scenarios. When testing criteria are satisfied, a deployment manifest will be sent to Kubernetes, which will pull the images and run them as containers, completing the continuous development cycle of an application,” Korren explains.
Next, you need to choose the right applications for CI/CD and double down on testing:
