When I joined CBRE back in 2012, there was a need to strengthen our project methodology standards. So one of the first initiatives for IT was to improve our agile project methodology process and roll it out globally for our entire IT team. We are now beginning to see the benefits of that effort. We’re delivering projects to the business on time, on budget and consistently because of our enhanced methodology.
That was step one. This year, we're working to make DevOps part of that standard methodology.
First, we identified three projects where we could implement a DevOps team and approach. Two of the projects were already up and running; our plan was to implement DevOps after the first phases were complete. The third is a greenfield project — something brand new that would be completed start-to-finish with a DevOps team and approach. We are doing it this way because DevOps is a culture change for a lot of people in the organization, and we wanted to start small so we could ensure success.
Optimizing Agile for DevOps
For our approach to DevOps, we took a wide-angle view of the entire IT lifecycle and all the capabilities needed to rapidly design, build and deliver robust quality systems. Then we color-coded it to show how mature we currently were in these processes. Green indicated the areas where we were doing very well, yellow were areas that needed further improvements, and red highlighted capabilities that were lacking within our agile process.
Our focus was on flow optimization. This exercise enabled us to figure out ways we could adjust and fine-tune our entire IT delivery pipeline to make it run faster and smoother. We identified five DevOps enablers and focused on these key areas to start.
Testing: We focused on establishing a testing center of excellence, a quality assurance team, to address questions such as: How can we get continually better at testing? How can we automate some of our testing so that we find issues quicker and run through test cycles a lot faster? Manual testing is limiting – it's incomplete as test cases are often times undocumented and this causes errors. Having this under the umbrella of a testing center of excellence will enable us to document, test, and report out value to the businesses much more efficiently (faster testing cycles and automation for cost savings).
Release modernization: We also took a look at our continuous delivery process and how we could improve our release output. In order to improve, we asked: How can we deploy more application changes quickly to production? And, how can we increase the quality of what we do release so we have improved SLAs with less system downtime, less manual intervention, and less room for error? How can we automate that whole process? Leveraging our Microsoft Team Foundation Server (TFS) platform along with custom scripting, we were able to pilot release automation for a few applications and were able to quickly see the benefits. Our operations team saw the reduction in manual processes and how much easier and faster it was to deploy the changes - therefore freeing them up to focus on other value added tasks. And they were fully onboard to work on a roll out strategy to automate deployments for additional applications. The key to our success was ensuring the applications delivery teams worked collaboratively with the operations team during the pilot, addressing concerns and ensuring buy-in throughout every step of the process. We now have a repeatable and reliable push-button activity for releasing software changes that didn’t require us to purchase any additional tools or software.
Readiness to serve: Right now we have IT resources focused on support and small projects and resources focused on large capital projects. We don’t have a team in place for rapid prototyping of ideas that come from the business. To address this issue, we're doing a technology refresh assessment to ensure we have the right skills and knowledge of emerging, flexible open stack technologies. This will enable usto move rapidly on new projects, while reducing software licensing and support costs. We are focusing on retaining a lean delivery team (jacks-of-all-trades type resources) who can help us provide the business with a readiness-to-serve capability. All of this is self-funded by finding efficiencies in other areas (continued use of offshore resources for support and delivery, application rationalization, etc.). This approach also allows existing resources to become cross-trained on the latest open source technologies.
Operational insight: Currently, most of our monitoring is done at the hardware level. A health check system is in place but processes and access around tooling need improvement. We know a server is up and running, but we really don’t have insights into the application level. Teams rely solely on symptoms to determine problem root cause and this lack of tooling significantly delays application recovery times as well as development break/fix times. Also, lack of diagnostic tooling during Quality Assurance (QA) testing means more defects slip through to production. We have selected tools that provide deep visibility into the cause of application performance issues down to the tiniest detail. The tools also allow us to get access to code level diagnostics as well as full stack traces across tiers. This in turn helps us address issues more efficiently and recognize anomalies that might be taking place in areas where we may not have been otherwise notified, allowing us to proactively address issues before our end users experience them.
Platform-as-a-Service: The ability to provision hardware and core software quickly and efficiently is a critical component of high velocity IT organizations. We’re looking at Platform-as-a-Service for self-service provisioning that will allow tremendous App/Dev flexibility. How can we quickly spin up virtual environments for our application teams so that they don’t have to wait until hardware is procured, as an example, or the whole application environment stood up? How can we push a button and provision an environment so the developers can be off and coding right away? This is a shift in mindset for how we operate today, from our Infrastructure team fulfilling requests to becoming a solution enabler; not only for IT but for our business partners.
These are the five major items that we’ve focused on to start our DevOps journey. And, more important, we’ve clearly defined metrics to show improvements in each of these five areas. We will be able to report on the number of releases, cost savings, efficiencies in test automation, and more. We will have a clear before-and-after picture showcasing the value that DevOps can bring to the business.
