Are your automated software delivery processes safe?
With the emergence of new technologies and methodologies, the frequency of releasing software changes into production environments has increased exponentially over the last decade. The DevOps approach with its automated CI/CD pipelines made it possible to drastically shorten the lead time required for changes to become available to the end-users. At the same time, live releases have become much more reliable. “Automate (almost) everything” is one of the DevOps principles that replaced complex, repetitive tasks that were prone to human errors with automated procedures that can be run on-demand with little human input or oversight. As a result, businesses, empowered with new, shiny DevOps tools and processes, have drastically reduced their time to market from months or weeks to days or sometimes even hours.
But with great power comes great responsibility. It is a fact that smaller, frequent, automated releases are less risky and offer more stability, however, automation can give us sometimes a false sense of security. An automated software delivery pipeline can be a vector for large cyber-attacks as the major hacking campaign of SolarWinds CI/CD solution showed us back in 2020. In the SolarWinds hack, attackers legitimized their malware by injecting it into the SolarWinds build pipeline. The pipeline produced trusted software artefacts that obtained digital certifications and were subsequently released to the customers. The hack affected many fortune 500 companies and large government agencies. Allegedly, even the cybersecurity and infrastructure security US government agency fell victim to the hackers.
The speed at which we deliver changes must be underpinned by smart governance to achieve what we call equilibrium
It is important to have processes automated and it is fantastic to be able to deliver software changes at speed but doing so at a suicidal speed should certainly be avoided. We need proper checks and balances in place to avoid disasters. The speed at which we deliver changes must be underpinned by smart governance to achieve, what we call, equilibrium. Equilibrium is defined as the right balance between agility (the speed of delivery) and cybersecurity and is specific to your organisation.
So, what is kind of policies and standards we should consider for our CI/CD governance to achieve a state of equilibrium? There are some obvious ones such as secret management for full control over your most sensitive details and role-based access control for securing your pipelines not only from malicious actors but also from accidents and mistakes. Vulnerability scanning could and in most cases should be incorporated into your CI/CD pipelines. Adding scanners is a relatively easy and very effective way of implementing vulnerability management. Automated manual gates could also be considered. What sounds like an oxymoron is in fact a solution that will programmatically pause your pipeline and ask for authorisation to proceed. This approach can be used to perform sensitive actions in production environments where customers can be easily affected.
CI/CD governance is absolutely not about taking away the freedom, speed and agility from engineers.
In large organisations with many CI/CD pipelines, it may be worth looking into creating standardised pipeline components. Templates for particular stages or tasks can be created and subsequently reused across different pipelines to ensure consistency and compliance. Pipelines should be always defined in code and stored in a repository so that changes to their configurations could be tracked. For the pipelines to attain legitimacy, they should be registered with metadata pertaining to their purpose, users, and owners. Last but not least we should be able to always answer the question of who did what, where, when and why, and therefore there should always be an audit trial available.
The controls must be considered in the context of your organisation, its size, and its risk appetite. You have to decide what controls you need in your CI/CD pipelines so that you can attest to the integrity of software assets produced by them. CI/CD governance is absolutely not about taking away the freedom, speed and agility from engineers. It is about providing the necessary minimum viable guardrails to ensure that self-organising, agile teams do not shoot themselves in the foot. With smart CI/CD governance and equilibrium, organisations will become less susceptible to cyber threats and still deliver software changes at speed.
