Whether we are working from home or our offices, the challenge of ensuring the secure delivery of business value to our customers is a difficult one.
On the one hand, we need to enable our teams to deliver value quickly as this increases our ability to learn from our customers. On the other hand, we want to stop the unwanted exposure of customer data by ensuring secure delivery. These two goals may seem at odds. After all, a high-rate of change increases the chance something goes wrong. This results in a natural tendency to want to slow down the delivery process. However, there are ways to ensure you get both speed and safety.
Let us look at some of the common failure patterns and how to approach creating a strategy to remediate.
Why the conflict
Security is often a small but essential part of an organization. In start-ups it is not uncommon for security to be a “problem for tomorrow”. In larger organizations, security often exists in a separate silo of security creating communication overhead and conflicting goals.
There are not enough security professionals to go around. The current estimate is that there will be a shortage of 3.5 million cybersecurity roles by 2021. On top of that, the security professional’s role is to protect the whole organization, not just your software delivery pipeline. Often their understanding of the development process is limited, resulting in an overcompensation of controls that slows down the delivery pipeline.
Even if they have the time to grasp the complexity of the delivery pipeline, their goals are opposed to those of the delivery team. I have been in organizations where security’s solution to secure the pipe was to ask for security approval at change review, that being the only consistent gate along all pipelines. This was not a popular idea with the business team when security tried stopping their latest feature release (many other problems there, but those are for another day).
So if security is overloaded, suffering from a gap in understanding and fundamentally at odds with modern delivery practices, how do we create a strategy to move forward?
Breaking down the problem
The first thing we need to do is understand our current delivery practices. Creating a sound Value Stream Map to give you an understanding of where you stand today and allow you to measure the impact of changes as you make them. This VSM exercise does not need to be exhaustive and time-consuming. Still, it should capture the delivery process from as high in the organization as possible to capture all the areas you may impact. If you have been undergoing an Agile or DevOps transformation, there may be existing information you can use to help build this map.
Once you have an understanding of your current delivery processes, you can identify where the biggest gaps in the process are and where you need to introduce checks. I have found it helpful to use a straw man approach to the pipeline to help guide the conversation. Something like this:
Understanding visually where we need to introduce changes helps create alignment in the group. Getting everyone onto the same page about what is required is essential.
From this exercise, we come away with a list of activities categorized into:
People – Required education, missing skills, etc.
Process - How issues are handled, reporting false-positives, etc.
Technology - Tools for static scanning, open-source scanning, etc.
From here, we can prioritize this list and build our initial plan.
Common mistakes
Having used this process at several large organizations, here are a few of the common errors I have seen people make:
Forgetting to build feedback loops. Integrating SAST, DAST, RASP and any other type of security scanning into your pipeline is one thing. Filtering out false positives and creating a useful feedback loop into development teams is another. Having your tools scan for new vulnerabilities on each commit, especially on legacy code, is more valuable than providing the same list of 10,000 things that need fixing. It is essential to work with delivery teams to educate them and build those feedback loops together.
Creating more than one source of truth. Having multiple scanners each doing the same thing is a bad idea. There is a tendency for the tools to contradict each other, the licensing and execution time cost is expensive, and your teams may get overloaded chasing ghosts.
Forgetting about scaling the solution as more teams come on board. It is critical to get a handle on how licensing and capacity scale for the solutions you are introducing into the pipeline so that they don’t become an unintended bottleneck.
Not understanding your legacy applications. I have seen scan times in the weeks for legacy applications, which have resulted in blaming the tool vendor. While there may be problems with the tooling, it is also worth looking at the make-up of your application. Ensure you are not pulling in and scanning libraries that can be handled out of band or whether there are ways of breaking the application apart for scanning purposes.
Failing to involve your delivery teams as a part of the solution. The easiest way to help scale your security team is to involve the delivery team in designing the solution. It also helps remove the inevitable conflict between the priorities of the delivery team and those of security and operations. If the three areas continue to operate independently and only communicate through policy documents and tickets, you may have problems sustaining the change.
Conclusion
Hopefully, this has given you some insights into how to go about securing your software delivery practices. Two good places to start are:
Bring the key stakeholders from development, security and operations together to create a strategy for what is needed.
Take a look at your current practices and identify gaps and opportunities for improvement.
Xodiac’s Discovery and Insights services are great places to start addressing these items and we are always happy to have the conversation.