Security is a peculiar topic in software because it is, at the same time:
Undisputably crucial to the long-term success of an organization, and
Utterly abhorred by almost all developers I know.
I can go even further and say that, in an ideal ranking of the most despised areas of dev work, security ranks higher than writing docs and maintaining tests.
I believe there are two reasons for this:
Perceived utility — while things like tests and docs display at least some short-term benefits, taking care of security looks like pure overhead (until it doesn’t).
Complexity — security topics are genuinely complex, and most developers are not trained in them. Managers, of course, even less so.
So, in many teams, this responsibility is all left on the shoulders of a few savvy senior engineers, who have to fight basically everybody else — their peers, PMs, leadership — to get things done.
Of course, we can do better than this.
This article is meant to be a primer on how to create a secure development process. I will cover the most sensible things you should do, without overwhelming engineers.
So, if you are starting from scratch, you should be able to go through this article and walk away with a minimum viable security plan. If you already have solid practices in place, you may find advice that complements them, and plenty of tools you can start using right now.
Here is an outline of the article:
🌡️ Risk tolerance — each product is different. You should figure out where you stand and act accordingly.
🏢 Security is organizational — why you can’t achieve security through individual skills, and what you should do to make it happen.
💻 Secure development process — the core of the article. We will go through practices that embed security into requirements, design, coding, testing, and maintenance.
📚 Resources — plenty of material to learn more, including checklists and training programs.
🔨 Tools — the most recommended security tools in my circle.
Let’s dive in!
🌡️ Understand your risk tolerance
To set expectations right, it’s important to understand that there is no silver bullet for security, nor a standard process that works for all companies.
In fact, different companies are exposed to different kinds of threats, based on the products they build: a healthcare or financial product carries different risks than e.g. a remote job board.
You should be aware of such risks, decide your level of tolerance, and act accordingly. Investing in security often trades off other dimensions, like speed, or profitability, so you should be intentional about such an investment.
In the end you won’t probably be able to zero in on all the possible risks, and that’s… fine.
🏢 Security is organizational
The second posture to get right is that security is mostly an organizational topic. Good security is achieved via good process and culture, rather than the heroic skills of single individuals.
The dev team has to be empowered by management to refuse to ship a change if it is insecure, or to drop everything to fix a vulnerability on production. If there are no strongly established cultural norms that security is important, it will get left by the wayside in favor of feature development.
— Kendrick Curtis, Engineering Manager Lead
To be honest this is true for most software qualities, but especially for security. In fact, security is a complex topic that requires specialized knowledge, and is unreasonable to expect all engineers to become experts in that. All engineers, though, will write code — so what ultimately enforces security is process.
To achieve this, there are three main areas you should invest in:
1) Hire a security leader early on
A security leader is an expert that can spearhead security efforts throughout the engineering org. They can:
Embed security into the development process
Design the right amount of training
Act as a point of reference for any security concern
As for when to hire for this role, your mileage may vary based on your risk tolerance and how skilled in security your early engineers are, but a good number is within your first 30-50 employees.
2) Train employees in security
This article specifically focuses on secure development, rather than security in general, but all employees arguably benefit from security training.
Pagerduty has some great open-source courses you can use to get started. You will find both high-level ones, aimed at everyone, and specific engineering ones.
When it comes to developers, aside from the basics, you don’t need all of them to become experts. Based on your needs, and the attitude of the people on your team, here are two options you can implement:
Create a security champion program — designed for those who have a special interest in security. They may exercise this function alongside their other duties.
Include security in your career framework — if your senior engineers have to deal regularly with security concerns, it is a good idea to make this official by writing down such skills and expectations in your career framework. You may design more in-depth training to help engineers match them.
Ben Chiverton, Director of Engineering, weighs in on this 👇
[security as a competency in career frameworks] helps emphasise that security is just as important as delivery. At the end of the day, if there is a culture of delivery over security then I don't have to tell you what will be dropped when there is a tight deadline! Org structure helps too here, make sure you have very senior security officers.
3) Embed security into the dev process
The only way to create secure code is to embed security into every step of the dev lifecycle.
Many teams only take care of it during the initial design and with a final review, implementing the infamous security sandwich. The sandwich fails for two main reasons:
Specs often change during development, which leads to inconsistent or incomplete reviews.
It casts security and development as two adversarial categories — which leads to conflict.
Instead, just like DevOps makes infra work well with regular development, the same can happen with security.
Let’s see what a secure dev process looks like 👇
💻 Secure Development Process
Let’s organize the development process into five main stages:
📋 Requirements
✏️ Design
⌨️ Coding
🔍 Testing
🔄 Maintenance
Let’s see how you can embed security in each of them.
For each step I will list the practices you can adopt to embed security properly. I will also separate between two levels: