A risk based security project 📢

Driving an awareness project with a risk measurement ethos.

Let’s walk through the steps a fictional security manager takes to pursue a typical security awareness project. We will observe them as they target a risk for mitigation and subject it to risk measurement rather than launching into go-to awareness approaches arbitrarily.

This approach invites a wide range of quantitative and rigorous methods to better decompose a decision to mitigate a risk with the chosen methods.

This person found a risk they want to mitigate.

A security manager is hit with a bolt of insight. Their attention has been drawn to a potential gap at their current company:

Employees aren’t engaging with our security team.

The security team isn’t looped into discussions, receiving escalations, or asked to consult on tasks as frequently as the manager would expect.

🏝️The Security Island 🏝️

This manager desires better and more frequent interactions between the security team and company’s other internal teams to improve the security organizations early awareness of potential incidents. So far these interactions do not seem as common or fruitful as the manager’s experiences with other organizations in the industry.

Some incidents may not otherwise be surfaced if peer employees aren’t in the habit of escalating issues to the security team. It feels like a detection gap.

The manager now figures that spending some time and effort in this area might be beneficial in mitigating risk to the company.

They approach the problem while measuring their subjective, probabilistic beliefs in about the risks along the way. They do this in percentage form. This allows them to invite strategic discussion and compare the pros and cons of mitigations quantitatively.

1. 👨‍💻👩‍💻 Get some perspective from others.

The manager has conversations with other security team members, the IT help desk, and any other interested parties. Their goal is to develop a sense of frequency around potential risk scenarios and incident volume.

The manager writes a quick forecast to estimate: How many security team escalations do we see? They create a forecast like:

Forecast: With 90% confidence, the security organization would see:

• Between 5–20 escalated tasks.
• 0–2 SEV0 incidents produced from these tasks.
• 1–5 SEV1 incidents produced from these tasks.

…within the next month

The following are credible interval forecasts.

A “SEV” is just a formal classification of incident severity for for incident management. There is also P0 and likely many others used for incident management.

During this process, the manager learns that many of these are historically related to (spear) phishing attempts and other suspicious emails. Not only do we want to make sure that these are reported when received… we also want to avoid having employees fall victim to any type of attack that may come through one of these channels.

2. ✍️ Write goals to build awareness measurement infrastructure

The manager sets some goals to lay groundwork around the problem.

• ✅ Set up formal email, Slack, and task management channels for security escalation with oncall support.
• ✅ Announce these channels to the company.
• ✅ Measure and report on inbound escalations
• ✅ Analyze and report on suspicious (spear)phishing.
• ✅ Develop risk scenarios and forecast them (Covered next!)

The manager’s previous forecast is checked at month end. They find that 16 tasks were escalated, 1SEV0 and 3 SEV1 incidents resulted from these escalations. That results in the manager’s three forecasts at 90% confidence resulting in a Brier Score of .02.

I have some notes on scoring forecasts with Brier scores here and here. The manager wants to see values under 0.5 as much as possible, as these are all YES / NO forecasts.

The manager has now begun the accounting necessary to show that their judgement in estimating risks is reasonable and testable to demonstrate to someone who is curious about the strength of their judgement.

Accounting with Brier scores allows an individual to show a track record of being good at predictions as compared to coin flipping and is evidence of better judgement for risk assessment.

3. 🎯 Target specific risk scenarios.

The manager designs scenarios around a few areas of concern that they don’t ever want to see happen to their company. With their track record in hand, they look to make progress influencing future outcomes instead of just predicting them.

They write down some draft scenarios and shop them around with the team. After some discussion, they agree to three scenarios and codify them to be targeted with mitigations. They represent the outcomes a security awareness program might influence favorably, avoiding them from occurring.

Scenario A: A SEV0 security incident has triggered our disclosure policies.

Scenario B: An employee was successfully (spear) phished.

Scenario C: A SEV0 incident was escalated from an external party. (Law Enforcement, Twitter, Krebs, Press)

Now that these are on paper, the manager moves to forecasting how likely these scenarios are in their organization.

4. 🔭 Forecast these scenarios with a year long scope.

The manager writes down their forecasts for each scenario. They expect that in the next four quarters:

Scenario A: Has a 13% chance of a SEV0 regulatory disclosure occurring in the four quarter time frame. This scenario has not occurred yet in the existence of the company… but it has happened to its competitors, and certainly could happen here.

Scenario B: Will be seen 0 – 6 times (90% confidence). The interval forecast is wide. The manager has seen personalized spear phishing attacks occur in bursts and can’t say how well the company’s employees would resist it. The manager makes the interval wide to account for that uncertainty.

Scenario C: Has a 60% chance of occurring. The manager knows that external notification has happened on several occasions for SEV0 in this workplace for incidents in the past, not always turning into larger security events, or being security relevant at all.

5. 🛠️ Mitigate those scenarios with the awareness strategy.

The manager wants to mitigate these scenarios with an awareness strategy. They float the idea of a goal, or OKR, they could manage with their available resources and maintain throughout the year: If they are not run well operationally, it’s predicted that they won’t be effective.

Objective: Reduce the risk of a security incident with improvements to the relationship between our employee partners and the security team.

• Develop and deploy a propaganda campaign: Bathroom notices, customized posters, stickers, and related fun things designed to keep the security escalation path at the top of employee minds.
• Present at every new employee onboarding: Understand how to escalate incidents and better identify (spear)phishing over SMS, EMail, and other social platforms. Advertise the benefits of consulting from security engineers.
• Present at the monthly all-hands: Reward notable employee escalations and good behavior and highlight effective collaboration in engineering.
• License / Deploy FOSS phishing simulation software: Write and follow runbooks quarterly.
• Reward good behaviors: Gift Cards and Slack shoutouts when phishing simulations are caught.

6. 🔭 Forecast again, *assuming* new mitigations in place.

The manager then considers their previous forecasts. Assuming that they can accomplish the above goals, how would our scenarios look?

They forecast again with the assumption that they’ve successfully accomplished their goals. At the end of this time period, given that we completed our goals, what does our future outlook look like?

Scenario A (+ Mitigations): 11% (reduction of 2%!)
Scenario B (+ Mitigations): 0–3 at 90% confidence (improved from 0–6!)
Scenario C (+ Mitigations): 55% (reduction of 5%!)

These forecasts from the manager express a positive impact on these risks, given that the mitigations were effective.

They’re open to inspection, inviting questions like “Why do you think these mitigations are so effective for Scenario A?” or “Don’t you think a 2% reduction is a bit much?” or “would a different mitigation perform better?”.

The manager defends their reasoning as their degree of belief, or as the betting odds they are setting for the scenario. In the event of disagreement or, those parties asking those questions can submit their own forecasts for comparison as well and discuss numerically instead of conceptually.

Now that we’re all planned and measured, the manager implements these mitigations and sticks to them throughout the year like the good employee and manager they are.

7. 🔢 Score forecasts with historical events.

The manager reviews the incidents at year end, when their forecasts have expired and we can review history to double check the correctness of our forecasts.

Scenario A didn’t occur, resulting in a Brier Score of 0.0242.
Scenario B saw 1 spearphish succeed, though it didn’t turn into a larger scale incident. This results in a Brier Score of .02.

These scores are good. Remember, 0 is a perfect score and 2 is a full on prediction failure.

Scenario C, will have a worse score. An external party told the company about an S3 bucket they didn’t know was public. It didn’t result in any external notification, but it triggered an on-call engineer and automatically became a SEV0.

However, the forecast was assigned a relatively high expectation that this would occur at 55%. As a result, the Brier Scores weren’t harmed by much, at 0.405.

Had the manager more aggressively forecasted this scenario more towards 0% or 100%, the error would have grown significantly.

8. 🔁 Review: Are we still good at forecasting risk?

The manager is practicing good accounting for the risk forecasts they make by documenting their forecasts and periodically reviewing them for accuracy using a Brier Score. They review the six forecasts produced from this project:

Since these were all YES or NO forecasts (two options), the manager wants to see values under 0.5 as much as possible. As Brier Scores err towards or above 0.5, we are either working with more volatile risks or we are becoming worse at understanding our risks.

💥 Discussion: What if the manager faced opposition?

Nothing is ever this easy.

This rhetorical manager faced no friction along the way. I wrote this intentionally to isolate and demonstrate the measurement method when a project is moving along without much interference. Very idealistic.

There are a few places where this sort of project would usually break down, as most of us who do blue team work are familiar with in a typical workplace.

The go-to qualitative approaches to justify a security project are usually best to start with. A quantitative one, however, shows the work better than appealing to best practices, citing a checklist, or describing risks with qualitative language, which can often trigger a sense of FUD.

Quantitative methods offer opportunities to settle disagreement more precisely and objectively. Quantitative discussions can help avoid some qualitative aspects of debate. For example:

• A second opinion may have drastically disagreed with the forecasts the manager produced. A panel may be used for consensus instead of relying on one single person’s judgement or the other, needing to pick sides.
• Other mitigation strategies may have suggested a better approach than an awareness strategy. When everyone first agrees on the the well defined scenario, the conversation can at least stay centered on a specific risk while discussion mitigation options.
• If the manager isn’t trusted with their professional experience or judgement, a data gathering effort can be used to bolster the forecasts (i,e,, breach frequency data) and replace their expert judgement entirely if their judgement is truly not considered wise.
• Additionally, if the manager isn’t trusted with their professional experience or judgement, they can bolster their credibility with a track record of reliable predictions and calibration training. (link)

A final idealistic point: Security data is sparse and often incomplete. In many cases, information security risk is too conditional to precisely pinpoint using reference data alone. A foundation of probabilistic methods and decision science tools will help bridge that gap. I write about this here.

That said — while I write often about quant, pure risk based decision making will ultimately be a small part among many aspects of security influence in a workplace.

Conclusion

Using the quantitative forecasting approach outlined above, the manager has demonstrated their ability to estimate the probability or frequency of future events. They used this same skill to estimate the impact that various mitigation projects would have on these risks. Later, they demonstrated how correct their forecasts were. This shows that they are professionally concerned with their ability to assess risk and able to course correct if they’re off the mark.

Ryan McGeehan writes about security on scrty.io.

Thanks to Samantha Davison for a walk through on a typical awareness OKR, and Adrian Sanabria and Kelly Shortridge for feedback.