Skip to main content

How to Measure Anything in Cybersecurity Risk, 2nd Edition

This is a green book cover with a big lock in the middle and wires behind it.

Book written by Douglas W. Hubbard and Richard Seiersen

Book review by Rick Howard 

Bottom Line

I recommend this nonfiction book for the Cybersecurity Canon Hall of Fame.

Review

The Cybersecurity Canon Committee inducted the first edition of this book into the Hall of Fame back in 2017. Steve Winterfeld did the initial book review and Bob Clark interviewed the authors at the time for the Cybersecurity Canon Gala awards show that year (See References below). The authors, Douglas Hubbard and Richard Seiersen, published a second edition in 2023 and I wanted to take a look at the updated material. I reached out to Richard Seiersen, a friend of mine now (he and I met at the gala and even presented together at the RSA Conference in 2018 on this topic) and asked him to summarize the update. Here's what he said.

  • A new and simpler "Rapid Risk Audit" along with the one-for-one substitution
  • New statistical methods for quick estimates
  • Updated research on the impact of data breaches
  • New Bayesian examples
  • Additional methods on decomposing risk by subsystem and simple adjustments for the effectiveness of controls
  • Simple methods for combining the estimates of multiple experts in a way that outperforms individual experts
  • New methods using the R statistics language
  • New guest contributors
  • A forward by Jack Jones himself!

All that's true. In terms of outline, they replaced Chapter 3 (Model Now) with a new chapter called The Rapid Risk Audit and sprinkled in new sections where needed:

  • A Taxonomy of Measurement Scales
  • More Hints for Controlling Overconfidence
  • Beyond Initial Calibration Training: More methods for improving subjective judgment
  • An Example from Little Data: Does Multifactor Authentication Work?
  • Other Ways Bayes Applies
  • More Advanced Modeling Considerations
  • Functional Security Metrics Applied: Boom!
  • Wait-Time Baselines
  • Security Metrics with the Modern Data Stack
  • Modeling for Security Business Intelligence
  • Integrating CSRM with the Rest of the Enterprise

And before I start to throw my opinions around about this edition, let me say up front that these guys have found some fantastic quotes about probability and risk forecasting to begin each chapter that brought me great joy. Here are three of my favorites:

  • "Bayesian inference is the reallocation of credibility across possibilities ... Credibility is synonymous with probability." --John K. Krushke's “Doing Bayesian Data Analysis”
  • "It is unanimously agreed that statistics depends somehow on probability. But, as to what probability is and how it is connected with statistics, there has seldom been such complete disagreement and breakdown of communication since the Tower of Babel." —L. J. Savage, American mathematician
  • "The most important questions of life are indeed, for the most part, really only problems of probability. —Pierre‐Simon Laplace, Théorie Analytiquedes Probabilités, 1812

The first edition of this book, and another Canon Hall of Fame book, "Measuring and Managing Information Risk: A FAIR Approach" published in 2014 by Jack Freund & Jack Jones, introduced me to the idea of better risk forecasting. The industry has been using Risk Matrix Heat Maps as a best practice since the early 1990s to convey cyber risk to senior leaders and I used to be one of those guys. But, as Hubbard and Seiersen point out in both editions, statisticians have written reams of research papers showing that heat maps are just bad science for this task.

For the uninitiated, a heat map puts all the bad things that can happen to your organization on an x-y coordinate system. The x-axis tracks how likely it is that the bad thing will happen from "unlikely" on the far left to "highly likely" on the far right. The Y axis tracks the potential damage from the bad thing from "not much" to  "existential threat to the business." The really dangerous risks float up and to the right of the chart. Heat map designers color code the matrix so that risks high and to the right are red, risks in the middle are yellow, and risks low and to the left are green. That's why they call it a heat map.

Out of all the reasons that heat maps are bad science, two strongly resonated with me. The first is that heat maps use ordinal scales (high, medium, low, likely, and unlikely) that nobody really understands. When I ask you what "Likely" means, do you think to yourself, "Well, that's almost a sure thing" or do you say to yourself, "Well, that's better than 50/50.” The research shows that even if I tell you what I think it means (close to 100%), your own personal bias kicks in and you might use your own number (50/50).

The second is even a bigger flaw.  A risk matrix doesn't provide business leaders a way to judge if some risk placed high and to the right is within their risk tolerance. The map represents the risk as a scary thing (fear, uncertainty, and doubt) that needs addressing, not as a business risk that leaders need to make a decision about.

For both of these reasons and others, the authors proved to me that the better chart for this task is something called a Loss Exceedance Curve and I have spent almost a decade trying to learn how to practically build these things myself. More on that project in a bit.

A Loss Exceedance Curve shows the probability of losses exceeding different values. It's another x-y coordinate system but this time, the x axis shows the estimated dollar losses from zero dollars on the very left to something very high on the right (like $500M for example). The y axis shows the probability. The curve snakes through the grid matching probabilities to the dollar amounts. You might see that there is a 30% chance that the company could lose $100K in the next year due to a cyber event. You might also see that there is a 5% chance of losing $1M.

Loss Exceedance Curves correct my two biggest objections to heat maps. Readers of the chart know exactly what the labels mean. They are precise. The impact is that leaders can make judgements regarding their own risk tolerance. CEOs might say that they can live with the 5% chance of losing $1M but that the 10% chance of losing $100K is too high. In that case, they would direct the CISO to bring that probability down.

The book's main thesis is that in order to build these charts, calculating probabilities and estimating dollar loss are two essential skills. They advocate for the idea of using the math principles behind the Bayes Algorithm for calculating probabilities, and they suggest using simple Monte Carlo simulations within a spreadsheet to estimate dollar losses.

The Bayes Algorithm is the underlying bedrock idea for the entire book. It is a statistical method that updates beliefs based on new evidence using conditional probability to make more accurate predictions. In other words, you make an initial estimate regardless of how broad it is using in-house experts. Over time, you collect more evidence that will allow you to adjust the estimate up or down based on the information collected. Each time you adjust the estimate, the answer brings you closer to the actual risk.

Note: For a deeper review of the history of Bayes' Algorithm and how it has been used in the past 200 years to solve highly complex problems, see Sharon Bertsch McGrayne's 2011 book, "The Theory That Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy." I summarized some of those stories in the appendix to my book called "Cybersecurity First Principles Appendix - Bayes Success Stories" (See References).

A Monte Carlo simulation is a computational technique used to model and analyze complex systems. You build a reasonable model that provides a random answer within the parameters you specify (like there is a 15% chance of losing between $100K and $500M). You run the model 10,000 times and collect the data. You use that data to build the Loss Exceedance Curve. The authors walk the reader through a practical example of just how to create a Loss Exceedance Curve and provide other more complex models on their book website.

I'm not a math guy and frankly, whenever a statistician starts throwing around the names of various distributions and why you should use one over the other (like Triangular, Binary, Normal, Lognormal, Beta, and Power Law), I want to throw myself out the window. But even I understood the author's explanation of why you would use a  lognormal distribution over a normal distribution for these calculations; essentially, the lognormal distribution can't generate a zero or negative amount, but it has a tail to the right that allows for the possibility of extremely large outcomes. The great news about this is that spreadsheets do all the work for you. You just have to use the correct formula.

Hubbard and Seiersen spend time explaining how to improve the ability of experts to make better risk forecast estimates. They call it calibration. They recall the research done by  Dr. Philip Tetlock in his 2005 book, "Expert Political Judgment: How Good Is It? How Can We Know?" about, in part, why some forecasters are better than others. I would also point readers to Tetlock's follow-on 2015 Cybersecurity Canon Hall of Fame book, "Superforecasting: The Art and Science of Prediction" that refines the thought. Hubbard and Seiersen recommend various strategies to make anybody better at this task and provide practice exercises to see how good you are.

They quote the famous British statistician George E. P. Box, “All models are wrong, but some are useful”. The rest of the book is dedicated to making their recommended models more useful.

That is why the Canon Committee selected the first edition of the book for the Hall of Fame. It is a new and better way to think about risk for your organization. The second edition expands on those ideas. I endorse completely the committee's hall of fame induction of the book. That said, I do have some quibbles.

First, this is not an easy read. It's not a book where you grab the audio version, take the dogs around the block, and hope to learn the bulk of it. You have to sit with it, read it and re-read it, and try the examples. The authors assume you know the math and don't waste a lot of time explaining things. You can tell they tried to improve on that between the two editions, but it's still dense if you're not a math person comfortable with probabilities. I'm not, so I had to wade through it. I got most of it but the advanced stuff in the later chapters was beyond me.

Second, the supplemental materials were not that helpful. I applaud any author that supplies supplemental material on a web page somewhere and Hubbard and Seiersen do that in spades. They provide numerous spreadsheets with practical examples of the models they describe in the book. But, they offer no explanation of how they work. The book refers to them but doesn't explain them and, like I said, I'm not a math guy, a spreadsheet guy, nor am I a probabilities guy. The spreadsheets didn't match exactly what they were talking about in the book either so they weren't that much help to me. Your mileage may vary.

Lastly, my biggest criticism of the book, and for "Measuring and Managing Information Risk" too, is that I kept waiting for the chapter at the end that showed how to practically use these techniques in the real world. Don't get me wrong, they show how to build a loss exceedance curve (and I spent the 2023 holiday break learning how to do it myself) and how to make the models better, but they didn't spend any time describing how you might use these techniques to build a better deck for the board or senior leadership. Once you get done with all of the math, how does the answer help you convey risk to the leaders of the company? That chapter doesn't exist. So, I wrote it myself (See Chapter 6, Cybersecurity First Principles in the references below).

Those complaints aside, Hubbard and Seiersen have written a must-read cybersecurity book. The second edition has only made the material better. If you're still using heat maps to convey risk to senior leadership and you haven't read this book yet, you have a giant hole in your education. Stop what you're doing right now and read this book.