A couple of months back, I was talking to a fellow peer about security in our respective companies. Quite a formal exchange where we discussed the current challenges, our respective careers and so on.
At some point in the exchange, he shared what he regarded as a one of his great accomplishments. He said, and I quote, “I helped this company achieve a 2.9 average score on the ISO framework. We really started from zero.”
As soon as he uttered these words, I was instantly taken back a few years. Back to my auditing career where Fortune 500 companies would often hire us to evaluate their so-called security maturity. Big fat checks thrown to the Big-4 who happily stormed in with their suits and PowerPoint decks to help the CISO make sense of the confusing world of cybersecurity.
They’d show up with a list of hundred security controls compacted in a dreadful Excel sheet. For each security control, the auditors would evaluate the company’s performance according to the following grid:
One point for roughly following the control with no documented process. (e.g, reviewing access rights)
Two points for documenting the process (e.g. having an access rights policy)
Three points for controlling the process (e.g. auditing that access rights were properly reviewed).
Four points for following metrics and KPIs (e.g. following metrics and setting goals for access rights).
Apply this scoring to each of the 140-odd lines of the ISO 27001 and you end up with a quantitative maturity score and beautiful diagrams that nicely fit in any executive’s PowerPoint deck. The CISO could then confidently walk into their CEO’s office and explain their “data-driven” approach:
“We scored a 1.8 average in Access Control this year. Our goal is to push it to 3.1 in 2 years. We need $2M to make it happen.”
Of course the small detail that everyone seems to eagerly ignore in these simplistic equations is…well the attacker! Where is the attacker in all of this? How do they fit in that list of controls? How are they really impacted by that scoring ? Is a 2 point score on “9.2.4 Management of secret authentication information of users” good enough to stop industrial espionage? How about a disgruntled employee ? What does a single digit score even mean? There are thousands of servers, machines and systems each with their own settings and peculiarities. How do we synthesize a unique score based of these disparate entities? Are they all weighed equally ? Should they all be treated equally? Which of the 140 controls should you focus on in priority?
And probably most importantly…why the hell are we talking about an aggregate score anyway? An attacker does not care. They find that one unprotected machine and exploit the shit out of it. That’s the reality that we live in. They don’t care about that 3.1 average…and neither should you.
I am not criticizing the ISO 27001 per se, that will probably come in another post. I am merely pointing out the idiosyncrasies of treating information security as a point-based system or as a compliance program where the goal is to satisfy some fixed list of controls. We’re effectively perverting what little benefits controls bring to the table. They are great at providing starting points, maybe even reaching exhaustivity. Instead we’re making them the final target in a classic manifestation of Goodhart's Law
when a measure becomes a target, it ceases to be a good measure
Compliance programs, regulatory texts and security frameworks were supposed to help enforce security in companies. Yet they ended up diluting it as they hijacked its meaning and purpose.
The goal of security is not compliance. Never was. Never should be. The goal of information security is to fend off attackers while boosting the business. That’s the mission. The objective. Any action, metric or measurement should help answer that question.
How many attack scenarios can reach critical assets? How many security layers are stacked in front of them? How efficient are they?
Yes, one can take inspiration from this or that framework to map out their areas of improvement, but, one should never substitute that initial goal for something else just because it’s easier to track and measure. Otherwise the focus gradually shifts in the wrong direction.
You forget about your initial goal—stopping attackers—and start haggling meaningless points on control “13.1.2 Security of network services”. You spend too much time on policies and documents just to scrap that additional 1 point increase to reach your KPI targets. All the while, an attacker is happily surfing on that ElasticSearch freely open on the Internet…
Don’t worry though, it’s just one system. It won’t affect the total score ;)