Revisiting the Super Micro Story
Checking in on our forecasts from “The Big Hack”
An extraordinary claim was published by Bloomberg in October of 2018:
The attack by Chinese spies reached almost 30 U.S. companies, including Amazon and Apple, by compromising America’s technology supply chain, according to extensive interviews with government and corporate sources.
I immediately organized a panel of 22 security professionals to elicit their beliefs about potential future events presented by this journalism. This resulted in a panel estimated belief of 44.82% (YES) that certain events would take place by January 1st 2020. If these events took place, it would largely confirm the article’s claims.
That time window has passed! Happy New Year. 🎉
The Forecast Recapped
As discussed in the previous essay… any of the following four events needed to occur between October 2018 and January 1st 2020 for the article to confirm:
- Official and on the record confirmation of the incident from any Amazon, Apple, or Super Micro representative.
- Official and on the record confirmation by an unnamed Bloomberg victim company that is linked to the attack.
- Indictment or confirmation of the described incident from a government institution.
- An officially published hardware forensic analysis of the described chip from a security vendor confirms this incident.
This forecast acted as a point in time measurement towards understanding the risks presented by the journalists. A greater confidence (YES) published by the panel would suggest greater urgency to take action based on the report. Alternatively, a lower confidence (NO) from the panel would suggest incredulity towards the reports claims.
The Result: “NO”
I can’t find any of these four events taking place within the time frame and I’m happy to be corrected and pointed to anything that says otherwise. This means I’d judge the outcome of the forecast as a No which had a panel belief associated of 55.18% of occurring.
This gives the panel a Brier Score of 0.40176648.
More on Brier Score at Simple Risk Measurement
The panelists I use have been useful for uncertain situations and have performed better than guesses. This is often not the case for predictive risk measurements according to easily reproduce-able decision science research. This is a small reason why I am a big fan of structured expert forecasting.
Examining Bloomberg’s claims as a hypothesis
Bloomberg’s claim is testable if viewed as a hypothesis. We can develop evidence that suggests aspects of the hypothesis as true or false. However, a claim of “was breached” is a very different standard than “wasn’t breached” and these need to be discussed individually.
You may compare this to a weather forecast. If you received a 50% Rain forecast today… you wouldn’t be able to confirm it until tomorrow. However, the measurement was useful today in making decisions under uncertainty… even before the error is calculated with an actual outcome.
The forecast was evidence that suggested that an outcome of rain was entirely possible. It influences your decision making as a result. Bring an umbrella, even if I might not use it.
Instead of rain, or no rain… our forecast was designed to proxy the truthfulness of the journalism. If the journalism was true, certain follow up events were expected to occur soon after.
WAS BREACHED
The standard of evidence to prove a breach occurred is a very low bar to hit, compared to its alternative.
The proof that Super Micro was breached so far comes only from what we can draw from their article. Despite having the lower bar to meet… the journalist’s reliance on anonymous sources were a source of uncertainty for the security community. This, of course, inspired me to tackle a risk measurement.
WASN’T BREACHED
Any claim that something wasn’t breached is a certification of absence. That’s hard.
We can’t prove that Super Micro wasn’t breached as Bloomberg suggests. There is extreme difficulty (or, impossibility) in proving the absence of a data breach. A “no breach” claim is an extremely difficult hypothesis to confirm and is often why we are left to hear no evidence of a breach by PR teams as opposed to having them certify for the negative claim. Even in the unlikely case of fraudulent Bloomberg journalism… it’s still possible the underlying events occurred exactly as written, or partially.
We can only gather evidence that suggests this claim is partially or entirely false and we can’t certify that it’s false. Forecasting offers a novel way of developing this evidence that can at least produce a suggestion.
The Bloomberg hypothesis invited predictions to confirm it. Any properly testable claim or hypothesis has a quality that invites predictions. In this case, we (as a panel, and as a security community) predicted a list of potential events that we would expect to see over the next year that would suggest confirmation of the Bloomberg hypothesis. Those events were listed as the four points above, and none of them occurred. This evidence is encoded in the quantitative forecast produced by the panel.
Interpretations
The panelist odds were about even at 55% NO to 45% YES. Which is near total uncertainty (50/50). The track record of the panels I have organized so far has been useful enough for me to be influenced by their aggregated beliefs as useful information for decision making and risk measurement when no other sources of measurement are available.
It suggested to me: be wary of Bloomberg’s claims. If the forecast produced a number above 90%, I might suggest a plan of action to the security teams I work with who rely on the companies mentioned as victims. However, the uncertainty produced was not enough to push me to suggest any drastic and urgent actions to anyone. Personally, Heartbleed and Spectre/Meltdown caused me to drive more urgent action than this journalism.
However, someone directly holding a lot of Super Micro gear might be convinced to take immediate action, even with a much smaller estimate. A measurement and the decisions that result are very different concepts. Depending on who is reading the forecast, they might bring an umbrella… or unplug all of their Super Micro Hardware.
The Panel
The conclusion of this forecast adds one more data point of evidence informing the reliability of the panel now that the forecast has expired. Additionally, the absence of these four events occurring is also informative to a future panel about this journalism, if we wanted to follow up again.
The panel’s uncertainty is not any certification about the truthiness of the journalism either way. Any of these events can still occur in the future, which would suggest aspects of this breach claim are true. The long range forecast we created gave us an immediate measurement based on expert judgement as to whether we should act quickly based on the journalism.
Ryan McGeehan writes about security on scrty.io
