OneLogin Breach (2017) Retrospective

(draft / comment if inaccurate / work in progress )

Many IT teams were up late last night working on OneLogin’s announcement of a security breach.

From an incident responder’s point of view, these are some of the action items you’d have taken away from it. This is mostly pulled from the support article they released for customers to mitigate their risk.

This is not meant to replace that support article. Please use their documentation as a source of truth. Will make additions / corrections throughout the day as I talk to more teams working through rotation snags.

This is merely some observations for incident handlers from personal notes in the last night of supporting some response efforts. Some of the mitigation items in the support article are not discussed here.

Attack was AWS and database centric, beginning around May 31, 2am, PST.

Notes from a customer update sent out today:

“a threat actor obtained access to a set of AWS keys and used them to access the AWS API from an intermediate host with another, smaller service provider in the US.”

“Evidence shows the attack started on May 31, 2017 around 2 am PST.”

Through the AWS API, the actor created several instances in our infrastructure to do reconnaissance.

“OneLogin staff was alerted of unusual database activity around 9 am PST and within minutes shut down the affected instance as well as the AWS keys that were used to create it.”

The threat actor was able to access database tables that contain information about users, apps, and various types of keys.

“We cannot rule out the possibility that the threat actor also obtained the ability to decrypt data.”

Rotate SAML certificates that were compromised.

Every application you’ve added into OneLogin is integrated via a SAML certificate. Because these were compromised, OneLogin risks being entirely bypassed by this attacker regardless of your 2FA integration settings until you’ve rotated these certs. You need to generate new certificates for every application you’ve added right away.

Rotate your 2FA integration tokens.

Most OneLogin customer companies I work with are using DUO. DUO’s Security team is already advising that you rotate your integration token and secret, and whatever method of multifactor you’ve chosen should be rotated. In the case of DUO , this is very simple and does not orphan all of the 2FA clients or anything catastrophic.

So far I do not see an announcement that 2FA integration tokens were breached, but rotation is straightforward and doesn’t seem like an outage of any kind and helps mitigate unknown risk.

Work with employees to rotate passwords they may have stored in “Secure Notes”.

This is the second time OneLogin has had a “Secure Notes” breach. This type feature is widely used to store infrastructure secrets, and should be assumed to be in the wild. A predictable attacker’s first step would be to decrypt all of these notes and grep them for AWS infrastructure secret keys, if they were interested in any and all access they could manage instead of individual victims. There may be passwords to other products in here as well.

Encrypted notes can also be disabled as a feature, but this will not mitigate data currently in the wild.

As far as I know, there is no way to list your employees who have used this feature.

Rotate any non-SAML passwords being managed.

This feature is mostly described here and is meant to support shared applications that are not SAML managed.

Important detail: This means OneLogin stores encrypted and not hashed passwords on behalf of the organization so they can forward them to the application. These seem to be decrypt-able by the adversary.

These can be configured to come from a directory, which is why they advise pushing down a rotation from a directory product you’ve integrated, if you have this feature enabled to replicate them. Depends on your implementation.

Rotation includes going to every service you’ve added to OneLogin and changing the password from there.

Advise employees to rotate personal passwords they’ve stored.

The “Personal Apps” feature, if enabled, allows your employees to manage personal passwords outside of your purview. This is outside of your IT control. This could be their personal bank account, social media, email, etc. Because you can’t force these resets, this may require an awareness campaign to mitigate.

Assume compromise and investigate activity in important applications.

So far OneLogin has not mentioned that everyone should rotate passwords altogether, a kind-of-good sign that password material wasn’t breached, or stored safely. A cautious move would be to rotate passwords anyway, and is a typical after action in a breach like this.

Additionally, most applications that are enterprise-y enough to accept SAML logins, should have some method in their administrative panels or dashboards to look at recent activity or changes in an account. Assuming a full authentication bypass of OneLogin, an attacker would still be subject to a forensic audit trail in these applications, if available.

Demand better audit trails and account activity in apps you purchase.

In the future, when you’re discussing a license for an application with a sales team, make sure you express concern for situations around account takeover so you can investigate malicious activity in case of a service failure like this.

That is proper defense in depth, but is harder to manage when you have a constellation of apps and different perspectives on incident response capability. Ask for security features and audit trails in every product you buy.