Today I received an e-mail from a website I am using regarding a data breach that seems to have leaked the user database. The message was well written, and it prompted users to reset their passwords and enable 2FA. That’s great, right?
In this post I’d like to argue that blindly moving to password resets and calls for 2FA isn’t necessarily the best first step for such situations. There’s more that needs to happen in advance, otherwise you may even lower the security of your users.
Before getting started, I’d like to point out that I won’t be talking specifically about this incident, but in general about situations like this, since I also don’t know exactly what happened in that one case.
When attackers gain access to your systems and are able to exfiltrate your user database, it means that access was somehow gained through a vulnerability or weakness. This method may be repeatable. For example, if an SQL Injection attack was carried out, or some employee’s credentials were used, an attacker can keep using these until they are patched.
If you detect a data breach and immediately request users to reset their passwords, without identifying all the methods the attackers used to gain access to your systems, then they can always log in again tomorrow, and download another copy of your data. If you keep asking your users to reset their passwords multiple times, eventually they’ll get tired and either stop doing it, or move elsewhere. It doesn’t inspire confidence to request 5 identical actions for a single incident within a short amount of time.
It’s therefore important to make sure you understand exactly how attackers got in, patch all the issues they were able to exploit (address software vulnerabilities, rotate credentials, etc.) and only then move forward. It’s rarely effective to do a Hail Mary and try everything because you can’t figure out how they gained access.
Time is definitely running out, and you can’t take months, but if you don’t know what you’re dealing with, whatever you do won’t be effective. If you want an indication of approximately how much time you have, you can take a look at your password storage strategy. How much time would it take for 8 of Nvidia’s latest high end GPUs to crack the weakest passwords (bottom 1%? 5%?) allowed by your password policy using the (weakest) hashing algorithm you have? One second? One day? One month? Whatever that may be, don’t try to exhaust this timeframe, as the clock started when the breach was made, not when you noticed. And you may have had more in the past.
If access was gained to your system, you can’t rule out the fact that the attackers made changes, and didn’t just read information. You should consider your environment tainted. It’s extremely difficult to be certain that everything is safe and you can keep doing business as usual. No matter how many forensics experts you hire, it’s rarely a good idea to trust that machine or system anymore.
You need to build that system from scratch, hopefully using your existing
automated tools, and reload all configuration and code from your source control
(or worst case a trusted backup, whatever that may mean for you), as you can
never be sure what may have changed. Your code may have been altered, or an
application or system configuration file. Maybe there’s a new user account with
SSH enabled on your system? Or a new root password? Or a process that listens
to an unknown port. Or a more relaxed firewall rule. It could be a new kernel,
or boot option, or module. The possibilities are endless.
And what do you use for data? Do you trust the existing database, in whatever state it may be? Do you reload an old backup? Is this even valid business logic? Do you compare the current database state with a few old backups and look for suspicious changes? What changes are suspicious?
If you ask your users to change their password now, in a tainted system, it can actually weaken their security. With a database copy the attackers have to deal with (hopefully) (properly) hashed passwords. If you ask all your users to start logging in or submitting password change forms, you may be giving your attackers all the secrets completely unprotected. Imagine if there’s a backdoor intercepting all form submissions. This can be in the code, the configuration, or the system. It can read all fields and then send them externally over the network, store them in a file that’s accessible by external entities, or whatever. Now your call-to-action e-mail just turned into fuel for the credential stealing fire that’s burning. And until you actually clean that system, no matter how many times users rotate credentials, they’ll still be exfiltrated in plain text.
Stolen passwords aren’t an immediate threat if you have 2FA, right? It buys you more time to reset these passwords and protects accounts in the meantime. Right?
Unfortunately not. At least not for the most common ways of doing Two Factor Authentication.
If your service is using HOTP or TOTP, which is that 6 or so digit code you have in an app on your phone or computer, then that’s probably useless. In all implementations that you’ll run into, this works by storing a secret in both the server and the client. You then do some math that involves this secret as well as the time or another number and you end up with the 6 digit code.
Guess where this shared secret is almost always stored: the user database. That means that if this database is copied, these symmetric secrets are also copied. Effectively now the attackers can generate the correct code of every user until the End of Time.
That means that if your service is using TOTP, the most common way to do 2FA, you need to reset that as well. All your users have to unenroll and re-enroll back so a new secret can be generated for them.
And of course, if the system is still tainted or vulnerable, attackers can be intercepting and accessing these new symmetric secrets as well.
There are secure 2FA methods that wouldn’t be vulnerable to this, but you need to implement them, and then you need to move all your users to them. And make sure they still work and there’s no backdoor added to the code…
Data breaches happen to everyone. If you haven’t been affected yet, you either don’t know about it, or you will be in the future. It sure is hopefully rare, but you need to be ready for that regardless.
Make sure you have a plan for when it happens, keep this plan up to date with all the developments and changes in the code, industry standards, etc. and ensure that it’s actionable. Do you have all the tools, staff, training, etc. to carry it out in a moment’s notice? Does it cover everything? Does it have timelines? When does it start and when does it end? Who’s responsible for each part of it?
Without a process in place that works, you’ll be left researching and guessing when you barely have enough time to act. You may not have time to be informed, create tools, and you may do things as a quick reaction that can eventually end up hurting you.
If this blog post made you consider changing things for the better, please take a look at WebAuthn, which can be used as a strong 2FA mechanism (linked above), as well as passwordless login with Passkeys. It’s worth looking into and can minimize the impact of a future data breach.