Metrics for evaluating passkeys

Long-time reader Antonio sent me an article that has the following intriguing line:

Microsoft revealed today that 68 percent of all password sign ins fail. In other words, only 32 percent of all Microsoft users manage to sign in when they are prompted to do so when they use passwords.

This statistics is shocking but this line for me raises more questions than it provides answers.

The claim that two out of three attempts to use a password failed seems unrealistically high. (The author does not reveal the basis of this number.)

In my experience, most people don't log out after they log in. They think it's more convenient. Developers also think it's more convenient, and thus many application interfaces actively hide the log-out buttons. These users will only attempt log-ins if they unexpectedly got kicked off. They are unlikely to remember any password.

Even though I try to log off frequently, I still can't retain all passwords. I tend to forget passwords for those accounts I don't use often. For those who don't log off, it's a miracle if they could remember their passwords. For this segment of users, the password success rate is very low.

For users like me, who log in and log out constantly, the success rate should be much higher. But I suspect we are a small minority.

So the 32% password success rate may be explained by selection bias - most of the attempts are made by people who don't care to remember passwords.

What other group might contribute log-in attempts?

Of course, malicious actors. I have no idea whether the data analysts filtered those out, or, indeed, whether they are able to differentiate between a legitimate user mistyping a password, or a bad actor trying to guess the password. Let's assume they can't tell those apart. Then, it's not true that 32 percent of the time, legit users failed the password test. The real percentage depends on how much malicious activity there is.

The article pitches an alternative to passwords, known as passkeys. It makes a further claim:

Users who sign in with passkeys manage to do so successfully 98 percent of the time.

Ironically, this statistic makes me nervous. The purpose of user authentication is to stop imposters from entering one's account. Analogously, we put locks on our front doors to prevent strangers from entering our homes.

If the door lock salesperson boasts that their lock lets in 98% of entry attempts, do you feel convenienced or insecure?

I feel insecure, because I believe all services face a healthy amount of malicious activities so I expect a lower success rate.

I was chatting with Perplexity.ai about passkeys, and it offered another statistic - that attacks via stolen passwords have plunged as more users switch to passkeys. No kidding. Since passkeys don't use passwords, bad actors aren't going to need stolen passwords, so password stealing has crashed.

To properly measure this, the analysts must figure out how malicious actors would adjust their tactics. They can certainly try to steal passkeys, or session tokens. If these developers switched back to passwords, passkey theft would also collapse!

The point is that it's important to define the right metrics. Passwords and passkeys are security measures, and the highlighted metric concerns convenience, which may be negatively correlated with security.