Fix username/email case issue in login/signup

[shakti97]

I have verified that this pull request:

• has no linting errors (npm run lint)
• is from a uniquely-named feature branch and has been rebased on top of the latest master. (If I was asked to make more changes, I have made sure to rebase onto master then too)
• is descriptively named and links to an issue number, i.e. Fixes #123

Fixes #1314

[shakti97]

@catarak have a look and suggest required changes:)

[andrewn]

This code looks good.

One problem that this uncovers is that it's completely possible to create an account with the username "ABC" or email "SOMETHING@EXAMPLE.COM" but then it's impossible to log in with those credentials?

I think before merging we should check if there are any existing users that will suffer from this problem? And perhaps migrate the DB to use lowercase everywhere.

[catarak]

Thanks for working on this! The case where this would break would be for all of the users that have uppercase letters stored in the database. I think the way to fix this would be:

1. Add some code to this commit that checks when a user was created, to not apply the toLowercase() to these users. The date can be an environment variable.
2. Merge this PR.
3. Write a migration script that updates all of the users username and password to lowercase.
4. Run this script.
5. Remove the date check code and submit a PR.

[catarak]

I managed to figure out an alternative Mongo query that handles both cases... in the process of testing now!

[catarak]

I now see the error in this logic... trying to figure out another solution!

[catarak]

Did some more thinking about what the right behavior is here.

• Email address should be saved as all lowercase in database. We can run a migration script to update this.
• When logging in with an email, it is normalized to lower case and then queried with the database.
• Usernames are stored with their case preserved, but should be are case sensitive unique.
• When logging in with a username, it is case insensitive.

I'm not sure how to implement the username. In order to do case insensitive querying in MongoDB, you have to use a Regex, which you can't use with an index, or create a text index. Or you could store the username both case sensitive and the username in all lowercase.

[catarak]

This further supports my theory that building an authentication/authorization system is the hardest problem in software development.

[catarak]

Got a solution going that uses a RegEx search to lay out how this should actually function. It appears the RegEx search is slow there are other options to explore, but this at least lays out all of the places where the case sensitivity of the username/email matters.

[catarak]

I had a breakthrough and now case-insensitive searches are working! Stuff that's left to do:

• Write script to migrate all emails to lowercase. (This is technically unnecessary, but it will remove the necessity for case-insensitive searches for email, which is a nice plus)
• Make a plan for how to handle emails and usernames that are currently duplicated by case. I imagine this is a relatively small number.

[andrewn]

We could add statics on the User model called User.findByUsername() and User.findByEmail() that will perform the User.findOne().collation(...) query.

This avoids duplication and ensures we won't forget to call it. It also means those statics can have a comment about what collation is doing as it's not very obvious.

[catarak]

Thanks for the suggestion! Good idea.

[catarak]

I did some thinking about how to handle case-sensitive duplicates of email addresses currently in the database, and this is my plan of action:

1. Make a separate PR that just adds a case-insensitive index to User.emails, and handles all of the case sensitivity issues for just emails. Additionally, make sure users cannot create new usernames that already exist in the database with different case.
2. Write a script that iterates through all of the accounts with the same email (with different case), merges them to one account (the first one the user created), and then notifies the user via email.
3. Write a script that migrates all email addresses to be all lowercase.
4. Remove collation/case-insensitive handling for email addresses.
5. Handle username case sensitivity issues, and make a plan for dealing with duplicate usernames in the database.

WHEW.

[catarak]

Since I've now run the update script, I'm going to merge this into develop, so it gets into the next release and prevent further duplicate emails. I will figure out the email migration/how to handle duplicate usernames in another PR.

[catarak]

Just did a query, and it turns out there are 5449 usernames that have duplicates. I unfortunately don't really see a way to allow users to login with their username, case-insensitive, that wouldn't involve changing usernames for users and therefore possibly changing urls. For that reason, I may just leave usernames as case sensitive, but prevent future duplicates.