regex=False as default behavior, and expand regex disabling options

[davclark]

I was very happy to see pr #5879, allowing disabling the auto-regex feature for functions in the .str namespace. Doing something about this has been on my todo list for a while. Automatic promotion of strings to regular expressions is one of the two big points of difficulty for beginners using pandas (in my experience anyway). The most obvious problem comes up when you have literal dollar signs you wish to operate on.

I'd like to argue that the faster, less confusing regex=False be the default.

This is a trivial thing to do, and if folks are willing to accept it, I'm happy to submit a pull request.

I also ran into this problem when using unicode delimiters with the read_* family of functions:

ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators; you can avoid this warning by specifying engine='python'.

This is a another case where we're losing performance and increasing confusion. So, I'd like to expand this option elsewhere, and could submit this all as one pull request, or two (if it seems reasonable).

[jreback]

Well you are suggesting 2 things:

• changing str.contains default, which I think is an appropriate change (in 0.15)
• doing something? with the fallback of read_csv when specifying more than a single character deliminter. This is well doced and the error message is appropriate. What are you suggesting?

[davclark]

I agree that there are two changes - but they stem from common underlying concerns.

I was also confused - I thought a general shift to allow disabling regex compilation was enabled in pandas.core.strings. But for example, str_replace still uses this logic:

use_re = not case or len(pat) > 1 or flags

I agree that the behavior of read_csv is well documented and the error mesage is appropriate for, e.g., read_csv. (That said, the docs are a bit hard to navigate for newcomers... but that's a totally separate issue.) But the reasoning about efficiency and "least surprisingness" applies just as well here, I think. In particular, a single multi-byte unicode character triggers the switch to regexps, which is inefficient (and confusing to a newbie).

The auto-regexp-conversion logic seems parallel, and seems like a high-level design decision that should be applied across the library. And as you can see from the above, the high-level discussion ends up being more far reaching than I thought (there are several functions like str_replace that still only have auto-regexp).

In particular, I absolutely DO NOT want to change str_contains to stick with string matching by default, while other core.string functions continue to auto-convert patterns to regexps. Consistency is more valuable than one function being more efficient / less surprising.

But let me know if I should split the issue! As I said, these aren't hard changes, and I'm happy to do pull request(s).

[jreback]

ok, so let's split this issue.

In this one why don't you put checkboxes in the top for those functions that need changing the default to regex=False (most in strings, but maybe not all)

If I read you correctly, then you are saying that a multi-BYTE character triggers python mode. This could be a limitation of the C-parser which doesn't just works on the encoded bytes (and so multi-byte appears as more than 1 character). It is pretty non-trivial to use a multi-character separate in the parser (that said it could be implemented). related : #2667, related: #7470

[davclark]

I completely lost track of this issue, and feel I should probably close it at this point... this functionality has been in place long enough that I suspect changing it is more trouble than it's worth. I'm not sure how I missed the request to add the checkboxes to the issue, so please accept my apologies for dropping the ball there!