Linting and PEP8 compliance

[wesm]

• update docs with tools to use & that Travis is checking

I have been away from the codebase for a while and ouch, my eyes. I have used pyflakes or flake8 assiduously for Python development for a long time now and it seems that conventional style isn't being followed in the codebase:

$ flake8 pandas/core | wc -l
2047

It's a big codebase so adding a pre-commit hook might be too onerous, but can we get flake8 back to zero (with some well-placed # noqa marks on unintentionally-unused-imports)?

Secondly, perhaps we should add a Readability or Style label for these kinds of issues?

[jorisvandenbossche]

See also #8317 and #6248

So there has been some discussion before, and in principle we ask of PRs to be PEP8 compatible, only we do not really enforce it in practice.
To better enforce it on new code, something like checking for this in the test suite (eg matplotlib and scipy do this) is probably a better idea than manual reviewing.

[wesm]

Okay. IMHO the code smell has reached pretty intense levels. I suggest we engage in a cleanup to get flake8 passing cleanly. I see concerns in #6248 that it will break PRs and git blame, and this is true, but it's the price we pay for having not been more fastidious on an on going basis. Let's please stick to 80 characters per line.

[jreback]

So prob a lot my fault. I tend to use long lines. :)

[jreback-~/pandas] flake8 pandas/core/|wc
    1922   15211  134394
[jreback-~/pandas] flake8 pandas/core/ | grep 'line too long'|wc
     924    8316   66073
[jreback-~/pandas] flake8 pandas/core/ | grep 'missing whitespace' | wc
     351    2106   23052
[jreback-~/pandas] flake8 pandas/core/ | grep 'never used' | wc
      24     264    2135
[jreback-~/pandas] flake8 pandas/core/ | grep 'expected 2 ' | wc
     131    1048    8789

[max-sixty]

How about hooking a check into GitHub? A service such as Landscape or Code Climate. Many of these have a broader scope but could be restricted to checking style only.

If that were enforced for new commits, the library would get there over time...

[wesm]

I would be strongly +1 on adding a flake8 check to our Travis CI builds to make linting errors be failed builds. Any thoughts on how to proceed with the cleanup?

[shoyer]

I have not been impressed by the reliability of Landscape as a CI system.

That said, I'm +1 for giving this a try and taking a more stringent approach going forward. This should give me a little more leverage when reviewing @jreback's long lines with missing whitespace :).

[jreback]

hah, this turned out to be easy to fix in for example this PR: #11892

[wesm]

Perhaps to get our training wheels let's fix the style in pandas/core and add a flake8 pandas/core to Travis and see how it goes?

[jreback]

okie dokie (always wanted to write that)!

[wesm]

I'm not sure what's the best practice for using flake8 in CI; I've used prospector but it combines pylint with pep8 which is a little too draconian (pylint gets grumpy about a host of things).