Cleanup CI environment files after #27542

[TomAugspurger]

#27542 got CI passing again by removing matplotlib from a couple of the slow travis jobs.

We should verify that

1. We still have adequate matplotlib coverage (I think we do)
2. We can remove the pytest and pytest-xdist pins

Also monitor https://travis-ci.community/t/builds-often-fail-because-of-lack-of-resources-memory-cpu/4331 to see if travis brings back higher memory workers.

[WillAyd]

It looks the issue on the Travis side was cleaned up a few hours ago, and the pytest stuff I think should be a non-issue so long as we pick up the latest (i.e. pytest >5 AND pytest-xdist 1.29)

I'll push a cleanup...let's see

[jreback]

k great

we also need to push to 0.25.1