CI: unpin numpy and matplotlib

[fangchenli]

revert #35312 and #35358

[danieldjewell]

Does pyproject.toml also need to be changed? Currently, it appears locked to numpy 1.17.3 for python >= 3.8

[fangchenli]

Does pyproject.toml also need to be changed? Currently, it appears locked to numpy 1.17.3 for python >= 3.8

If I understand it correctly, what is specified in pyproject.toml is the minimum requirements.