CI: compile fails sometimes on ci because of undefined object

Currently, the ci is failing occasionally (increasing frequency I think) during the compile process on arm (not sure If this is also true for our other builds).

https://app.circleci.com/pipelines/github/pandas-dev/pandas/15347/workflows/ae3dcf3d-210a-4e9f-b6a9-ac0bdce57c98/jobs/46179

cc @mroeschke I think we had something similar in the past that was fixed by you?