CI/PERF: Windows builds fail due to potential MemoryErrors

[mroeschke]

Debugging attempt: #47318

First PR exhibiting these failures: #47299

May point to a potentially memory leak in recent non-nano changes.

[jbrockmendel]

Did this get resolved or just papered over?

[mroeschke]

Papered over, but I can try undoing our workaround one of these days