Skip to content

ENH: Explicit garbage collection in MultiProc #2315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 3, 2017

Conversation

effigies
Copy link
Member

With large numbers of processors, moderate (~1GB) increases in memory consumption can produce large spikes in memory committed by the kernel when a batch of processes is forked. For example, if 16 1-core nodes are executed in one round of the MultiProc scheduler, then 16x the virtual memory footprint of the main process is committed by the kernel - so a 500MB increase in main process size would correspond to an 8GB increase in committed memory. For systems which disallow over-commitment of memory, these brief spikes can produce crashes even if the maximum resident memory never approaches the system limits.

This PR adds garbage collection at two points to reduce the memory footprint before forking and after running nodes in the main process, which has the most potential to accumulate objects that need to be freed.

Changes proposed in this pull request

  • Garbage collect before submitting jobs
  • Garbage collect after run_without_submitting jobs
  • PEP8/pyflakes cleanup

Happy to argue/discuss alternatives. We found that unsetting run_without_submitting on some nodes that were a little more resource-intensive than we originally thought reduced spikes in main process memory consumption. (Would you say this is an accurate characterization?)

@oesteban
Copy link
Contributor

@effigies would you like to place this in the 0.14.1 milestone?

@oesteban oesteban merged commit 331e939 into nipy:master Dec 3, 2017
@effigies effigies deleted the garbage_collect branch December 4, 2017 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants