ENH: Explicit garbage collection in MultiProc #2315
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With large numbers of processors, moderate (~1GB) increases in memory consumption can produce large spikes in memory committed by the kernel when a batch of processes is forked. For example, if 16 1-core nodes are executed in one round of the MultiProc scheduler, then 16x the virtual memory footprint of the main process is committed by the kernel - so a 500MB increase in main process size would correspond to an 8GB increase in committed memory. For systems which disallow over-commitment of memory, these brief spikes can produce crashes even if the maximum resident memory never approaches the system limits.
This PR adds garbage collection at two points to reduce the memory footprint before forking and after running nodes in the main process, which has the most potential to accumulate objects that need to be freed.
Changes proposed in this pull request
run_without_submitting
jobsHappy to argue/discuss alternatives. We found that unsetting
run_without_submitting
on some nodes that were a little more resource-intensive than we originally thought reduced spikes in main process memory consumption. (Would you say this is an accurate characterization?)