[ENH] Migrating resource handler to Node level #1942
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR: - Sets the default `estimated_memory_gb` to 0 (most of the nodes of a workflow will never reach 1GB) - Makes two new arguments available to creating Nodes (n_procs and mem_gb), that will set those two values in the inner interface (no need to call node.interface.num_threads anymore). - Fixes a small error formating one Exception. TODO list: - Remove estimated_memory_gb and num_threads from the BaseInterface - Find a way to synchronize the Node level num_threads and the inner interface inputs (maybe a new metadata threads=True to identify these inputs?) - Resource handling of MapNodes (check if these changes would affect those).
|
@ccraddock may you have a look into these changes, and maybe give some feedback about the TODO list? Thanks! |
|
I will have a look over the next few days. Really busy at the moment. But at first, I would recommend to make the default memory something > 0, such as .5 or .25 |
Codecov Report
@@ Coverage Diff @@
## master #1942 +/- ##
==========================================
- Coverage 72.49% 72.47% -0.02%
==========================================
Files 1063 1063
Lines 54159 54151 -8
Branches 7811 7812 +1
==========================================
- Hits 39260 39246 -14
- Misses 13680 13685 +5
- Partials 1219 1220 +1
Continue to review full report at Codecov.
|
|
Up! cc/ @ccraddock |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR:
estimated_memory_gbto 0.25 (most of the nodes of a workflow will never reach 1GB)__interfaceinstead of_interface).TODO list: