Nipype refactoring

[satra]

• merge Node/MapNode/BaseInterface into a single construct
  • a workflow can be a node in the graph
  • a mapnode can support a workflow
  • file inputs automatically encode full paths
• expansion happens during execution, unlike currently where expansion happens before execution
  • this will allow better integration of mapnode-like expansion
  • also allow for much greater scalability
  • much better semantic control
    • allow sequence of nodes or a subgraph to run completely in memory without writing anything to disk
    • allow chunking workflows depending on execution environment
• better handling of rerun
  • when assuming that disks are immutable, we can greatly speed up re-execution by eliminating parts of graphs during the hash-checks
  • this will require an input hash and an output hash. in fact i have been thinking of breaking the input hash into hash([connection hash, parameter hash])
• make the workflow engine just depend on scipy and networkx
  • or even better be pure python if we can figure out a simple set of sparse matrix operations
• allow URLs as inputs and checking/caching of URLs
• Interfaces
  • all type checking happens within interfaces
  • implement some CWL ideas into interface specification (we have most covered)
  • read/write CWL from interfaces/workflows
  • support local and remote services
    • file i/o from/to a remote store (file system, ssh, s3, http(s), dropbox, gdrive, etc.,.)
    • interfaces as containers (docker, vagga, etc.,.), remote services
  • version matching for reproducibility
  • flexible and extensible type checking including file types, file content, etc.,.
• simpler workflow specification
  • simpler non-pythonic support (CWL, DSL, etc.,.)
  • possible FASTR like magic - wf.connect(a, 'out', b, 'in') --> b.in = a.out

[satra]

• workflows
  • full support of workflow patterns
  • specification and development of distributed computing of workflows on distributed data and compute engines
  • integrate provenance services into intelligent re/execution.
    • don't submit jobs that take seconds to run - instead group them with others as a single job.

[mwaskom]

Do you see this as being pre- or post- a 1.0 release?

[satra]

@mwaskom - don't know - leave it up to the community. so we may have a 1.0 release with current API, and this becomes 2.0 or we directly make this the target for 1.0.

[blakedewey]

I vote for this being the target for 1.0. I think these improvements plus what we have done with Python 3 support would be more than enough to push us over the 1.0 hump

[mwaskom]

It is a very ambitious sketch — it reads like a complete rewrite of how nipype works! I would vote for aiming for a stable version 1.0 before changing everything :)

[oesteban]

• workflows
  • Execution modes: add a 'dry_run' execution mode that checks the logic of the pipeline, regardless the required software is installed and assuming that all the involved interfaces generate the necessary outputs (maybe using empty files?).

[mgxd]

#1843

[satra]

@djarecka - this was setup before you joined, but a lot of these features are dependent on the new api.