Skip to content

Commit c319ca8

Browse files
Nic-Mapre-commit-ci[bot]wyli
authored
3482 add run API for common training, evaluation and inference (#3832)
* [DLMED] add config parser Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] simplify parse logic Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add more function tests Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix torchvision tests Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add standard run API Signed-off-by: Nic Ma <nma@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [DLMED] enhance update_config to support multiple items Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update API Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] simplify API Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] enhance doc and unit tests Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] clear API Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] simplify usage APIs Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] support root config Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix update reference typo Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update usage API Signed-off-by: Nic Ma <nma@nvidia.com> * revise APIs and docstrings Signed-off-by: Wenqi Li <wenqil@nvidia.com> * remove->discard Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [DLMED] minor change Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update code Signed-off-by: Nic Ma <nma@nvidia.com> * update usage Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [DLMED] update doc Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add end-to-end test Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update doc Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix flake8 Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix optional import Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] refine APIs Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix typo Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add support to only use part of the file to override Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add more tests Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update to latest Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to comments Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix flake8 Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to comments Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update `fire` supported dict arg Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add examples to doc-string Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] skip windows test for paths Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] skip windows Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update windows Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update command Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix windows test Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] construct class Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] restore design Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix args override Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix typo Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] test windows Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] test windows Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to comments Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] refactor config reading logic Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add doc-string Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add export and test Signed-off-by: Nic Ma <nma@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * python -m monai.bundle Signed-off-by: Wenqi Li <wenqil@nvidia.com> * fixes doc tests Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [DLMED] enhance test Signed-off-by: Nic Ma <nma@nvidia.com> * rel import Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [DLMED] refine macro Signed-off-by: Nic Ma <nma@nvidia.com> * fixes flake8 f401 Signed-off-by: Wenqi Li <wenqil@nvidia.com> * mv to monai.bundle Signed-off-by: Wenqi Li <wenqil@nvidia.com> * fixes split_path_id, update docstrings Signed-off-by: Wenqi Li <wenqil@nvidia.com> * fixes mypy Signed-off-by: Wenqi Li <wenqil@nvidia.com> * update cli, update according to the comments Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [DLMED] fix typo in doc-string Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to discussion Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix path_id match Signed-off-by: Nic Ma <nma@nvidia.com> * revise according to the comments Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [DLMED] update config item Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update patterns Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] enhance disabled Signed-off-by: Nic Ma <nma@nvidia.com> * special chars to utils Signed-off-by: Wenqi Li <wenqil@nvidia.com> * remove circular import Signed-off-by: Wenqi Li <wenqil@nvidia.com> * configreader -> configparser following cpython/configparser Signed-off-by: Wenqi Li <wenqil@nvidia.com> * printing Signed-off-by: Wenqi Li <wenqil@nvidia.com> * fixes typing Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [DLMED] unify "_name_" and "_path_" Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update to "_target_" Signed-off-by: Nic Ma <nma@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Wenqi Li <wenqil@nvidia.com>
1 parent 4b66746 commit c319ca8

23 files changed

+717
-152
lines changed

docs/requirements.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,5 @@ mlflow
2525
tensorboardX
2626
imagecodecs; platform_system == "Linux"
2727
tifffile; platform_system == "Linux"
28+
pyyaml
29+
fire

docs/source/bundle.rst

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,3 +32,7 @@ Model Bundle
3232
---------------
3333
.. autoclass:: ConfigParser
3434
:members:
35+
36+
`Scripts`
37+
---------
38+
.. autofunction:: run

docs/source/installation.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -190,10 +190,9 @@ Since MONAI v0.2.0, the extras syntax such as `pip install 'monai[nibabel]'` is
190190

191191
- The options are
192192
```
193-
[nibabel, skimage, pillow, tensorboard, gdown, ignite, torchvision, itk, tqdm, lmdb, psutil, cucim, openslide, pandas, einops, transformers, mlflow, matplotlib, tensorboardX, tifffile, imagecodecs]
193+
[nibabel, skimage, pillow, tensorboard, gdown, ignite, torchvision, itk, tqdm, lmdb, psutil, cucim, openslide, pandas, einops, transformers, mlflow, matplotlib, tensorboardX, tifffile, imagecodecs, pyyaml, fire]
194194
```
195195
which correspond to `nibabel`, `scikit-image`, `pillow`, `tensorboard`,
196-
`gdown`, `pytorch-ignite`, `torchvision`, `itk`, `tqdm`, `lmdb`, `psutil`, `cucim`, `openslide-python`, `pandas`, `einops`, `transformers`, `mlflow`, `matplotlib`, `tensorboardX`,
197-
`tifffile`, `imagecodecs`, respectively.
196+
`gdown`, `pytorch-ignite`, `torchvision`, `itk`, `tqdm`, `lmdb`, `psutil`, `cucim`, `openslide-python`, `pandas`, `einops`, `transformers`, `mlflow`, `matplotlib`, `tensorboardX`, `tifffile`, `imagecodecs`, `pyyaml`, `fire`, respectively.
198197

199198
- `pip install 'monai[all]'` installs all the optional dependencies.

environment-dev.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,8 @@ dependencies:
4242
- transformers
4343
- mlflow
4444
- tensorboardX
45+
- pyyaml
46+
- fire
4547
- pip
4648
- pip:
4749
# pip for itk as conda-forge version only up to v5.1

monai/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@
3939

4040
# handlers_* have some external decorators the users may not have installed
4141
# *.so files and folder "_C" may not exist when the cpp extensions are not compiled
42-
excludes = "(^(monai.handlers))|((\\.so)$)|(^(monai._C))"
42+
excludes = "(^(monai.handlers))|(^(monai.bundle))|((\\.so)$)|(^(monai._C))"
4343

4444
# load directory modules only, skip loading individual files
4545
load_submodules(sys.modules[__name__], False, exclude_pattern=excludes)

monai/bundle/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,5 @@
1212
from .config_item import ComponentLocator, ConfigComponent, ConfigExpression, ConfigItem, Instantiable
1313
from .config_parser import ConfigParser
1414
from .reference_resolver import ReferenceResolver
15+
from .scripts import run
16+
from .utils import EXPR_KEY, ID_REF_KEY, ID_SEP_KEY, MACRO_KEY

monai/bundle/__main__.py

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Copyright (c) MONAI Consortium
2+
# Licensed under the Apache License, Version 2.0 (the "License");
3+
# you may not use this file except in compliance with the License.
4+
# You may obtain a copy of the License at
5+
# http://www.apache.org/licenses/LICENSE-2.0
6+
# Unless required by applicable law or agreed to in writing, software
7+
# distributed under the License is distributed on an "AS IS" BASIS,
8+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
9+
# See the License for the specific language governing permissions and
10+
# limitations under the License.
11+
12+
13+
from monai.bundle.scripts import run
14+
15+
if __name__ == "__main__":
16+
from monai.utils import optional_import
17+
18+
fire, _ = optional_import("fire")
19+
fire.Fire()

monai/bundle/config_item.py

Lines changed: 28 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
from importlib import import_module
1818
from typing import Any, Dict, List, Mapping, Optional, Sequence, Union
1919

20+
from monai.bundle.utils import EXPR_KEY
2021
from monai.utils import ensure_tuple, instantiate
2122

2223
__all__ = ["ComponentLocator", "ConfigItem", "ConfigExpression", "ConfigComponent"]
@@ -160,25 +161,24 @@ def __repr__(self) -> str:
160161

161162
class ConfigComponent(ConfigItem, Instantiable):
162163
"""
163-
Subclass of :py:class:`monai.apps.ConfigItem`, this class uses a dictionary with string keys to
164+
Subclass of :py:class:`monai.bundle.ConfigItem`, this class uses a dictionary with string keys to
164165
represent a component of `class` or `function` and supports instantiation.
165166
166-
Currently, four special keys (strings surrounded by ``<>``) are defined and interpreted beyond the regular literals:
167+
Currently, two special keys (strings surrounded by ``_``) are defined and interpreted beyond the regular literals:
167168
168169
- class or function identifier of the python module, specified by one of the two keys.
169-
- ``"<name>"``: indicates build-in python classes or functions such as "LoadImageDict".
170-
- ``"<path>"``: full module name, such as "monai.transforms.LoadImageDict".
171-
- ``"<args>"``: input arguments to the python module.
172-
- ``"<disabled>"``: a flag to indicate whether to skip the instantiation.
170+
- ``"_target_"``: indicates build-in python classes or functions such as "LoadImageDict",
171+
or full module name, such as "monai.transforms.LoadImageDict".
172+
- ``"_disabled_"``: a flag to indicate whether to skip the instantiation.
173+
174+
Other fields in the config content are input arguments to the python module.
173175
174176
.. code-block:: python
175177
176178
locator = ComponentLocator(excludes=["modules_to_exclude"])
177179
config = {
178-
"<name>": "LoadImaged",
179-
"<args>": {
180-
"keys": ["image", "label"]
181-
}
180+
"_target_": "LoadImaged",
181+
"keys": ["image", "label"]
182182
}
183183
184184
configer = ConfigComponent(config, id="test", locator=locator)
@@ -195,6 +195,8 @@ class ConfigComponent(ConfigItem, Instantiable):
195195
196196
"""
197197

198+
non_arg_keys = {"_target_", "_disabled_"}
199+
198200
def __init__(
199201
self,
200202
config: Any,
@@ -214,52 +216,45 @@ def is_instantiable(config: Any) -> bool:
214216
config: input config content to check.
215217
216218
"""
217-
return isinstance(config, Mapping) and ("<path>" in config or "<name>" in config)
219+
return isinstance(config, Mapping) and "_target_" in config
218220

219221
def resolve_module_name(self):
220222
"""
221223
Resolve the target module name from current config content.
222-
The config content must have ``"<path>"`` or ``"<name>"``.
223-
When both are specified, ``"<path>"`` will be used.
224+
The config content must have ``"_target_"`` key.
224225
225226
"""
226227
config = dict(self.get_config())
227-
path = config.get("<path>")
228-
if path is not None:
229-
if not isinstance(path, str):
230-
raise ValueError(f"'<path>' must be a string, but got: {path}.")
231-
if "<name>" in config:
232-
warnings.warn(f"both '<path>' and '<name>', default to use '<path>': {path}.")
233-
return path
234-
235-
name = config.get("<name>")
236-
if not isinstance(name, str):
237-
raise ValueError("must provide a string for `<path>` or `<name>` of target component to instantiate.")
228+
target = config.get("_target_")
229+
if not isinstance(target, str):
230+
raise ValueError("must provide a string for the `_target_` of component to instantiate.")
238231

239-
module = self.locator.get_component_module_name(name)
232+
module = self.locator.get_component_module_name(target)
240233
if module is None:
241-
raise ModuleNotFoundError(f"can not find component '{name}' in {self.locator.MOD_START} modules.")
234+
# target is the full module name, no need to parse
235+
return target
236+
242237
if isinstance(module, list):
243238
warnings.warn(
244-
f"there are more than 1 component have name `{name}`: {module}, use the first one `{module[0]}."
245-
f" if want to use others, please set its module path in `<path>` directly."
239+
f"there are more than 1 component have name `{target}`: {module}, use the first one `{module[0]}."
240+
f" if want to use others, please set its full module path in `_target_` directly."
246241
)
247242
module = module[0]
248-
return f"{module}.{name}"
243+
return f"{module}.{target}"
249244

250245
def resolve_args(self):
251246
"""
252247
Utility function used in `instantiate()` to resolve the arguments from current config content.
253248
254249
"""
255-
return self.get_config().get("<args>", {})
250+
return {k: v for k, v in self.get_config().items() if k not in self.non_arg_keys}
256251

257252
def is_disabled(self) -> bool: # type: ignore
258253
"""
259254
Utility function used in `instantiate()` to check whether to skip the instantiation.
260255
261256
"""
262-
_is_disabled = self.get_config().get("<disabled>", False)
257+
_is_disabled = self.get_config().get("_disabled_", False)
263258
return _is_disabled.lower().strip() == "true" if isinstance(_is_disabled, str) else bool(_is_disabled)
264259

265260
def instantiate(self, **kwargs) -> object: # type: ignore
@@ -283,7 +278,7 @@ def instantiate(self, **kwargs) -> object: # type: ignore
283278

284279
class ConfigExpression(ConfigItem):
285280
"""
286-
Subclass of :py:class:`monai.apps.ConfigItem`, the `ConfigItem` represents an executable expression
281+
Subclass of :py:class:`monai.bundle.ConfigItem`, the `ConfigItem` represents an executable expression
287282
(execute based on ``eval()``).
288283
289284
See also:
@@ -308,7 +303,7 @@ class ConfigExpression(ConfigItem):
308303
309304
"""
310305

311-
prefix = "$"
306+
prefix = EXPR_KEY
312307
run_eval = False if os.environ.get("MONAI_EVAL_EXPR", "1") == "0" else True
313308

314309
def __init__(self, config: Any, id: str = "", globals: Optional[Dict] = None) -> None:

0 commit comments

Comments
 (0)