Wire without cascading import

[ssheng]

First of all. Thanks for creating and maintaining this great library. I have a question regarding wire.

It seems like wire does a pkgutil.walk_packages and import all modules through importlib.import_module. This prevents the ability to lazily import dependencies as they are needed especially when there are large libs like numpy and grpcio involved. What is the recommendation to wire without cascading imports?

Thanks,
Sean

[rmk135]

@ssheng
Hi Sean,

Please check this section of the docs: https://python-dependency-injector.ets-labs.org/wiring.html#wiring-of-dynamically-imported-modules

You could also avoid using of pkgutil.walk_packages by wiring modules, not packages:

import one
import two

...

if __name__ == '__main__':
    container.wire(modules=[one, two])

Best,
Roman

[ssheng]

Thanks, @rmk135 for the quick response. register_loader_containers seems like the solution i'm looking for.

However, I'm getting the following error. I've tried brew install gdbm and conda install gdbm without success. I tried this in Python 3.6, 3.7, and 3.8. Python 3.9 gives a different error I guess we can discuss separately.

Traceback (most recent call last):
  File "/Users/ssheng/anaconda3/envs/bentoml/bin/bentoml", line 33, in <module>
    sys.exit(load_entry_point('BentoML', 'console_scripts', 'bentoml')())
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/pkg_resources/__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2846, in load_entry_point
    return ep.load()
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2450, in load
    return self.resolve()
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/ssheng/github/BentoML/bentoml/__init__.py", line 29, in <module>
    from bentoml.saved_bundle import load_from_dir, save_to_dir  # noqa: E402
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/dependency_injector/wiring.py", line 878, in exec_module
    super().exec_module(module)
  File "/Users/ssheng/github/BentoML/bentoml/saved_bundle/__init__.py", line 15, in <module>
    from bentoml.saved_bundle.bundler import save_to_dir
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/dependency_injector/wiring.py", line 878, in exec_module
    super().exec_module(module)
  File "/Users/ssheng/github/BentoML/bentoml/saved_bundle/bundler.py", line 27, in <module>
    import requests
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/dependency_injector/wiring.py", line 878, in exec_module
    super().exec_module(module)
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/dependency_injector/wiring.py", line 878, in exec_module
    super().exec_module(module)
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/urllib3/__init__.py", line 7, in <module>
    from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/dependency_injector/wiring.py", line 878, in exec_module
    super().exec_module(module)
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/urllib3/connectionpool.py", line 11, in <module>
    from .exceptions import (
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/dependency_injector/wiring.py", line 878, in exec_module
    super().exec_module(module)
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/urllib3/exceptions.py", line 2, in <module>
    from .packages.six.moves.http_client import IncompleteRead as httplib_IncompleteRead
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/dependency_injector/wiring.py", line 879, in exec_module
    loader.wire_module(module)
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/dependency_injector/wiring.py", line 859, in wire_module
    container.wire(modules=[module])
  File "src/dependency_injector/containers.pyx", line 256, in dependency_injector.containers.DynamicContainer.wire
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/dependency_injector/wiring.py", line 345, in wire
    for cls_member_name, cls_member in inspect.getmembers(cls):
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/inspect.py", line 342, in getmembers
    value = getattr(object, key)
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/urllib3/packages/six.py", line 91, in __get__
    result = self._resolve()
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/urllib3/packages/six.py", line 113, in _resolve
    return _import_module(self.mod)
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/urllib3/packages/six.py", line 82, in _import_module
    __import__(name)
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/site-packages/dependency_injector/wiring.py", line 878, in exec_module
    super().exec_module(module)
  File "/Users/ssheng/anaconda3/envs/bentoml/lib/python3.6/dbm/gnu.py", line 3, in <module>
    from _gdbm import *
ModuleNotFoundError: No module named '_gdbm'

Thanks,
Sean

[rmk135]

Hey @ssheng ,

The issue comes from wiring not properly working with six._MovedItems. Working on the fix.

[rmk135]

@ssheng I fixed the issue in version 4.32.2. Please upgrade. My apologies that you had to wait for the fix so long. Closing the issue. Feel free to comment.

[ssheng]

Thanks for the resolution, @rmk135. I upgraded to version 4.32.2 and confirmed the ModuleNotFoundError has been solved.

However, I ran into another issue which could be specific to the project I was working on. I was getting the following error message suggesting that multiprocessing start method has been set to fork. Using wire(), on the other hand, does not result in the same error.

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

I wonder if any logic in register_loader_containers has changed the start method, e.g. multiprocessing.set_start_method("fork")?

[rmk135]

Hi @ssheng ,

register_loader_containers does not set multiprocessing mode. I googled for this error and found this link: https://stackoverflow.com/questions/30669659/multiproccesing-and-error-the-process-has-forked-and-you-cannot-use-this-corefou. Try to set start method to "spawn" as suggested.

Also when you set multiprocessing start method you might need to initialize DI explicitly again. Check out this issue: #340

[ssheng]

Thanks, @rmk135. I managed to get it work with some refactoring. The issue was not related to spawn but Gunicorn.

I found out that the performance of a CLI tool we built using dependency-injector became significantly slower after using register_loader_containers. The code seems to perform a wire operation on each module import. Is the performance degradation expected?

CLI timing with wire:
bentoml list 1.81s user 0.59s system 128% cpu 1.866 total
bentoml list 1.82s user 0.55s system 129% cpu 1.835 total
bentoml list 1.79s user 0.53s system 129% cpu 1.797 total

CLI timing with register_loader_containers:
bentoml list 4.17s user 0.53s system 114% cpu 4.117 total
bentoml list 4.14s user 0.51s system 116% cpu 4.001 total
bentoml list 4.13s user 0.51s system 116% cpu 3.982 total

[rmk135]

Hi @ssheng , I've created a separate issue to address register_loader_containers() performance. See #464