feature request: Load multiple .env files from different paths, or additionally support .ini files

[Teklvtn]

Currently only one .env file is supported according to the source code, but sometimes the submodules of the project will have some custom variables that need to be set, and all these variables are not very well managed in one .env file, and it is easy to make changes to the wrong situation. For this reason, I think it should be managed in separate .env files

Consider multiple .env source reads, or dynamic .ini reads (this part of the dynamic allows the user to stitch the path to the ini file based on the module path)

[Teklvtn]

The specific scenario is a project that operates in sub-modules. There is a top-level parent class that is configured through Config, but this cannot realize the loading of global env and sub-module env. After reading the specific code, the specific env call will only call the data source when it is specifically instantiated in the Settings subclass, so it can support the feature I mentioned

[hramezani]

good idea. I think it shouldn't be too hard to implement

[Teklvtn]

good idea. I think it shouldn't be too hard to implement

yeah，In the data source configuration class, implement the env source in the form of a list, and in the read method, process multiple env sources, which should be fine

[hramezani]

After checking the code I realized pydantic-settings already have this feature. and you can have multiple env files.

take a look at this test

[Teklvtn]

After checking the code I realized pydantic-settings already have this feature. and you can have multiple env files.

take a look at this test

It looks like you need to have a ConfigDict variable in the Settings class, but I actually think that you can store the corresponding value of the corresponding variable in the Settings class. I don't know if ConfigDict can provide variable hinting, after all, code hinting is still quite important

[Teklvtn]

After checking the code I realized pydantic-settings already have this feature. and you can have multiple env files.

take a look at this test

Maybe I should provide a simple example to further illustrate my idea？

[Teklvtn]

It seems that the above test cases are not the functions I want to express. The following are the test cases I provided, hoping to help communicate

# import module
from pydantic import ConfigDict, BaseSettings
from pydantic.env_settings import SettingsSourceCallable
from typing import Tuple


# in main project
class ParentSettings(BaseSettings):
    project_home: str = '.'

    class Config:
        # the env_file should allow multi source file
        env_file = '.env'  
        # such as env_file = ['.env', 'sub_module.env']  sub_module.env usually a dynamically generated path
        env_file_encoding = 'utf-8'
        extra = 'allow'
    
        @classmethod
        def customise_sources(
            cls,
            init_settings: SettingsSourceCallable,
            env_settings: SettingsSourceCallable,
            file_secret_settings: SettingsSourceCallable,
        ) -> Tuple[SettingsSourceCallable, ...]:
            return (
                init_settings,
                env_settings,
                file_secret_settings
            )


# in sub module
class ChildSettings(ParentSettings):
    # i hope the var can auto load from sub module .env file by the parent class‘s Config class
    sub_module_var: str = '.'

The above is my specific usage scenario, because I am using it in the production environment, and the v2 version of pydantic is still in the development stage, so use v1 as an example

[martinherrerias]

I wrote a BaseSettings wrapper that allows subclasses to preserve their parent's .env file as fallback, and to override the path via a --config argument. It's not pretty, but it works... I think. If there is a cleaner way to achieve the same, I'd love to read about it.

import os
import sys
from pathlib import Path

from pydantic import Field
from pydantic_settings import BaseSettings, CliApp
from pydantic_settings.sources import DotenvType, ENV_FILE_SENTINEL


def _to_list(value: DotenvType | None) -> list:
    if not value or value == ENV_FILE_SENTINEL:
        return []
    if isinstance(value, (str, Path)):
        return [value]
    if isinstance(value, (set, tuple)):
        return list(value)
    if isinstance(value, list):
        return value
    raise ValueError(f'Invalid type: {type(value)}')


class ConfigBaseSettings(BaseSettings, extra='ignore'):
    """
    `pydantic_settings.BaseSettings` subclass that supports inheritance-based
    dotenv configuration, and a (configargparse-style) `config` option to set
    the `_env_file` attribute at runtime.
    """

    config: DotenvType | None = Field(None, description="Path to configuration file")

    @classmethod
    def _env_file_list(cls, cfg = None) -> DotenvType:
        """
        Returns a list of dotenv files from the class hierarchy, in order of
        inheritance, with the option to append additional files `cfg`.
        """

        if cfg is None:
            cfg = []
        else:
            cfg = _to_list(cfg)
            cfg.reverse()

        for c in cls.__mro__:
            if issubclass(c, BaseSettings):
                for f in _to_list(c.model_config.get('env_file')):
                    if f and not f == ENV_FILE_SENTINEL and not f in cfg:
                        cfg.append(f)
        cfg.reverse()
        return cfg

    def __init__(self, **kwargs) -> None:
        """
        Wrapper for BaseSettings.__init__ to add the --config file and the
        class-hierarchy's dotenv files to `_env_file`.
        """

        # Get config file directly, e.g. from CliApp.run
        env_file = kwargs.get('config', None) or kwargs.get('_env_file', None)

        if not env_file:
            # ... or get --config option from sys.argv / _cli_parse_args
            cli_parse_args = (
                kwargs["_cli_parse_args"] if kwargs.get("_cli_parse_args", None) is not None
                else self.model_config.get("cli_parse_args")
            )

            if cli_parse_args is True:
                cli_args = sys.argv
            elif isinstance(cli_parse_args, list):
                cli_args = cli_parse_args
            else:
                cli_args = []

            try:
                idx = cli_args.index('--config')
                env_file = cli_args[idx+1]
            except ValueError:
                pass

        # Add the class-hierarchy's dotenv files as backup
        kwargs['_env_file'] = self.__class__._env_file_list(env_file)
        if env_file:
            kwargs['config'] = env_file

        super().__init__(**kwargs)


# --- Example ---

TEMP_DIR = '/tmp/ConfigBaseSettings'
os.makedirs(TEMP_DIR, exist_ok=True)

def mkenv(name, contents) -> str:
    file = os.path.join(TEMP_DIR, name )
    with open(file, 'w', encoding='utf-8') as f:
        f.writelines(contents)
    return file

class Foo(ConfigBaseSettings, env_file=mkenv('foo.env','foo=foo.env')):
    foo: str

class Bar(ConfigBaseSettings, env_file=mkenv('bar.env','bar=bar.env')):
    bar: str

class FooJr(Foo, env_file=mkenv('jr.env','jr=jr.env')):
    jr: str

class FooBar(FooJr, Bar):
    pass


assert Foo().model_dump() == {'config': None, 'foo': 'foo.env'}
assert Bar().model_dump() == {'config': None, 'bar': 'bar.env'}

assert FooJr().model_dump() == {'config': None, 'foo': 'foo.env', 'jr': 'jr.env'}
assert FooBar().model_dump() == {'config': None, 'bar': 'bar.env', 'foo': 'foo.env', 'jr': 'jr.env'}

assert Foo(foo="direct").model_dump() == {'config': None, 'foo': 'direct'}

# Override foo on object instances
custom = mkenv('custom.env','foo=custom.env')

assert Foo(_env_file=custom).model_dump() == {'config': custom, 'foo': 'custom.env'}
assert FooBar(config=custom).model_dump() == {'config': custom, 'bar': 'bar.env', 'foo': 'custom.env', 'jr': 'jr.env'}

run = CliApp.run(FooBar, cli_args=['--config', custom])
assert run.model_dump() == {'config': custom, 'bar': 'bar.env', 'foo': 'custom.env', 'jr': 'jr.env'}