Merge pull request #4479 from pgjones/env_config

Allow loading of environment variables into the config

CHANGES.rst

@@ -55,6 +55,10 @@ Unreleased
 -   From Werkzeug, for redirect responses the ``Location`` header URL
     will remain relative, and exclude the scheme and domain, by default.
     :pr:`4496`
+-   Add ``Config.from_prefixed_env()`` to load config values from
+    environment variables that start with ``FLASK_`` or another prefix.
+    This parses values as JSON by default, and allows setting keys in
+    nested dicts. :pr:`4479`
 
 
 Version 2.0.3

docs/config.rst

@@ -515,11 +515,14 @@ Or from a JSON file:
 Configuring from Environment Variables
 --------------------------------------
 
-In addition to pointing to configuration files using environment variables, you
+In addition to pointing to configuration files using environment
-may find it useful (or necessary) to control your configuration values directly
+variables, you may find it useful (or necessary) to control your
-from the environment.
+configuration values directly from the environment. Flask can be
+instructed to load all environment variables starting with a specific
+prefix into the config using :meth:`~flask.Config.from_prefixed_env`.
 
-Environment variables can be set in the shell before starting the server:
+Environment variables can be set in the shell before starting the
+server:
 
 .. tabs::
 
@@ -527,8 +530,8 @@ Environment variables can be set in the shell before starting the server:
 
       .. code-block:: text
 
-         $ export SECRET_KEY="5f352379324c22463451387a0aec5d2f"
+         $ export FLASK_SECRET_KEY="5f352379324c22463451387a0aec5d2f"
-         $ export MAIL_ENABLED=false
+         $ export FLASK_MAIL_ENABLED=false
          $ flask run
           * Running on http://127.0.0.1:5000/
 
@@ -536,8 +539,8 @@ Environment variables can be set in the shell before starting the server:
 
       .. code-block:: text
 
-         $ set -x SECRET_KEY "5f352379324c22463451387a0aec5d2f"
+         $ set -x FLASK_SECRET_KEY "5f352379324c22463451387a0aec5d2f"
-         $ set -x MAIL_ENABLED false
+         $ set -x FLASK_MAIL_ENABLED false
          $ flask run
           * Running on http://127.0.0.1:5000/
 
@@ -545,8 +548,8 @@ Environment variables can be set in the shell before starting the server:
 
       .. code-block:: text
 
-         > set SECRET_KEY="5f352379324c22463451387a0aec5d2f"
+         > set FLASK_SECRET_KEY="5f352379324c22463451387a0aec5d2f"
-         > set MAIL_ENABLED=false
+         > set FLASK_MAIL_ENABLED=false
          > flask run
           * Running on http://127.0.0.1:5000/
 
@@ -554,36 +557,51 @@ Environment variables can be set in the shell before starting the server:
 
       .. code-block:: text
 
-         > $env:SECRET_KEY = "5f352379324c22463451387a0aec5d2f"
+         > $env:FLASK_SECRET_KEY = "5f352379324c22463451387a0aec5d2f"
-         > $env:MAIL_ENABLED = "false"
+         > $env:FLASK_MAIL_ENABLED = "false"
          > flask run
           * Running on http://127.0.0.1:5000/
 
-While this approach is straightforward to use, it is important to remember that
+The variables can then be loaded and accessed via the config with a key
-environment variables are strings -- they are not automatically deserialized
+equal to the environment variable name without the prefix i.e.
-into Python types.
 
-Here is an example of a configuration file that uses environment variables::
+.. code-block:: python
 
-    import os
+    app.config.from_prefixed_env()
+    app.config["SECRET_KEY"]  # Is "5f352379324c22463451387a0aec5d2f"
 
-    _mail_enabled = os.environ.get("MAIL_ENABLED", default="true")
+The prefix is ``FLASK_`` by default. This is configurable via the
-    MAIL_ENABLED = _mail_enabled.lower() in {"1", "t", "true"}
+``prefix`` argument of :meth:`~flask.Config.from_prefixed_env`.
 
-    SECRET_KEY = os.environ.get("SECRET_KEY")
+Values will be parsed to attempt to convert them to a more specific type
+than strings. By default :func:`json.loads` is used, so any valid JSON
+value is possible, including lists and dicts. This is configurable via
+the ``loads`` argument of :meth:`~flask.Config.from_prefixed_env`.
 
-    if not SECRET_KEY:
+When adding a boolean value with the default JSON parsing, only "true"
-        raise ValueError("No SECRET_KEY set for Flask application")
+and "false", lowercase, are valid values. Keep in mind that any
+non-empty string is considered ``True`` by Python.
 
+It is possible to set keys in nested dictionaries by separating the
+keys with double underscore (``__``). Any intermediate keys that don't
+exist on the parent dict will be initialized to an empty dict.
 
-Notice that any value besides an empty string will be interpreted as a boolean
+.. code-block:: text
-``True`` value in Python, which requires care if an environment explicitly sets
-values intended to be ``False``.
 
-Make sure to load the configuration very early on, so that extensions have the
+    $ export FLASK_MYAPI__credentials__username=user123
-ability to access the configuration when starting up.  There are other methods
+
-on the config object as well to load from individual files.  For a complete
+.. code-block:: python
-reference, read the :class:`~flask.Config` class documentation.
+
+    app.config["MYAPI"]["credentials"]["username"]  # Is "user123"
+
+On Windows, environment variable keys are always uppercase, therefore
+the above example would end up as ``MYAPI__CREDENTIALS__USERNAME``.
+
+For even more config loading features, including merging and
+case-insensitive Windows support, try a dedicated library such as
+Dynaconf_, which includes integration with Flask.
+
+.. _Dynaconf: https://www.dynaconf.com/
 
 
 Configuration Best Practices
@@ -603,6 +621,10 @@ that experience:
     limit yourself to request-only accesses to the configuration you can
     reconfigure the object later on as needed.
 
+3.  Make sure to load the configuration very early on, so that
+    extensions can access the configuration when calling ``init_app``.
+
+
 .. _config-dev-prod:
 
 Development / Production

src/flask/config.py

@@ -1,4 +1,5 @@
 import errno
+import json
 import os
 import types
 import typing as t
@@ -6,6 +7,13 @@ import typing as t
 from werkzeug.utils import import_string
 
 
+def _json_loads(raw: t.Union[str, bytes]) -> t.Any:
+    try:
+        return json.loads(raw)
+    except json.JSONDecodeError:
+        return raw
+
+
 class ConfigAttribute:
     """Makes an attribute forward to the config"""
 
@@ -70,7 +78,7 @@ class Config(dict):
     """
 
     def __init__(self, root_path: str, defaults: t.Optional[dict] = None) -> None:
-        dict.__init__(self, defaults or {})
+        super().__init__(defaults or {})
         self.root_path = root_path
 
     def from_envvar(self, variable_name: str, silent: bool = False) -> bool:
@@ -97,6 +105,70 @@ class Config(dict):
             )
         return self.from_pyfile(rv, silent=silent)
 
+    def from_prefixed_env(
+        self, prefix: str = "FLASK", *, loads: t.Callable[[str], t.Any] = json.loads
+    ) -> bool:
+        """Load any environment variables that start with ``FLASK_``,
+        dropping the prefix from the env key for the config key. Values
+        are passed through a loading function to attempt to convert them
+        to more specific types than strings.
+
+        Keys are loaded in :func:`sorted` order.
+
+        The default loading function attempts to parse values as any
+        valid JSON type, including dicts and lists.
+
+        Specific items in nested dicts can be set by separating the
+        keys with double underscores (``__``). If an intermediate key
+        doesn't exist, it will be initialized to an empty dict.
+
+        :param prefix: Load env vars that start with this prefix,
+            separated with an underscore (``_``).
+        :param loads: Pass each string value to this function and use
+            the returned value as the config value. If any error is
+            raised it is ignored and the value remains a string. The
+            default is :func:`json.loads`.
+
+        .. versionadded:: 2.1
+        """
+        prefix = f"{prefix}_"
+        len_prefix = len(prefix)
+
+        for key in sorted(os.environ):
+            if not key.startswith(prefix):
+                continue
+
+            value = os.environ[key]
+
+            try:
+                value = loads(value)
+            except Exception:
+                # Keep the value as a string if loading failed.
+                pass
+
+            # Change to key.removeprefix(prefix) on Python >= 3.9.
+            key = key[len_prefix:]
+
+            if "__" not in key:
+                # A non-nested key, set directly.
+                self[key] = value
+                continue
+
+            # Traverse nested dictionaries with keys separated by "__".
+            current = self
+            *parts, tail = key.split("__")
+
+            for part in parts:
+                # If an intermediate dict does not exist, create it.
+                if part not in current:
+                    current[part] = {}
+
+                current = current[part]
+
+            current[tail] = value
+
+        return True
+
     def from_pyfile(self, filename: str, silent: bool = False) -> bool:
         """Updates the values in the config from a Python file.  This function
         behaves as if the file was imported as module with the

tests/test_config.py

@@ -38,6 +38,68 @@ def test_config_from_file():
     common_object_test(app)
 
 
+def test_from_prefixed_env(monkeypatch):
+    monkeypatch.setenv("FLASK_STRING", "value")
+    monkeypatch.setenv("FLASK_BOOL", "true")
+    monkeypatch.setenv("FLASK_INT", "1")
+    monkeypatch.setenv("FLASK_FLOAT", "1.2")
+    monkeypatch.setenv("FLASK_LIST", "[1, 2]")
+    monkeypatch.setenv("FLASK_DICT", '{"k": "v"}')
+    monkeypatch.setenv("NOT_FLASK_OTHER", "other")
+
+    app = flask.Flask(__name__)
+    app.config.from_prefixed_env()
+
+    assert app.config["STRING"] == "value"
+    assert app.config["BOOL"] is True
+    assert app.config["INT"] == 1
+    assert app.config["FLOAT"] == 1.2
+    assert app.config["LIST"] == [1, 2]
+    assert app.config["DICT"] == {"k": "v"}
+    assert "OTHER" not in app.config
+
+
+def test_from_prefixed_env_custom_prefix(monkeypatch):
+    monkeypatch.setenv("FLASK_A", "a")
+    monkeypatch.setenv("NOT_FLASK_A", "b")
+
+    app = flask.Flask(__name__)
+    app.config.from_prefixed_env("NOT_FLASK")
+
+    assert app.config["A"] == "b"
+
+
+def test_from_prefixed_env_nested(monkeypatch):
+    monkeypatch.setenv("FLASK_EXIST__ok", "other")
+    monkeypatch.setenv("FLASK_EXIST__inner__ik", "2")
+    monkeypatch.setenv("FLASK_EXIST__new__more", '{"k": false}')
+    monkeypatch.setenv("FLASK_NEW__K", "v")
+
+    app = flask.Flask(__name__)
+    app.config["EXIST"] = {"ok": "value", "flag": True, "inner": {"ik": 1}}
+    app.config.from_prefixed_env()
+
+    if os.name != "nt":
+        assert app.config["EXIST"] == {
+            "ok": "other",
+            "flag": True,
+            "inner": {"ik": 2},
+            "new": {"more": {"k": False}},
+        }
+    else:
+        # Windows env var keys are always uppercase.
+        assert app.config["EXIST"] == {
+            "ok": "value",
+            "OK": "other",
+            "flag": True,
+            "inner": {"ik": 1},
+            "INNER": {"IK": 2},
+            "NEW": {"MORE": {"k": False}},
+        }
+
+    assert app.config["NEW"] == {"K": "v"}
+
+
 def test_config_from_mapping():
     app = flask.Flask(__name__)
     app.config.from_mapping({"SECRET_KEY": "config", "TEST_KEY": "foo"})