From 428d9430bc69031150a8d50ac8af17cc8082ea4e Mon Sep 17 00:00:00 2001 From: "Maxim G. Ivanov" Date: Thu, 9 Feb 2023 09:38:57 +0700 Subject: [PATCH 1/6] Fix command-line formatting --- docs/patterns/packages.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/patterns/packages.rst b/docs/patterns/packages.rst index 13f8270e..3c7ae425 100644 --- a/docs/patterns/packages.rst +++ b/docs/patterns/packages.rst @@ -66,6 +66,8 @@ To use the ``flask`` command and run your application you need to set the ``--app`` option that tells Flask where to find the application instance: +.. code-block:: text + $ flask --app yourapplication run What did we gain from this? Now we can restructure the application a bit From dca8cf013b2c0086539697ae6e4515a1c819092c Mon Sep 17 00:00:00 2001 From: David Lord Date: Wed, 8 Feb 2023 17:30:53 -0800 Subject: [PATCH 2/6] rewrite celery background tasks docs --- docs/patterns/celery.rst | 270 +++++++++++++++++++++++++++++---------- 1 file changed, 200 insertions(+), 70 deletions(-) diff --git a/docs/patterns/celery.rst b/docs/patterns/celery.rst index 228a04a8..a236f6dd 100644 --- a/docs/patterns/celery.rst +++ b/docs/patterns/celery.rst @@ -1,105 +1,235 @@ -Celery Background Tasks -======================= +Background Tasks with Celery +============================ -If your application has a long running task, such as processing some uploaded -data or sending email, you don't want to wait for it to finish during a -request. Instead, use a task queue to send the necessary data to another -process that will run the task in the background while the request returns -immediately. +If your application has a long running task, such as processing some uploaded data or +sending email, you don't want to wait for it to finish during a request. Instead, use a +task queue to send the necessary data to another process that will run the task in the +background while the request returns immediately. + +`Celery`_ is a powerful task queue that can be used for simple background tasks as well +as complex multi-stage programs and schedules. This guide will show you how to configure +Celery using Flask. Read Celery's `First Steps with Celery`_ guide to learn how to use +Celery itself. + +.. _Celery: https://celery.readthedocs.io +.. _First Steps with Celery: https://celery.readthedocs.io/en/latest/getting-started/first-steps-with-celery.html -Celery is a powerful task queue that can be used for simple background tasks -as well as complex multi-stage programs and schedules. This guide will show you -how to configure Celery using Flask, but assumes you've already read the -`First Steps with Celery `_ -guide in the Celery documentation. Install ------- -Celery is a separate Python package. Install it from PyPI using pip:: +Install Celery from PyPI, for example using pip: + +.. code-block:: text $ pip install celery -Configure ---------- -The first thing you need is a Celery instance, this is called the celery -application. It serves the same purpose as the :class:`~flask.Flask` -object in Flask, just for Celery. Since this instance is used as the -entry-point for everything you want to do in Celery, like creating tasks -and managing workers, it must be possible for other modules to import it. +Integrate Celery with Flask +--------------------------- -For instance you can place this in a ``tasks`` module. While you can use -Celery without any reconfiguration with Flask, it becomes a bit nicer by -subclassing tasks and adding support for Flask's application contexts and -hooking it up with the Flask configuration. +You can use Celery without any integration with Flask, but it's convenient to configure +it through Flask's config, and to let tasks access the Flask application. -This is all that is necessary to integrate Celery with Flask: +Celery uses similar ideas to Flask, with a ``Celery`` app object that has configuration +and registers tasks. While creating a Flask app, use the following code to create and +configure a Celery app as well. .. code-block:: python - from celery import Celery + from celery import Celery, Task - def make_celery(app): - celery = Celery(app.import_name) - celery.conf.update(app.config["CELERY_CONFIG"]) - - class ContextTask(celery.Task): - def __call__(self, *args, **kwargs): + def celery_init_app(app: Flask) -> Celery: + class FlaskTask(Task): + def __call__(self, *args: object, **kwargs: object) -> object: with app.app_context(): return self.run(*args, **kwargs) - celery.Task = ContextTask - return celery + celery_app = Celery(app.name, task_cls=FlaskTask) + celery_app.config_from_object(app.config["CELERY"]) + celery_app.set_default() + app.extensions["celery"] = celery_app + return celery_app -The function creates a new Celery object, configures it with the broker -from the application config, updates the rest of the Celery config from -the Flask config and then creates a subclass of the task that wraps the -task execution in an application context. +This creates and returns a ``Celery`` app object. Celery `configuration`_ is taken from +the ``CELERY`` key in the Flask configuration. The Celery app is set as the default, so +that it is seen during each request. The ``Task`` subclass automatically runs task +functions with a Flask app context active, so that services like your database +connections are available. -.. note:: - Celery 5.x deprecated uppercase configuration keys, and 6.x will - remove them. See their official `migration guide`_. +.. _configuration: https://celery.readthedocs.io/en/stable/userguide/configuration.html -.. _migration guide: https://docs.celeryproject.org/en/stable/userguide/configuration.html#conf-old-settings-map. +Here's a basic ``example.py`` that configures Celery to use Redis for communication. We +enable a result backend, but ignore results by default. This allows us to store results +only for tasks where we care about the result. -An example task ---------------- - -Let's write a task that adds two numbers together and returns the result. We -configure Celery's broker and backend to use Redis, create a ``celery`` -application using the factory from above, and then use it to define the task. :: +.. code-block:: python from flask import Flask - flask_app = Flask(__name__) - flask_app.config.update(CELERY_CONFIG={ - 'broker_url': 'redis://localhost:6379', - 'result_backend': 'redis://localhost:6379', - }) - celery = make_celery(flask_app) + app = Flask(__name__) + app.config.from_mapping( + CELERY=dict( + broker_url="redis://localhost", + result_backend="redis://localhost", + task_ignore_result=True, + ), + ) + celery_app = celery_init_app(app) - @celery.task() - def add_together(a, b): +Point the ``celery worker`` command at this and it will find the ``celery_app`` object. + +.. code-block:: text + + $ celery -A example worker --loglevel INFO + +You can also run the ``celery beat`` command to run tasks on a schedule. See Celery's +docs for more information about defining schedules. + +.. code-block:: text + + $ celery -A example beat --loglevel INFO + + +Application Factory +------------------- + +When using the Flask application factory pattern, call the ``celery_init_app`` function +inside the factory. It sets ``app.extensions["celery"]`` to the Celery app object, which +can be used to get the Celery app from the Flask app returned by the factory. + +.. code-block:: python + + def create_app() -> Flask: + app = Flask(__name__) + app.config.from_mapping( + CELERY=dict( + broker_url="redis://localhost", + result_backend="redis://localhost", + task_ignore_result=True, + ), + ) + app.config.from_prefixed_env() + celery_init_app(app) + return app + +To use ``celery`` commands, Celery needs an app object, but that's no longer directly +available. Create a ``make_celery.py`` file that calls the Flask app factory and gets +the Celery app from the returned Flask app. + +.. code-block:: python + + from example import create_app + + flask_app = create_app() + celery_app = flask_app.extensions["celery"] + +Point the ``celery`` command to this file. + +.. code-block:: text + + $ celery -A make_celery worker --loglevel INFO + $ celery -A make_celery beat --loglevel INFO + + +Defining Tasks +-------------- + +Using ``@celery_app.task`` to decorate task functions requires access to the +``celery_app`` object, which won't be available when using the factory pattern. It also +means that the decorated tasks are tied to the specific Flask and Celery app instances, +which could be an issue during testing if you change configuration for a test. + +Instead, use Celery's ``@shared_task`` decorator. This creates task objects that will +access whatever the "current app" is, which is a similar concept to Flask's blueprints +and app context. This is why we called ``celery_app.set_default()`` above. + +Here's an example task that adds two numbers together and returns the result. + +.. code-block:: python + + from celery import shared_task + + @shared_task(ignore_result=False) + def add_together(a: int, b: int) -> int: return a + b -This task can now be called in the background:: +Earlier, we configured Celery to ignore task results by default. Since we want to know +the return value of this task, we set ``ignore_result=False``. On the other hand, a task +that didn't need a result, such as sending an email, wouldn't set this. - result = add_together.delay(23, 42) - result.wait() # 65 -Run a worker ------------- +Calling Tasks +------------- -If you jumped in and already executed the above code you will be -disappointed to learn that ``.wait()`` will never actually return. -That's because you also need to run a Celery worker to receive and execute the -task. :: +The decorated function becomes a task object with methods to call it in the background. +The simplest way is to use the ``delay(*args, **kwargs)`` method. See Celery's docs for +more methods. - $ celery -A your_application.celery worker +A Celery worker must be running to run the task. Starting a worker is shown in the +previous sections. -The ``your_application`` string has to point to your application's package -or module that creates the ``celery`` object. +.. code-block:: python -Now that the worker is running, ``wait`` will return the result once the task -is finished. + from flask import request + + @app.post("/add") + def start_add() -> dict[str, object]: + a = request.form.get("a", type=int) + b = request.form.get("b", type=int) + result = add_together.delay(a, b) + return {"result_id": result.id} + +The route doesn't get the task's result immediately. That would defeat the purpose by +blocking the response. Instead, we return the running task's result id, which we can use +later to get the result. + + +Getting Results +--------------- + +To fetch the result of the task we started above, we'll add another route that takes the +result id we returned before. We return whether the task is finished (ready), whether it +finished successfully, and what the return value (or error) was if it is finished. + +.. code-block:: python + + from celery.result import AsyncResult + + @app.get("/result/") + def task_result(id: str) -> dict[str, object]: + result = AsyncResult(id) + return { + "ready": result.ready(), + "successful": result.successful(), + "value": result.result if result.ready() else None, + } + +Now you can start the task using the first route, then poll for the result using the +second route. This keeps the Flask request workers from being blocked waiting for tasks +to finish. + + +Passing Data to Tasks +--------------------- + +The "add" task above took two integers as arguments. To pass arguments to tasks, Celery +has to serialize them to a format that it can pass to other processes. Therefore, +passing complex objects is not recommended. For example, it would be impossible to pass +a SQLAlchemy model object, since that object is probably not serializable and is tied to +the session that queried it. + +Pass the minimal amount of data necessary to fetch or recreate any complex data within +the task. Consider a task that will run when the logged in user asks for an archive of +their data. The Flask request knows the logged in user, and has the user object queried +from the database. It got that by querying the database for a given id, so the task can +do the same thing. Pass the user's id rather than the user object. + +.. code-block:: python + + @shared_task + def generate_user_archive(user_id: str) -> None: + user = db.session.get(User, user_id) + ... + + generate_user_archive.delay(current_user.id) From 3f195248dcd59f8eb08e282b7980dc04b97d7391 Mon Sep 17 00:00:00 2001 From: David Lord Date: Thu, 9 Feb 2023 10:50:57 -0800 Subject: [PATCH 3/6] add celery example --- docs/patterns/celery.rst | 7 ++ examples/celery/README.md | 27 +++++ examples/celery/make_celery.py | 4 + examples/celery/pyproject.toml | 11 ++ examples/celery/requirements.txt | 56 +++++++++ examples/celery/src/task_app/__init__.py | 39 +++++++ examples/celery/src/task_app/tasks.py | 23 ++++ .../celery/src/task_app/templates/index.html | 108 ++++++++++++++++++ examples/celery/src/task_app/views.py | 38 ++++++ 9 files changed, 313 insertions(+) create mode 100644 examples/celery/README.md create mode 100644 examples/celery/make_celery.py create mode 100644 examples/celery/pyproject.toml create mode 100644 examples/celery/requirements.txt create mode 100644 examples/celery/src/task_app/__init__.py create mode 100644 examples/celery/src/task_app/tasks.py create mode 100644 examples/celery/src/task_app/templates/index.html create mode 100644 examples/celery/src/task_app/views.py diff --git a/docs/patterns/celery.rst b/docs/patterns/celery.rst index a236f6dd..2e9a43a7 100644 --- a/docs/patterns/celery.rst +++ b/docs/patterns/celery.rst @@ -14,6 +14,10 @@ Celery itself. .. _Celery: https://celery.readthedocs.io .. _First Steps with Celery: https://celery.readthedocs.io/en/latest/getting-started/first-steps-with-celery.html +The Flask repository contains `an example `_ +based on the information on this page, which also shows how to use JavaScript to submit +tasks and poll for progress and results. + Install ------- @@ -209,6 +213,9 @@ Now you can start the task using the first route, then poll for the result using second route. This keeps the Flask request workers from being blocked waiting for tasks to finish. +The Flask repository contains `an example `_ +using JavaScript to submit tasks and poll for progress and results. + Passing Data to Tasks --------------------- diff --git a/examples/celery/README.md b/examples/celery/README.md new file mode 100644 index 00000000..91782019 --- /dev/null +++ b/examples/celery/README.md @@ -0,0 +1,27 @@ +Background Tasks with Celery +============================ + +This example shows how to configure Celery with Flask, how to set up an API for +submitting tasks and polling results, and how to use that API with JavaScript. See +[Flask's documentation about Celery](https://flask.palletsprojects.com/patterns/celery/). + +From this directory, create a virtualenv and install the application into it. Then run a +Celery worker. + +```shell +$ python3 -m venv .venv +$ . ./.venv/bin/activate +$ pip install -r requirements.txt && pip install -e . +$ celery -A make_celery worker --loglevel INFO +``` + +In a separate terminal, activate the virtualenv and run the Flask development server. + +```shell +$ . ./.venv/bin/activate +$ flask -A task_app --debug run +``` + +Go to http://localhost:5000/ and use the forms to submit tasks. You can see the polling +requests in the browser dev tools and the Flask logs. You can see the tasks submitting +and completing in the Celery logs. diff --git a/examples/celery/make_celery.py b/examples/celery/make_celery.py new file mode 100644 index 00000000..f7d138e6 --- /dev/null +++ b/examples/celery/make_celery.py @@ -0,0 +1,4 @@ +from task_app import create_app + +flask_app = create_app() +celery_app = flask_app.extensions["celery"] diff --git a/examples/celery/pyproject.toml b/examples/celery/pyproject.toml new file mode 100644 index 00000000..88ba6b96 --- /dev/null +++ b/examples/celery/pyproject.toml @@ -0,0 +1,11 @@ +[project] +name = "flask-example-celery" +version = "1.0.0" +description = "Example Flask application with Celery background tasks." +readme = "README.md" +requires-python = ">=3.7" +dependencies = ["flask>=2.2.2", "celery[redis]>=5.2.7"] + +[build-system] +requires = ["setuptools"] +build-backend = "setuptools.build_meta" diff --git a/examples/celery/requirements.txt b/examples/celery/requirements.txt new file mode 100644 index 00000000..b2834013 --- /dev/null +++ b/examples/celery/requirements.txt @@ -0,0 +1,56 @@ +# +# This file is autogenerated by pip-compile with Python 3.10 +# by the following command: +# +# pip-compile pyproject.toml +# +amqp==5.1.1 + # via kombu +async-timeout==4.0.2 + # via redis +billiard==3.6.4.0 + # via celery +celery[redis]==5.2.7 + # via flask-example-celery (pyproject.toml) +click==8.1.3 + # via + # celery + # click-didyoumean + # click-plugins + # click-repl + # flask +click-didyoumean==0.3.0 + # via celery +click-plugins==1.1.1 + # via celery +click-repl==0.2.0 + # via celery +flask==2.2.2 + # via flask-example-celery (pyproject.toml) +itsdangerous==2.1.2 + # via flask +jinja2==3.1.2 + # via flask +kombu==5.2.4 + # via celery +markupsafe==2.1.2 + # via + # jinja2 + # werkzeug +prompt-toolkit==3.0.36 + # via click-repl +pytz==2022.7.1 + # via celery +redis==4.5.1 + # via celery +six==1.16.0 + # via click-repl +vine==5.0.0 + # via + # amqp + # celery + # kombu +wcwidth==0.2.6 + # via prompt-toolkit +werkzeug==2.2.2 + # via flask diff --git a/examples/celery/src/task_app/__init__.py b/examples/celery/src/task_app/__init__.py new file mode 100644 index 00000000..dafff8aa --- /dev/null +++ b/examples/celery/src/task_app/__init__.py @@ -0,0 +1,39 @@ +from celery import Celery +from celery import Task +from flask import Flask +from flask import render_template + + +def create_app() -> Flask: + app = Flask(__name__) + app.config.from_mapping( + CELERY=dict( + broker_url="redis://localhost", + result_backend="redis://localhost", + task_ignore_result=True, + ), + ) + app.config.from_prefixed_env() + celery_init_app(app) + + @app.route("/") + def index() -> str: + return render_template("index.html") + + from . import views + + app.register_blueprint(views.bp) + return app + + +def celery_init_app(app: Flask) -> Celery: + class FlaskTask(Task): + def __call__(self, *args: object, **kwargs: object) -> object: + with app.app_context(): + return self.run(*args, **kwargs) + + celery_app = Celery(app.name, task_cls=FlaskTask) + celery_app.config_from_object(app.config["CELERY"]) + celery_app.set_default() + app.extensions["celery"] = celery_app + return celery_app diff --git a/examples/celery/src/task_app/tasks.py b/examples/celery/src/task_app/tasks.py new file mode 100644 index 00000000..b6b3595d --- /dev/null +++ b/examples/celery/src/task_app/tasks.py @@ -0,0 +1,23 @@ +import time + +from celery import shared_task +from celery import Task + + +@shared_task(ignore_result=False) +def add(a: int, b: int) -> int: + return a + b + + +@shared_task() +def block() -> None: + time.sleep(5) + + +@shared_task(bind=True, ignore_result=False) +def process(self: Task, total: int) -> object: + for i in range(total): + self.update_state(state="PROGRESS", meta={"current": i + 1, "total": total}) + time.sleep(1) + + return {"current": total, "total": total} diff --git a/examples/celery/src/task_app/templates/index.html b/examples/celery/src/task_app/templates/index.html new file mode 100644 index 00000000..4e1145cb --- /dev/null +++ b/examples/celery/src/task_app/templates/index.html @@ -0,0 +1,108 @@ + + + + + Celery Example + + +

Celery Example

+Execute background tasks with Celery. Submits tasks and shows results using JavaScript. + +
+

Add

+

Start a task to add two numbers, then poll for the result. +

+
+
+ +
+

Result:

+ +
+

Block

+

Start a task that takes 5 seconds. However, the response will return immediately. +

+ +
+

+ +
+

Process

+

Start a task that counts, waiting one second each time, showing progress. +

+
+ +
+

+ + + + diff --git a/examples/celery/src/task_app/views.py b/examples/celery/src/task_app/views.py new file mode 100644 index 00000000..99cf92dc --- /dev/null +++ b/examples/celery/src/task_app/views.py @@ -0,0 +1,38 @@ +from celery.result import AsyncResult +from flask import Blueprint +from flask import request + +from . import tasks + +bp = Blueprint("tasks", __name__, url_prefix="/tasks") + + +@bp.get("/result/") +def result(id: str) -> dict[str, object]: + result = AsyncResult(id) + ready = result.ready() + return { + "ready": ready, + "successful": result.successful() if ready else None, + "value": result.get() if ready else result.result, + } + + +@bp.post("/add") +def add() -> dict[str, object]: + a = request.form.get("a", type=int) + b = request.form.get("b", type=int) + result = tasks.add.delay(a, b) + return {"result_id": result.id} + + +@bp.post("/block") +def block() -> dict[str, object]: + result = tasks.block.delay() + return {"result_id": result.id} + + +@bp.post("/process") +def process() -> dict[str, object]: + result = tasks.process.delay(total=request.form.get("total", type=int)) + return {"result_id": result.id} From ab93222bd6d4ea26e3aa832a0409489530f3f5e0 Mon Sep 17 00:00:00 2001 From: David Lord Date: Fri, 10 Feb 2023 09:58:48 -0800 Subject: [PATCH 4/6] point to app-scoped blueprint methods --- src/flask/blueprints.py | 70 ++++++++++++++++++++++------------------- src/flask/scaffold.py | 40 ++++++++++++++++++++++- 2 files changed, 77 insertions(+), 33 deletions(-) diff --git a/src/flask/blueprints.py b/src/flask/blueprints.py index f6d62ba8..eb664235 100644 --- a/src/flask/blueprints.py +++ b/src/flask/blueprints.py @@ -478,8 +478,11 @@ class Blueprint(Scaffold): provide_automatic_options: t.Optional[bool] = None, **options: t.Any, ) -> None: - """Like :meth:`Flask.add_url_rule` but for a blueprint. The endpoint for - the :func:`url_for` function is prefixed with the name of the blueprint. + """Register a URL rule with the blueprint. See :meth:`.Flask.add_url_rule` for + full documentation. + + The URL rule is prefixed with the blueprint's URL prefix. The endpoint name, + used with :func:`url_for`, is prefixed with the blueprint's name. """ if endpoint and "." in endpoint: raise ValueError("'endpoint' may not contain a dot '.' character.") @@ -501,8 +504,8 @@ class Blueprint(Scaffold): def app_template_filter( self, name: t.Optional[str] = None ) -> t.Callable[[T_template_filter], T_template_filter]: - """Register a custom template filter, available application wide. Like - :meth:`Flask.template_filter` but for a blueprint. + """Register a template filter, available in any template rendered by the + application. Equivalent to :meth:`.Flask.template_filter`. :param name: the optional name of the filter, otherwise the function name will be used. @@ -518,9 +521,9 @@ class Blueprint(Scaffold): def add_app_template_filter( self, f: ft.TemplateFilterCallable, name: t.Optional[str] = None ) -> None: - """Register a custom template filter, available application wide. Like - :meth:`Flask.add_template_filter` but for a blueprint. Works exactly - like the :meth:`app_template_filter` decorator. + """Register a template filter, available in any template rendered by the + application. Works like the :meth:`app_template_filter` decorator. Equivalent to + :meth:`.Flask.add_template_filter`. :param name: the optional name of the filter, otherwise the function name will be used. @@ -535,8 +538,8 @@ class Blueprint(Scaffold): def app_template_test( self, name: t.Optional[str] = None ) -> t.Callable[[T_template_test], T_template_test]: - """Register a custom template test, available application wide. Like - :meth:`Flask.template_test` but for a blueprint. + """Register a template test, available in any template rendered by the + application. Equivalent to :meth:`.Flask.template_test`. .. versionadded:: 0.10 @@ -554,9 +557,9 @@ class Blueprint(Scaffold): def add_app_template_test( self, f: ft.TemplateTestCallable, name: t.Optional[str] = None ) -> None: - """Register a custom template test, available application wide. Like - :meth:`Flask.add_template_test` but for a blueprint. Works exactly - like the :meth:`app_template_test` decorator. + """Register a template test, available in any template rendered by the + application. Works like the :meth:`app_template_test` decorator. Equivalent to + :meth:`.Flask.add_template_test`. .. versionadded:: 0.10 @@ -573,8 +576,8 @@ class Blueprint(Scaffold): def app_template_global( self, name: t.Optional[str] = None ) -> t.Callable[[T_template_global], T_template_global]: - """Register a custom template global, available application wide. Like - :meth:`Flask.template_global` but for a blueprint. + """Register a template global, available in any template rendered by the + application. Equivalent to :meth:`.Flask.template_global`. .. versionadded:: 0.10 @@ -592,9 +595,9 @@ class Blueprint(Scaffold): def add_app_template_global( self, f: ft.TemplateGlobalCallable, name: t.Optional[str] = None ) -> None: - """Register a custom template global, available application wide. Like - :meth:`Flask.add_template_global` but for a blueprint. Works exactly - like the :meth:`app_template_global` decorator. + """Register a template global, available in any template rendered by the + application. Works like the :meth:`app_template_global` decorator. Equivalent to + :meth:`.Flask.add_template_global`. .. versionadded:: 0.10 @@ -609,8 +612,8 @@ class Blueprint(Scaffold): @setupmethod def before_app_request(self, f: T_before_request) -> T_before_request: - """Like :meth:`Flask.before_request`. Such a function is executed - before each request, even if outside of a blueprint. + """Like :meth:`before_request`, but before every request, not only those handled + by the blueprint. Equivalent to :meth:`.Flask.before_request`. """ self.record_once( lambda s: s.app.before_request_funcs.setdefault(None, []).append(f) @@ -621,8 +624,8 @@ class Blueprint(Scaffold): def before_app_first_request( self, f: T_before_first_request ) -> T_before_first_request: - """Like :meth:`Flask.before_first_request`. Such a function is - executed before the first request to the application. + """Register a function to run before the first request to the application is + handled by the worker. Equivalent to :meth:`.Flask.before_first_request`. .. deprecated:: 2.2 Will be removed in Flask 2.3. Run setup code when creating @@ -642,8 +645,8 @@ class Blueprint(Scaffold): @setupmethod def after_app_request(self, f: T_after_request) -> T_after_request: - """Like :meth:`Flask.after_request` but for a blueprint. Such a function - is executed after each request, even if outside of the blueprint. + """Like :meth:`after_request`, but after every request, not only those handled + by the blueprint. Equivalent to :meth:`.Flask.after_request`. """ self.record_once( lambda s: s.app.after_request_funcs.setdefault(None, []).append(f) @@ -652,9 +655,8 @@ class Blueprint(Scaffold): @setupmethod def teardown_app_request(self, f: T_teardown) -> T_teardown: - """Like :meth:`Flask.teardown_request` but for a blueprint. Such a - function is executed when tearing down each request, even if outside of - the blueprint. + """Like :meth:`teardown_request`, but after every request, not only those + handled by the blueprint. Equivalent to :meth:`.Flask.teardown_request`. """ self.record_once( lambda s: s.app.teardown_request_funcs.setdefault(None, []).append(f) @@ -665,8 +667,8 @@ class Blueprint(Scaffold): def app_context_processor( self, f: T_template_context_processor ) -> T_template_context_processor: - """Like :meth:`Flask.context_processor` but for a blueprint. Such a - function is executed each request, even if outside of the blueprint. + """Like :meth:`context_processor`, but for templates rendered by every view, not + only by the blueprint. Equivalent to :meth:`.Flask.context_processor`. """ self.record_once( lambda s: s.app.template_context_processors.setdefault(None, []).append(f) @@ -677,8 +679,8 @@ class Blueprint(Scaffold): def app_errorhandler( self, code: t.Union[t.Type[Exception], int] ) -> t.Callable[[T_error_handler], T_error_handler]: - """Like :meth:`Flask.errorhandler` but for a blueprint. This - handler is used for all requests, even if outside of the blueprint. + """Like :meth:`errorhandler`, but for every request, not only those handled by + the blueprint. Equivalent to :meth:`.Flask.errorhandler`. """ def decorator(f: T_error_handler) -> T_error_handler: @@ -691,7 +693,9 @@ class Blueprint(Scaffold): def app_url_value_preprocessor( self, f: T_url_value_preprocessor ) -> T_url_value_preprocessor: - """Same as :meth:`url_value_preprocessor` but application wide.""" + """Like :meth:`url_value_preprocessor`, but for every request, not only those + handled by the blueprint. Equivalent to :meth:`.Flask.url_value_preprocessor`. + """ self.record_once( lambda s: s.app.url_value_preprocessors.setdefault(None, []).append(f) ) @@ -699,7 +703,9 @@ class Blueprint(Scaffold): @setupmethod def app_url_defaults(self, f: T_url_defaults) -> T_url_defaults: - """Same as :meth:`url_defaults` but application wide.""" + """Like :meth:`url_defaults`, but for every request, not only those handled by + the blueprint. Equivalent to :meth:`.Flask.url_defaults`. + """ self.record_once( lambda s: s.app.url_default_functions.setdefault(None, []).append(f) ) diff --git a/src/flask/scaffold.py b/src/flask/scaffold.py index ebfc741f..7277b33a 100644 --- a/src/flask/scaffold.py +++ b/src/flask/scaffold.py @@ -561,6 +561,11 @@ class Scaffold: a non-``None`` value, the value is handled as if it was the return value from the view, and further request handling is stopped. + + This is available on both app and blueprint objects. When used on an app, this + executes before every request. When used on a blueprint, this executes before + every request that the blueprint handles. To register with a blueprint and + execute before every request, use :meth:`.Blueprint.before_app_request`. """ self.before_request_funcs.setdefault(None, []).append(f) return f @@ -577,6 +582,11 @@ class Scaffold: ``after_request`` functions will not be called. Therefore, this should not be used for actions that must execute, such as to close resources. Use :meth:`teardown_request` for that. + + This is available on both app and blueprint objects. When used on an app, this + executes after every request. When used on a blueprint, this executes after + every request that the blueprint handles. To register with a blueprint and + execute after every request, use :meth:`.Blueprint.after_app_request`. """ self.after_request_funcs.setdefault(None, []).append(f) return f @@ -606,6 +616,11 @@ class Scaffold: ``try``/``except`` block and log any errors. The return values of teardown functions are ignored. + + This is available on both app and blueprint objects. When used on an app, this + executes after every request. When used on a blueprint, this executes after + every request that the blueprint handles. To register with a blueprint and + execute after every request, use :meth:`.Blueprint.teardown_app_request`. """ self.teardown_request_funcs.setdefault(None, []).append(f) return f @@ -615,7 +630,15 @@ class Scaffold: self, f: T_template_context_processor, ) -> T_template_context_processor: - """Registers a template context processor function.""" + """Registers a template context processor function. These functions run before + rendering a template. The keys of the returned dict are added as variables + available in the template. + + This is available on both app and blueprint objects. When used on an app, this + is called for every rendered template. When used on a blueprint, this is called + for templates rendered from the blueprint's views. To register with a blueprint + and affect every template, use :meth:`.Blueprint.app_context_processor`. + """ self.template_context_processors[None].append(f) return f @@ -635,6 +658,11 @@ class Scaffold: The function is passed the endpoint name and values dict. The return value is ignored. + + This is available on both app and blueprint objects. When used on an app, this + is called for every request. When used on a blueprint, this is called for + requests that the blueprint handles. To register with a blueprint and affect + every request, use :meth:`.Blueprint.app_url_value_preprocessor`. """ self.url_value_preprocessors[None].append(f) return f @@ -644,6 +672,11 @@ class Scaffold: """Callback function for URL defaults for all view functions of the application. It's called with the endpoint and values and should update the values passed in place. + + This is available on both app and blueprint objects. When used on an app, this + is called for every request. When used on a blueprint, this is called for + requests that the blueprint handles. To register with a blueprint and affect + every request, use :meth:`.Blueprint.app_url_defaults`. """ self.url_default_functions[None].append(f) return f @@ -667,6 +700,11 @@ class Scaffold: def special_exception_handler(error): return 'Database connection failed', 500 + This is available on both app and blueprint objects. When used on an app, this + can handle errors from every request. When used on a blueprint, this can handle + errors from requests that the blueprint handles. To register with a blueprint + and affect every request, use :meth:`.Blueprint.app_errorhandler`. + .. versionadded:: 0.7 Use :meth:`register_error_handler` instead of modifying :attr:`error_handler_spec` directly, for application wide error From ba2b3094d1a8177059ea68853a48fcd5e90920fe Mon Sep 17 00:00:00 2001 From: David Lord Date: Fri, 10 Feb 2023 10:50:48 -0800 Subject: [PATCH 5/6] fix test client arg for query string example --- docs/reqcontext.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/reqcontext.rst b/docs/reqcontext.rst index 2b109770..70ea13e3 100644 --- a/docs/reqcontext.rst +++ b/docs/reqcontext.rst @@ -69,11 +69,12 @@ everything that runs in the block will have access to :data:`request`, populated with your test data. :: def generate_report(year): - format = request.args.get('format') + format = request.args.get("format") ... with app.test_request_context( - '/make_report/2017', data={'format': 'short'}): + "/make_report/2017", query_string={"format": "short"} + ): generate_report() If you see that error somewhere else in your code not related to From a6cd8f212e762b8f70e00f3341b27799c0fb657a Mon Sep 17 00:00:00 2001 From: David Lord Date: Fri, 10 Feb 2023 14:48:02 -0800 Subject: [PATCH 6/6] document the lifecycle of a flask application and request --- docs/index.rst | 1 + docs/lifecycle.rst | 168 ++++++++++++++++++++++++++++++++++++++++++++ docs/reqcontext.rst | 5 +- 3 files changed, 172 insertions(+), 2 deletions(-) create mode 100644 docs/lifecycle.rst diff --git a/docs/index.rst b/docs/index.rst index 983f612f..747749d0 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -48,6 +48,7 @@ community-maintained extensions to add even more functionality. config signals views + lifecycle appcontext reqcontext blueprints diff --git a/docs/lifecycle.rst b/docs/lifecycle.rst new file mode 100644 index 00000000..2344d98a --- /dev/null +++ b/docs/lifecycle.rst @@ -0,0 +1,168 @@ +Application Structure and Lifecycle +=================================== + +Flask makes it pretty easy to write a web application. But there are quite a few +different parts to an application and to each request it handles. Knowing what happens +during application setup, serving, and handling requests will help you know what's +possible in Flask and how to structure your application. + + +Application Setup +----------------- + +The first step in creating a Flask application is creating the application object. Each +Flask application is an instance of the :class:`.Flask` class, which collects all +configuration, extensions, and views. + +.. code-block:: python + + from flask import Flask + + app = Flask(__name__) + app.config.from_mapping( + SECRET_KEY="dev", + ) + app.config.from_prefixed_env() + + @app.route("/") + def index(): + return "Hello, World!" + +This is known as the "application setup phase", it's the code you write that's outside +any view functions or other handlers. It can be split up between different modules and +sub-packages, but all code that you want to be part of your application must be imported +in order for it to be registered. + +All application setup must be completed before you start serving your application and +handling requests. This is because WSGI servers divide work between multiple workers, or +can be distributed across multiple machines. If the configuration changed in one worker, +there's no way for Flask to ensure consistency between other workers. + +Flask tries to help developers catch some of these setup ordering issues by showing an +error if setup-related methods are called after requests are handled. In that case +you'll see this error: + + The setup method 'route' can no longer be called on the application. It has already + handled its first request, any changes will not be applied consistently. + Make sure all imports, decorators, functions, etc. needed to set up the application + are done before running it. + +However, it is not possible for Flask to detect all cases of out-of-order setup. In +general, don't do anything to modify the ``Flask`` app object and ``Blueprint`` objects +from within view functions that run during requests. This includes: + +- Adding routes, view functions, and other request handlers with ``@app.route``, + ``@app.errorhandler``, ``@app.before_request``, etc. +- Registering blueprints. +- Loading configuration with ``app.config``. +- Setting up the Jinja template environment with ``app.jinja_env``. +- Setting a session interface, instead of the default itsdangerous cookie. +- Setting a JSON provider with ``app.json``, instead of the default provider. +- Creating and initializing Flask extensions. + + +Serving the Application +----------------------- + +Flask is a WSGI application framework. The other half of WSGI is the WSGI server. During +development, Flask, through Werkzeug, provides a development WSGI server with the +``flask run`` CLI command. When you are done with development, use a production server +to serve your application, see :doc:`deploying/index`. + +Regardless of what server you're using, it will follow the :pep:`3333` WSGI spec. The +WSGI server will be told how to access your Flask application object, which is the WSGI +application. Then it will start listening for HTTP requests, translate the request data +into a WSGI environ, and call the WSGI application with that data. The WSGI application +will return data that is translated into an HTTP response. + +#. Browser or other client makes HTTP request. +#. WSGI server receives request. +#. WSGI server converts HTTP data to WSGI ``environ`` dict. +#. WSGI server calls WSGI application with the ``environ``. +#. Flask, the WSGI application, does all its internal processing to route the request + to a view function, handle errors, etc. +#. Flask translates View function return into WSGI response data, passes it to WSGI + server. +#. WSGI server creates and send an HTTP response. +#. Client receives the HTTP response. + + +Middleware +~~~~~~~~~~ + +The WSGI application above is a callable that behaves in a certain way. Middleware +is a WSGI application that wraps another WSGI application. It's a similar concept to +Python decorators. The outermost middleware will be called by the server. It can modify +the data passed to it, then call the WSGI application (or further middleware) that it +wraps, and so on. And it can take the return value of that call and modify it further. + +From the WSGI server's perspective, there is one WSGI application, the one it calls +directly. Typically, Flask is the "real" application at the end of the chain of +middleware. But even Flask can call further WSGI applications, although that's an +advanced, uncommon use case. + +A common middleware you'll see used with Flask is Werkzeug's +:class:`~werkzeug.middleware.proxy_fix.ProxyFix`, which modifies the request to look +like it came directly from a client even if it passed through HTTP proxies on the way. +There are other middleware that can handle serving static files, authentication, etc. + + +How a Request is Handled +------------------------ + +For us, the interesting part of the steps above is when Flask gets called by the WSGI +server (or middleware). At that point, it will do quite a lot to handle the request and +generate the response. At the most basic, it will match the URL to a view function, call +the view function, and pass the return value back to the server. But there are many more +parts that you can use to customize its behavior. + +#. WSGI server calls the Flask object, which calls :meth:`.Flask.wsgi_app`. +#. A :class:`.RequestContext` object is created. This converts the WSGI ``environ`` + dict into a :class:`.Request` object. It also creates an :class:`AppContext` object. +#. The :doc:`app context ` is pushed, which makes :data:`.current_app` and + :data:`.g` available. +#. The :data:`.appcontext_pushed` signal is sent. +#. The :doc:`request context ` is pushed, which makes :attr:`.request` and + :class:`.session` available. +#. The session is opened, loading any existing session data using the app's + :attr:`~.Flask.session_interface`, an instance of :class:`.SessionInterface`. +#. The URL is matched against the URL rules registered with the :meth:`~.Flask.route` + decorator during application setup. If there is no match, the error - usually a 404, + 405, or redirect - is stored to be handled later. +#. The :data:`.request_started` signal is sent. +#. Any :meth:`~.Flask.url_value_preprocessor` decorated functions are called. +#. Any :meth:`~.Flask.before_request` decorated functions are called. If any of + these function returns a value it is treated as the response immediately. +#. If the URL didn't match a route a few steps ago, that error is raised now. +#. The :meth:`~.Flask.route` decorated view function associated with the matched URL + is called and returns a value to be used as the response. +#. If any step so far raised an exception, and there is an :meth:`~.Flask.errorhandler` + decorated function that matches the exception class or HTTP error code, it is + called to handle the error and return a response. +#. Whatever returned a response value - a before request function, the view, or an + error handler, that value is converted to a :class:`.Response` object. +#. Any :func:`~.after_this_request` decorated functions are called, then cleared. +#. Any :meth:`~.Flask.after_request` decorated functions are called, which can modify + the response object. +#. The session is saved, persisting any modified session data using the app's + :attr:`~.Flask.session_interface`. +#. The :data:`.request_finished` signal is sent. +#. If any step so far raised an exception, and it was not handled by an error handler + function, it is handled now. HTTP exceptions are treated as responses with their + corresponding status code, other exceptions are converted to a generic 500 response. + The :data:`.got_request_exception` signal is sent. +#. The response object's status, headers, and body are returned to the WSGI server. +#. Any :meth:`~.Flask.teardown_request` decorated functions are called. +#. The :data:`.request_tearing_down` signal is sent. +#. The request context is popped, :attr:`.request` and :class:`.session` are no longer + available. +#. Any :meth:`~.Flask.teardown_appcontext` decorated functions are called. +#. The :data:`.appcontext_tearing_down` signal is sent. +#. The app context is popped, :data:`.current_app` and :data:`.g` are no longer + available. +#. The :data:`.appcontext_popped` signal is sent. + +There are even more decorators and customization points than this, but that aren't part +of every request lifecycle. They're more specific to certain things you might use during +a request, such as templates, building URLs, or handling JSON data. See the rest of this +documentation, as well as the :doc:`api` to explore further. diff --git a/docs/reqcontext.rst b/docs/reqcontext.rst index 2b109770..70ea13e3 100644 --- a/docs/reqcontext.rst +++ b/docs/reqcontext.rst @@ -69,11 +69,12 @@ everything that runs in the block will have access to :data:`request`, populated with your test data. :: def generate_report(year): - format = request.args.get('format') + format = request.args.get("format") ... with app.test_request_context( - '/make_report/2017', data={'format': 'short'}): + "/make_report/2017", query_string={"format": "short"} + ): generate_report() If you see that error somewhere else in your code not related to