[pre-commit.ci lite] apply automatic fixes
This commit is contained in:
pre-commit-ci-lite[bot]
GitHub
parent
b3ae3117f9
commit
3d83d8138c
102 changed files with 26790 additions and 26749 deletions
|
|
@ -5,7 +5,7 @@
|
|||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Background Tasks with Celery — Flask Documentation (3.2.x)</title>
|
||||
<title>Uploading Files — Flask Documentation (3.2.x)</title>
|
||||
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=6625fa76" />
|
||||
<link rel="stylesheet" type="text/css" href="../_static/flask.css?v=b87c8d14" />
|
||||
<script src="../_static/documentation_options.js?v=56528222"></script>
|
||||
|
|
@ -15,8 +15,8 @@
|
|||
<link rel="icon" href="../_static/shortcut-icon.png"/>
|
||||
<link rel="index" title="Index" href="../genindex.html" />
|
||||
<link rel="search" title="Search" href="../search.html" />
|
||||
<link rel="next" title="Subclassing Flask" href="subclassing.html" />
|
||||
<link rel="prev" title="Request Content Checksums" href="requestchecksum.html" />
|
||||
<link rel="next" title="Caching" href="caching.html" />
|
||||
<link rel="prev" title="SQLAlchemy in Flask" href="sqlalchemy.html" />
|
||||
</head><body>
|
||||
<div class="related" role="navigation" aria-label="Related">
|
||||
<h3>Navigation</h3>
|
||||
|
|
@ -28,214 +28,189 @@
|
|||
<a href="../py-modindex.html" title="Python Module Index"
|
||||
>modules</a> |</li>
|
||||
<li class="right" >
|
||||
<a href="subclassing.html" title="Subclassing Flask"
|
||||
<a href="caching.html" title="Caching"
|
||||
accesskey="N">next</a> |</li>
|
||||
<li class="right" >
|
||||
<a href="requestchecksum.html" title="Request Content Checksums"
|
||||
<a href="sqlalchemy.html" title="SQLAlchemy in Flask"
|
||||
accesskey="P">previous</a> |</li>
|
||||
<li class="nav-item nav-item-0"><a href="../index.html">Flask Documentation (3.2.x)</a> »</li>
|
||||
<li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Patterns for Flask</a> »</li>
|
||||
<li class="nav-item nav-item-this"><a href="">Background Tasks with Celery</a></li>
|
||||
<li class="nav-item nav-item-this"><a href="">Uploading Files</a></li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="document">
|
||||
<div class="documentwrapper">
|
||||
<div class="bodywrapper">
|
||||
<div class="body" role="main">
|
||||
|
||||
<section id="background-tasks-with-celery">
|
||||
<h1>Background Tasks with Celery<a class="headerlink" href="#background-tasks-with-celery" title="Link to this heading">¶</a></h1>
|
||||
<p>If your application has a long running task, such as processing some uploaded data or
|
||||
sending email, you don’t want to wait for it to finish during a request. Instead, use a
|
||||
task queue to send the necessary data to another process that will run the task in the
|
||||
background while the request returns immediately.</p>
|
||||
<p><a class="reference external" href="https://celery.readthedocs.io">Celery</a> is a powerful task queue that can be used for simple background tasks as well
|
||||
as complex multi-stage programs and schedules. This guide will show you how to configure
|
||||
Celery using Flask. Read Celery’s <a class="reference external" href="https://celery.readthedocs.io/en/latest/getting-started/first-steps-with-celery.html">First Steps with Celery</a> guide to learn how to use
|
||||
Celery itself.</p>
|
||||
<p>The Flask repository contains <a class="reference external" href="https://github.com/pallets/flask/tree/main/examples/celery">an example</a>
|
||||
based on the information on this page, which also shows how to use JavaScript to submit
|
||||
tasks and poll for progress and results.</p>
|
||||
<section id="install">
|
||||
<h2>Install<a class="headerlink" href="#install" title="Link to this heading">¶</a></h2>
|
||||
<p>Install Celery from PyPI, for example using pip:</p>
|
||||
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>$ pip install celery
|
||||
</pre></div>
|
||||
</div>
|
||||
</section>
|
||||
<section id="integrate-celery-with-flask">
|
||||
<h2>Integrate Celery with Flask<a class="headerlink" href="#integrate-celery-with-flask" title="Link to this heading">¶</a></h2>
|
||||
<p>You can use Celery without any integration with Flask, but it’s convenient to configure
|
||||
it through Flask’s config, and to let tasks access the Flask application.</p>
|
||||
<p>Celery uses similar ideas to Flask, with a <code class="docutils literal notranslate"><span class="pre">Celery</span></code> app object that has configuration
|
||||
and registers tasks. While creating a Flask app, use the following code to create and
|
||||
configure a Celery app as well.</p>
|
||||
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">celery</span><span class="w"> </span><span class="kn">import</span> <span class="n">Celery</span><span class="p">,</span> <span class="n">Task</span>
|
||||
|
||||
<span class="k">def</span><span class="w"> </span><span class="nf">celery_init_app</span><span class="p">(</span><span class="n">app</span><span class="p">:</span> <span class="n">Flask</span><span class="p">)</span> <span class="o">-></span> <span class="n">Celery</span><span class="p">:</span>
|
||||
<span class="k">class</span><span class="w"> </span><span class="nc">FlaskTask</span><span class="p">(</span><span class="n">Task</span><span class="p">):</span>
|
||||
<span class="k">def</span><span class="w"> </span><span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="nb">object</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">:</span> <span class="nb">object</span><span class="p">)</span> <span class="o">-></span> <span class="nb">object</span><span class="p">:</span>
|
||||
<span class="k">with</span> <span class="n">app</span><span class="o">.</span><span class="n">app_context</span><span class="p">():</span>
|
||||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
|
||||
<section id="uploading-files">
|
||||
<h1>Uploading Files<a class="headerlink" href="#uploading-files" title="Link to this heading">¶</a></h1>
|
||||
<p>Ah yes, the good old problem of file uploads. The basic idea of file
|
||||
uploads is actually quite simple. It basically works like this:</p>
|
||||
<ol class="arabic simple">
|
||||
<li><p>A <code class="docutils literal notranslate"><span class="pre"><form></span></code> tag is marked with <code class="docutils literal notranslate"><span class="pre">enctype=multipart/form-data</span></code>
|
||||
and an <code class="docutils literal notranslate"><span class="pre"><input</span> <span class="pre">type=file></span></code> is placed in that form.</p></li>
|
||||
<li><p>The application accesses the file from the <code class="xref py py-attr docutils literal notranslate"><span class="pre">files</span></code>
|
||||
dictionary on the request object.</p></li>
|
||||
<li><p>use the <a class="reference external" href="https://werkzeug.palletsprojects.com/en/stable/datastructures/#werkzeug.datastructures.FileStorage.save" title="(in Werkzeug v3.1.x)"><code class="xref py py-meth docutils literal notranslate"><span class="pre">save()</span></code></a> method of the file to save
|
||||
the file permanently somewhere on the filesystem.</p></li>
|
||||
</ol>
|
||||
<section id="a-gentle-introduction">
|
||||
<h2>A Gentle Introduction<a class="headerlink" href="#a-gentle-introduction" title="Link to this heading">¶</a></h2>
|
||||
<p>Let’s start with a very basic application that uploads a file to a
|
||||
specific upload folder and displays a file to the user. Let’s look at the
|
||||
bootstrapping code for our application:</p>
|
||||
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">os</span>
|
||||
<span class="kn">from</span><span class="w"> </span><span class="nn">flask</span><span class="w"> </span><span class="kn">import</span> <span class="n">Flask</span><span class="p">,</span> <span class="n">flash</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">redirect</span><span class="p">,</span> <span class="n">url_for</span>
|
||||
<span class="kn">from</span><span class="w"> </span><span class="nn">werkzeug.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">secure_filename</span>
|
||||
|
||||
<span class="n">celery_app</span> <span class="o">=</span> <span class="n">Celery</span><span class="p">(</span><span class="n">app</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">task_cls</span><span class="o">=</span><span class="n">FlaskTask</span><span class="p">)</span>
|
||||
<span class="n">celery_app</span><span class="o">.</span><span class="n">config_from_object</span><span class="p">(</span><span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="p">[</span><span class="s2">"CELERY"</span><span class="p">])</span>
|
||||
<span class="n">celery_app</span><span class="o">.</span><span class="n">set_default</span><span class="p">()</span>
|
||||
<span class="n">app</span><span class="o">.</span><span class="n">extensions</span><span class="p">[</span><span class="s2">"celery"</span><span class="p">]</span> <span class="o">=</span> <span class="n">celery_app</span>
|
||||
<span class="k">return</span> <span class="n">celery_app</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>This creates and returns a <code class="docutils literal notranslate"><span class="pre">Celery</span></code> app object. Celery <a class="reference external" href="https://celery.readthedocs.io/en/stable/userguide/configuration.html">configuration</a> is taken from
|
||||
the <code class="docutils literal notranslate"><span class="pre">CELERY</span></code> key in the Flask configuration. The Celery app is set as the default, so
|
||||
that it is seen during each request. The <code class="docutils literal notranslate"><span class="pre">Task</span></code> subclass automatically runs task
|
||||
functions with a Flask app context active, so that services like your database
|
||||
connections are available.</p>
|
||||
<p>Here’s a basic <code class="docutils literal notranslate"><span class="pre">example.py</span></code> that configures Celery to use Redis for communication. We
|
||||
enable a result backend, but ignore results by default. This allows us to store results
|
||||
only for tasks where we care about the result.</p>
|
||||
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">flask</span><span class="w"> </span><span class="kn">import</span> <span class="n">Flask</span>
|
||||
<span class="n">UPLOAD_FOLDER</span> <span class="o">=</span> <span class="s1">'/path/to/the/uploads'</span>
|
||||
<span class="n">ALLOWED_EXTENSIONS</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'txt'</span><span class="p">,</span> <span class="s1">'pdf'</span><span class="p">,</span> <span class="s1">'png'</span><span class="p">,</span> <span class="s1">'jpg'</span><span class="p">,</span> <span class="s1">'jpeg'</span><span class="p">,</span> <span class="s1">'gif'</span><span class="p">}</span>
|
||||
|
||||
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
|
||||
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_mapping</span><span class="p">(</span>
|
||||
<span class="n">CELERY</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span>
|
||||
<span class="n">broker_url</span><span class="o">=</span><span class="s2">"redis://localhost"</span><span class="p">,</span>
|
||||
<span class="n">result_backend</span><span class="o">=</span><span class="s2">"redis://localhost"</span><span class="p">,</span>
|
||||
<span class="n">task_ignore_result</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
|
||||
<span class="p">),</span>
|
||||
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="p">[</span><span class="s1">'UPLOAD_FOLDER'</span><span class="p">]</span> <span class="o">=</span> <span class="n">UPLOAD_FOLDER</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>So first we need a couple of imports. Most should be straightforward, the
|
||||
<code class="xref py py-func docutils literal notranslate"><span class="pre">werkzeug.secure_filename()</span></code> is explained a little bit later. The
|
||||
<code class="docutils literal notranslate"><span class="pre">UPLOAD_FOLDER</span></code> is where we will store the uploaded files and the
|
||||
<code class="docutils literal notranslate"><span class="pre">ALLOWED_EXTENSIONS</span></code> is the set of allowed file extensions.</p>
|
||||
<p>Why do we limit the extensions that are allowed? You probably don’t want
|
||||
your users to be able to upload everything there if the server is directly
|
||||
sending out the data to the client. That way you can make sure that users
|
||||
are not able to upload HTML files that would cause XSS problems (see
|
||||
<a class="reference internal" href="../web-security.html#security-xss"><span class="std std-ref">Cross-Site Scripting (XSS)</span></a>). Also make sure to disallow <code class="docutils literal notranslate"><span class="pre">.php</span></code> files if the server
|
||||
executes them, but who has PHP installed on their server, right? :)</p>
|
||||
<p>Next the functions that check if an extension is valid and that uploads
|
||||
the file and redirects the user to the URL for the uploaded file:</p>
|
||||
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">allowed_file</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="n">filename</span> <span class="ow">and</span> \
|
||||
<span class="n">filename</span><span class="o">.</span><span class="n">rsplit</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="ow">in</span> <span class="n">ALLOWED_EXTENSIONS</span>
|
||||
|
||||
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'/'</span><span class="p">,</span> <span class="n">methods</span><span class="o">=</span><span class="p">[</span><span class="s1">'GET'</span><span class="p">,</span> <span class="s1">'POST'</span><span class="p">])</span>
|
||||
<span class="k">def</span><span class="w"> </span><span class="nf">upload_file</span><span class="p">():</span>
|
||||
<span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">method</span> <span class="o">==</span> <span class="s1">'POST'</span><span class="p">:</span>
|
||||
<span class="c1"># check if the post request has the file part</span>
|
||||
<span class="k">if</span> <span class="s1">'file'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">request</span><span class="o">.</span><span class="n">files</span><span class="p">:</span>
|
||||
<span class="n">flash</span><span class="p">(</span><span class="s1">'No file part'</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">redirect</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">url</span><span class="p">)</span>
|
||||
<span class="n">file</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">files</span><span class="p">[</span><span class="s1">'file'</span><span class="p">]</span>
|
||||
<span class="c1"># If the user does not select a file, the browser submits an</span>
|
||||
<span class="c1"># empty file without a filename.</span>
|
||||
<span class="k">if</span> <span class="n">file</span><span class="o">.</span><span class="n">filename</span> <span class="o">==</span> <span class="s1">''</span><span class="p">:</span>
|
||||
<span class="n">flash</span><span class="p">(</span><span class="s1">'No selected file'</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">redirect</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">url</span><span class="p">)</span>
|
||||
<span class="k">if</span> <span class="n">file</span> <span class="ow">and</span> <span class="n">allowed_file</span><span class="p">(</span><span class="n">file</span><span class="o">.</span><span class="n">filename</span><span class="p">):</span>
|
||||
<span class="n">filename</span> <span class="o">=</span> <span class="n">secure_filename</span><span class="p">(</span><span class="n">file</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
|
||||
<span class="n">file</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="p">[</span><span class="s1">'UPLOAD_FOLDER'</span><span class="p">],</span> <span class="n">filename</span><span class="p">))</span>
|
||||
<span class="k">return</span> <span class="n">redirect</span><span class="p">(</span><span class="n">url_for</span><span class="p">(</span><span class="s1">'download_file'</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="n">filename</span><span class="p">))</span>
|
||||
<span class="k">return</span> <span class="s1">'''</span>
|
||||
<span class="s1"> <!doctype html></span>
|
||||
<span class="s1"> <title>Upload new File</title></span>
|
||||
<span class="s1"> <h1>Upload new File</h1></span>
|
||||
<span class="s1"> <form method=post enctype=multipart/form-data></span>
|
||||
<span class="s1"> <input type=file name=file></span>
|
||||
<span class="s1"> <input type=submit value=Upload></span>
|
||||
<span class="s1"> </form></span>
|
||||
<span class="s1"> '''</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>So what does that <a class="reference external" href="https://werkzeug.palletsprojects.com/en/stable/utils/#werkzeug.utils.secure_filename" title="(in Werkzeug v3.1.x)"><code class="xref py py-func docutils literal notranslate"><span class="pre">secure_filename()</span></code></a> function actually do?
|
||||
Now the problem is that there is that principle called “never trust user
|
||||
input”. This is also true for the filename of an uploaded file. All
|
||||
submitted form data can be forged, and filenames can be dangerous. For
|
||||
the moment just remember: always use that function to secure a filename
|
||||
before storing it directly on the filesystem.</p>
|
||||
<div class="admonition-information-for-the-pros admonition">
|
||||
<p class="admonition-title">Information for the Pros</p>
|
||||
<p>So you’re interested in what that <a class="reference external" href="https://werkzeug.palletsprojects.com/en/stable/utils/#werkzeug.utils.secure_filename" title="(in Werkzeug v3.1.x)"><code class="xref py py-func docutils literal notranslate"><span class="pre">secure_filename()</span></code></a>
|
||||
function does and what the problem is if you’re not using it? So just
|
||||
imagine someone would send the following information as <code class="code docutils literal notranslate"><span class="pre">filename</span></code> to
|
||||
your application:</p>
|
||||
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">filename</span> <span class="o">=</span> <span class="s2">"../../../../home/username/.bashrc"</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>Assuming the number of <code class="docutils literal notranslate"><span class="pre">../</span></code> is correct and you would join this with
|
||||
the <code class="docutils literal notranslate"><span class="pre">UPLOAD_FOLDER</span></code> the user might have the ability to modify a file on
|
||||
the server’s filesystem he or she should not modify. This does require some
|
||||
knowledge about how the application looks like, but trust me, hackers
|
||||
are patient :)</p>
|
||||
<p>Now let’s look how that function works:</p>
|
||||
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">secure_filename</span><span class="p">(</span><span class="s1">'../../../../home/username/.bashrc'</span><span class="p">)</span>
|
||||
<span class="go">'home_username_.bashrc'</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<p>We want to be able to serve the uploaded files so they can be downloaded
|
||||
by users. We’ll define a <code class="docutils literal notranslate"><span class="pre">download_file</span></code> view to serve files in the
|
||||
upload folder by name. <code class="docutils literal notranslate"><span class="pre">url_for("download_file",</span> <span class="pre">name=name)</span></code> generates
|
||||
download URLs.</p>
|
||||
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">flask</span><span class="w"> </span><span class="kn">import</span> <span class="n">send_from_directory</span>
|
||||
|
||||
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'/uploads/<name>'</span><span class="p">)</span>
|
||||
<span class="k">def</span><span class="w"> </span><span class="nf">download_file</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">send_from_directory</span><span class="p">(</span><span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="p">[</span><span class="s2">"UPLOAD_FOLDER"</span><span class="p">],</span> <span class="n">name</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>If you’re using middleware or the HTTP server to serve files, you can
|
||||
register the <code class="docutils literal notranslate"><span class="pre">download_file</span></code> endpoint as <code class="docutils literal notranslate"><span class="pre">build_only</span></code> so <code class="docutils literal notranslate"><span class="pre">url_for</span></code>
|
||||
will work without a view function.</p>
|
||||
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">app</span><span class="o">.</span><span class="n">add_url_rule</span><span class="p">(</span>
|
||||
<span class="s2">"/uploads/<name>"</span><span class="p">,</span> <span class="n">endpoint</span><span class="o">=</span><span class="s2">"download_file"</span><span class="p">,</span> <span class="n">build_only</span><span class="o">=</span><span class="kc">True</span>
|
||||
<span class="p">)</span>
|
||||
<span class="n">celery_app</span> <span class="o">=</span> <span class="n">celery_init_app</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>Point the <code class="docutils literal notranslate"><span class="pre">celery</span> <span class="pre">worker</span></code> command at this and it will find the <code class="docutils literal notranslate"><span class="pre">celery_app</span></code> object.</p>
|
||||
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>$ celery -A example worker --loglevel INFO
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>You can also run the <code class="docutils literal notranslate"><span class="pre">celery</span> <span class="pre">beat</span></code> command to run tasks on a schedule. See Celery’s
|
||||
docs for more information about defining schedules.</p>
|
||||
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>$ celery -A example beat --loglevel INFO
|
||||
</pre></div>
|
||||
</div>
|
||||
</section>
|
||||
<section id="application-factory">
|
||||
<h2>Application Factory<a class="headerlink" href="#application-factory" title="Link to this heading">¶</a></h2>
|
||||
<p>When using the Flask application factory pattern, call the <code class="docutils literal notranslate"><span class="pre">celery_init_app</span></code> function
|
||||
inside the factory. It sets <code class="docutils literal notranslate"><span class="pre">app.extensions["celery"]</span></code> to the Celery app object, which
|
||||
can be used to get the Celery app from the Flask app returned by the factory.</p>
|
||||
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">create_app</span><span class="p">()</span> <span class="o">-></span> <span class="n">Flask</span><span class="p">:</span>
|
||||
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
|
||||
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_mapping</span><span class="p">(</span>
|
||||
<span class="n">CELERY</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span>
|
||||
<span class="n">broker_url</span><span class="o">=</span><span class="s2">"redis://localhost"</span><span class="p">,</span>
|
||||
<span class="n">result_backend</span><span class="o">=</span><span class="s2">"redis://localhost"</span><span class="p">,</span>
|
||||
<span class="n">task_ignore_result</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
|
||||
<span class="p">),</span>
|
||||
<span class="p">)</span>
|
||||
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_prefixed_env</span><span class="p">()</span>
|
||||
<span class="n">celery_init_app</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">app</span>
|
||||
</pre></div>
|
||||
<section id="improving-uploads">
|
||||
<h2>Improving Uploads<a class="headerlink" href="#improving-uploads" title="Link to this heading">¶</a></h2>
|
||||
<details class="changelog">
|
||||
<summary>Changelog</summary><div class="versionadded">
|
||||
<p><span class="versionmodified added">Added in version 0.6.</span></p>
|
||||
</div>
|
||||
<p>To use <code class="docutils literal notranslate"><span class="pre">celery</span></code> commands, Celery needs an app object, but that’s no longer directly
|
||||
available. Create a <code class="docutils literal notranslate"><span class="pre">make_celery.py</span></code> file that calls the Flask app factory and gets
|
||||
the Celery app from the returned Flask app.</p>
|
||||
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">example</span><span class="w"> </span><span class="kn">import</span> <span class="n">create_app</span>
|
||||
</details><p>So how exactly does Flask handle uploads? Well it will store them in the
|
||||
webserver’s memory if the files are reasonably small, otherwise in a
|
||||
temporary location (as returned by <a class="reference external" href="https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate"><span class="pre">tempfile.gettempdir()</span></code></a>). But how
|
||||
do you specify the maximum file size after which an upload is aborted? By
|
||||
default Flask will happily accept file uploads with an unlimited amount of
|
||||
memory, but you can limit that by setting the <code class="docutils literal notranslate"><span class="pre">MAX_CONTENT_LENGTH</span></code>
|
||||
config key:</p>
|
||||
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">flask</span><span class="w"> </span><span class="kn">import</span> <span class="n">Flask</span><span class="p">,</span> <span class="n">Request</span>
|
||||
|
||||
<span class="n">flask_app</span> <span class="o">=</span> <span class="n">create_app</span><span class="p">()</span>
|
||||
<span class="n">celery_app</span> <span class="o">=</span> <span class="n">flask_app</span><span class="o">.</span><span class="n">extensions</span><span class="p">[</span><span class="s2">"celery"</span><span class="p">]</span>
|
||||
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
|
||||
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="p">[</span><span class="s1">'MAX_CONTENT_LENGTH'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">16</span> <span class="o">*</span> <span class="mi">1000</span> <span class="o">*</span> <span class="mi">1000</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>Point the <code class="docutils literal notranslate"><span class="pre">celery</span></code> command to this file.</p>
|
||||
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>$ celery -A make_celery worker --loglevel INFO
|
||||
$ celery -A make_celery beat --loglevel INFO
|
||||
</pre></div>
|
||||
<p>The code above will limit the maximum allowed payload to 16 megabytes.
|
||||
If a larger file is transmitted, Flask will raise a
|
||||
<a class="reference external" href="https://werkzeug.palletsprojects.com/en/stable/exceptions/#werkzeug.exceptions.RequestEntityTooLarge" title="(in Werkzeug v3.1.x)"><code class="xref py py-exc docutils literal notranslate"><span class="pre">RequestEntityTooLarge</span></code></a> exception.</p>
|
||||
<div class="admonition-connection-reset-issue admonition">
|
||||
<p class="admonition-title">Connection Reset Issue</p>
|
||||
<p>When using the local development server, you may get a connection
|
||||
reset error instead of a 413 response. You will get the correct
|
||||
status response when running the app with a production WSGI server.</p>
|
||||
</div>
|
||||
<p>This feature was added in Flask 0.6 but can be achieved in older versions
|
||||
as well by subclassing the request object. For more information on that
|
||||
consult the Werkzeug documentation on file handling.</p>
|
||||
</section>
|
||||
<section id="defining-tasks">
|
||||
<h2>Defining Tasks<a class="headerlink" href="#defining-tasks" title="Link to this heading">¶</a></h2>
|
||||
<p>Using <code class="docutils literal notranslate"><span class="pre">@celery_app.task</span></code> to decorate task functions requires access to the
|
||||
<code class="docutils literal notranslate"><span class="pre">celery_app</span></code> object, which won’t be available when using the factory pattern. It also
|
||||
means that the decorated tasks are tied to the specific Flask and Celery app instances,
|
||||
which could be an issue during testing if you change configuration for a test.</p>
|
||||
<p>Instead, use Celery’s <code class="docutils literal notranslate"><span class="pre">@shared_task</span></code> decorator. This creates task objects that will
|
||||
access whatever the “current app” is, which is a similar concept to Flask’s blueprints
|
||||
and app context. This is why we called <code class="docutils literal notranslate"><span class="pre">celery_app.set_default()</span></code> above.</p>
|
||||
<p>Here’s an example task that adds two numbers together and returns the result.</p>
|
||||
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">celery</span><span class="w"> </span><span class="kn">import</span> <span class="n">shared_task</span>
|
||||
|
||||
<span class="nd">@shared_task</span><span class="p">(</span><span class="n">ignore_result</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
|
||||
<span class="k">def</span><span class="w"> </span><span class="nf">add_together</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
|
||||
<span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>Earlier, we configured Celery to ignore task results by default. Since we want to know
|
||||
the return value of this task, we set <code class="docutils literal notranslate"><span class="pre">ignore_result=False</span></code>. On the other hand, a task
|
||||
that didn’t need a result, such as sending an email, wouldn’t set this.</p>
|
||||
<section id="upload-progress-bars">
|
||||
<h2>Upload Progress Bars<a class="headerlink" href="#upload-progress-bars" title="Link to this heading">¶</a></h2>
|
||||
<p>A while ago many developers had the idea to read the incoming file in
|
||||
small chunks and store the upload progress in the database to be able to
|
||||
poll the progress with JavaScript from the client. The client asks the
|
||||
server every 5 seconds how much it has transmitted, but this is
|
||||
something it should already know.</p>
|
||||
</section>
|
||||
<section id="calling-tasks">
|
||||
<h2>Calling Tasks<a class="headerlink" href="#calling-tasks" title="Link to this heading">¶</a></h2>
|
||||
<p>The decorated function becomes a task object with methods to call it in the background.
|
||||
The simplest way is to use the <code class="docutils literal notranslate"><span class="pre">delay(*args,</span> <span class="pre">**kwargs)</span></code> method. See Celery’s docs for
|
||||
more methods.</p>
|
||||
<p>A Celery worker must be running to run the task. Starting a worker is shown in the
|
||||
previous sections.</p>
|
||||
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">flask</span><span class="w"> </span><span class="kn">import</span> <span class="n">request</span>
|
||||
|
||||
<span class="nd">@app</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s2">"/add"</span><span class="p">)</span>
|
||||
<span class="k">def</span><span class="w"> </span><span class="nf">start_add</span><span class="p">()</span> <span class="o">-></span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">object</span><span class="p">]:</span>
|
||||
<span class="n">a</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">form</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"a"</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">)</span>
|
||||
<span class="n">b</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">form</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"b"</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">)</span>
|
||||
<span class="n">result</span> <span class="o">=</span> <span class="n">add_together</span><span class="o">.</span><span class="n">delay</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="p">{</span><span class="s2">"result_id"</span><span class="p">:</span> <span class="n">result</span><span class="o">.</span><span class="n">id</span><span class="p">}</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>The route doesn’t get the task’s result immediately. That would defeat the purpose by
|
||||
blocking the response. Instead, we return the running task’s result id, which we can use
|
||||
later to get the result.</p>
|
||||
</section>
|
||||
<section id="getting-results">
|
||||
<h2>Getting Results<a class="headerlink" href="#getting-results" title="Link to this heading">¶</a></h2>
|
||||
<p>To fetch the result of the task we started above, we’ll add another route that takes the
|
||||
result id we returned before. We return whether the task is finished (ready), whether it
|
||||
finished successfully, and what the return value (or error) was if it is finished.</p>
|
||||
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">celery.result</span><span class="w"> </span><span class="kn">import</span> <span class="n">AsyncResult</span>
|
||||
|
||||
<span class="nd">@app</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"/result/<id>"</span><span class="p">)</span>
|
||||
<span class="k">def</span><span class="w"> </span><span class="nf">task_result</span><span class="p">(</span><span class="nb">id</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">object</span><span class="p">]:</span>
|
||||
<span class="n">result</span> <span class="o">=</span> <span class="n">AsyncResult</span><span class="p">(</span><span class="nb">id</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="p">{</span>
|
||||
<span class="s2">"ready"</span><span class="p">:</span> <span class="n">result</span><span class="o">.</span><span class="n">ready</span><span class="p">(),</span>
|
||||
<span class="s2">"successful"</span><span class="p">:</span> <span class="n">result</span><span class="o">.</span><span class="n">successful</span><span class="p">(),</span>
|
||||
<span class="s2">"value"</span><span class="p">:</span> <span class="n">result</span><span class="o">.</span><span class="n">result</span> <span class="k">if</span> <span class="n">result</span><span class="o">.</span><span class="n">ready</span><span class="p">()</span> <span class="k">else</span> <span class="kc">None</span><span class="p">,</span>
|
||||
<span class="p">}</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>Now you can start the task using the first route, then poll for the result using the
|
||||
second route. This keeps the Flask request workers from being blocked waiting for tasks
|
||||
to finish.</p>
|
||||
<p>The Flask repository contains <a class="reference external" href="https://github.com/pallets/flask/tree/main/examples/celery">an example</a>
|
||||
using JavaScript to submit tasks and poll for progress and results.</p>
|
||||
</section>
|
||||
<section id="passing-data-to-tasks">
|
||||
<h2>Passing Data to Tasks<a class="headerlink" href="#passing-data-to-tasks" title="Link to this heading">¶</a></h2>
|
||||
<p>The “add” task above took two integers as arguments. To pass arguments to tasks, Celery
|
||||
has to serialize them to a format that it can pass to other processes. Therefore,
|
||||
passing complex objects is not recommended. For example, it would be impossible to pass
|
||||
a SQLAlchemy model object, since that object is probably not serializable and is tied to
|
||||
the session that queried it.</p>
|
||||
<p>Pass the minimal amount of data necessary to fetch or recreate any complex data within
|
||||
the task. Consider a task that will run when the logged in user asks for an archive of
|
||||
their data. The Flask request knows the logged in user, and has the user object queried
|
||||
from the database. It got that by querying the database for a given id, so the task can
|
||||
do the same thing. Pass the user’s id rather than the user object.</p>
|
||||
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nd">@shared_task</span>
|
||||
<span class="k">def</span><span class="w"> </span><span class="nf">generate_user_archive</span><span class="p">(</span><span class="n">user_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
|
||||
<span class="n">user</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">User</span><span class="p">,</span> <span class="n">user_id</span><span class="p">)</span>
|
||||
<span class="o">...</span>
|
||||
|
||||
<span class="n">generate_user_archive</span><span class="o">.</span><span class="n">delay</span><span class="p">(</span><span class="n">current_user</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<section id="an-easier-solution">
|
||||
<h2>An Easier Solution<a class="headerlink" href="#an-easier-solution" title="Link to this heading">¶</a></h2>
|
||||
<p>Now there are better solutions that work faster and are more reliable. There
|
||||
are JavaScript libraries like <a class="reference external" href="https://jquery.com/">jQuery</a> that have form plugins to ease the
|
||||
construction of progress bar.</p>
|
||||
<p>Because the common pattern for file uploads exists almost unchanged in all
|
||||
applications dealing with uploads, there are also some Flask extensions that
|
||||
implement a full fledged upload mechanism that allows controlling which
|
||||
file extensions are allowed to be uploaded.</p>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
|
|
@ -247,23 +222,20 @@ do the same thing. Pass the user’s id rather than the user object.</p>
|
|||
<span id="sidebar-top"></span>
|
||||
<div class="sphinxsidebar" role="navigation" aria-label="Main">
|
||||
<div class="sphinxsidebarwrapper">
|
||||
|
||||
|
||||
|
||||
|
||||
<p class="logo"><a href="../index.html">
|
||||
<img class="logo" src="../_static/flask-vertical.png" alt="Logo of Flask"/>
|
||||
</a></p>
|
||||
|
||||
|
||||
|
||||
<h3>Contents</h3>
|
||||
<ul>
|
||||
<li><a class="reference internal" href="#">Background Tasks with Celery</a><ul>
|
||||
<li><a class="reference internal" href="#install">Install</a></li>
|
||||
<li><a class="reference internal" href="#integrate-celery-with-flask">Integrate Celery with Flask</a></li>
|
||||
<li><a class="reference internal" href="#application-factory">Application Factory</a></li>
|
||||
<li><a class="reference internal" href="#defining-tasks">Defining Tasks</a></li>
|
||||
<li><a class="reference internal" href="#calling-tasks">Calling Tasks</a></li>
|
||||
<li><a class="reference internal" href="#getting-results">Getting Results</a></li>
|
||||
<li><a class="reference internal" href="#passing-data-to-tasks">Passing Data to Tasks</a></li>
|
||||
<li><a class="reference internal" href="#">Uploading Files</a><ul>
|
||||
<li><a class="reference internal" href="#a-gentle-introduction">A Gentle Introduction</a></li>
|
||||
<li><a class="reference internal" href="#improving-uploads">Improving Uploads</a></li>
|
||||
<li><a class="reference internal" href="#upload-progress-bars">Upload Progress Bars</a></li>
|
||||
<li><a class="reference internal" href="#an-easier-solution">An Easier Solution</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
|
|
@ -273,8 +245,8 @@ do the same thing. Pass the user’s id rather than the user object.</p>
|
|||
<ul>
|
||||
<li><a href="index.html">Patterns for Flask</a>
|
||||
<ul>
|
||||
<li>Previous: <a href="requestchecksum.html" title="previous chapter">Request Content Checksums</a>
|
||||
<li>Next: <a href="subclassing.html" title="next chapter">Subclassing Flask</a></ul>
|
||||
<li>Previous: <a href="sqlalchemy.html" title="previous chapter">SQLAlchemy in Flask</a>
|
||||
<li>Next: <a href="caching.html" title="next chapter">Caching</a></ul>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
|
|
@ -298,4 +270,4 @@ do the same thing. Pass the user’s id rather than the user object.</p>
|
|||
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 8.1.3.
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
</html>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue