orbit-oss/flask
4
4
Fork 2
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #3562 from pallets/remove-simplejson

Browse source 
Remove simplejson and deprecate encoding options
This commit is contained in:
David Lord 2020-04-07 13:40:39 -07:00 • committed by GitHub
commit e69b49bd3d
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
7 changed files with 210 additions and 284 deletions
Download patch file Download diff file Expand all files Collapse all files

3
CHANGES.rst
View file

@ -6,6 +6,9 @@ Version 2.0.0
Unreleased
- Drop support for Python 2 and 3.5.
- JSON support no longer uses simplejson. To use another JSON module,
override ``app.json_encoder`` and ``json_decoder``. :issue:`3555`
- The ``encoding`` option to JSON functions is deprecated. :pr:`3562`
- Add :meth:`sessions.SessionInterface.get_cookie_name` to allow
setting the session cookie name dynamically. :pr:`3369`
- Add :meth:`Config.from_file` to load config using arbitrary file

55
docs/api.rst
View file

@ -279,58 +279,34 @@ Message Flashing
.. autofunction:: get_flashed_messages
JSON Support
------------
.. module:: flask.json
Flask uses ``simplejson`` for the JSON implementation. Since simplejson
is provided by both the standard library as well as extension, Flask will
try simplejson first and then fall back to the stdlib json module. On top
of that it will delegate access to the current application's JSON encoders
and decoders for easier customization.
Flask uses the built-in :mod:`json` module for handling JSON. It will
use the current blueprint's or application's JSON encoder and decoder
for easier customization. By default it handles some extra data types:
So for starters instead of doing::
- :class:`datetime.datetime` and :class:`datetime.date` are serialized
to :rfc:`822` strings. This is the same as the HTTP date format.
- :class:`uuid.UUID` is serialized to a string.
- :class:`dataclasses.dataclass` is passed to
:func:`dataclasses.asdict`.
- :class:`~markupsafe.Markup` (or any object with a ``__html__``
method) will call the ``__html__`` method to get a string.
try:
import simplejson as json
except ImportError:
import json
You can instead just do this::
from flask import json
For usage examples, read the :mod:`json` documentation in the standard
library. The following extensions are by default applied to the stdlib's
JSON module:
1. ``datetime`` objects are serialized as :rfc:`822` strings.
2. Any object with an ``__html__`` method (like :class:`~flask.Markup`)
will have that method called and then the return value is serialized
as string.
The :func:`~htmlsafe_dumps` function of this json module is also available
as a filter called ``|tojson`` in Jinja2. Note that in versions of Flask prior
to Flask 0.10, you must disable escaping with ``|safe`` if you intend to use
``|tojson`` output inside ``script`` tags. In Flask 0.10 and above, this
happens automatically (but it's harmless to include ``|safe`` anyway).
:func:`~htmlsafe_dumps` is also available as the ``|tojson`` template
filter. The filter marks the output with ``|safe`` so it can be used
inside ``script`` tags.
.. sourcecode:: html+jinja
<script type=text/javascript>
doSomethingWith({{ user.username|tojson|safe }});
renderChart({{ axis_data|tojson }});
</script>
.. admonition:: Auto-Sort JSON Keys
The configuration variable :data:`JSON_SORT_KEYS` can be set to
``False`` to stop Flask from auto-sorting keys. By default sorting
is enabled and outside of the app context sorting is turned on.
Notice that disabling key sorting can cause issues when using
content based HTTP caches and Python's hash randomization feature.
.. autofunction:: jsonify
.. autofunction:: dumps
@ -349,6 +325,7 @@ happens automatically (but it's harmless to include ``|safe`` anyway).
.. automodule:: flask.json.tag
Template Rendering
------------------

4
docs/installation.rst
View file

@ -39,16 +39,12 @@ These distributions will not be installed automatically. Flask will detect and
use them if you install them.
* `Blinker`_ provides support for :doc:`signals`.
* `SimpleJSON`_ is a fast JSON implementation that is compatible with
Python's ``json`` module. It is preferred for JSON operations if it is
installed.
* `python-dotenv`_ enables support for :ref:`dotenv` when running ``flask``
commands.
* `Watchdog`_ provides a faster, more efficient reloader for the development
server.
.. _Blinker: https://pythonhosted.org/blinker/
.. _SimpleJSON: https://simplejson.readthedocs.io/
.. _python-dotenv: https://github.com/theskumar/python-dotenv#readme
.. _watchdog: https://pythonhosted.org/watchdog/

398
src/flask/json/__init__.py
View file

@ -1,11 +1,11 @@
import codecs
import io
import json as _json
import uuid
import warnings
from datetime import date
from datetime import datetime
from itsdangerous import json as _json
from jinja2 import Markup
from markupsafe import Markup
from werkzeug.http import http_date
from ..globals import current_app
@ -17,66 +17,29 @@ except ImportError:
# Python < 3.7
dataclasses = None
# Figure out if simplejson escapes slashes. This behavior was changed
# from one version to another without reason.
_slash_escape = "\\/" not in _json.dumps("/")
__all__ = [
"dump",
"dumps",
"load",
"loads",
"htmlsafe_dump",
"htmlsafe_dumps",
"JSONDecoder",
"JSONEncoder",
"jsonify",
]
def _wrap_reader_for_text(fp, encoding):
if isinstance(fp.read(0), bytes):
fp = io.TextIOWrapper(io.BufferedReader(fp), encoding)
return fp
def _wrap_writer_for_text(fp, encoding):
try:
fp.write("")
except TypeError:
fp = io.TextIOWrapper(fp, encoding)
return fp
class JSONEncoder(_json.JSONEncoder):
"""The default Flask JSON encoder. This one extends the default
encoder by also supporting ``datetime``, ``UUID``, ``dataclasses``,
and ``Markup`` objects.
"""The default JSON encoder. Handles extra types compared to the
built-in :class:`json.JSONEncoder`.
``datetime`` objects are serialized as RFC 822 datetime strings.
This is the same as the HTTP date format.
- :class:`datetime.datetime` and :class:`datetime.date` are
serialized to :rfc:`822` strings. This is the same as the HTTP
date format.
- :class:`uuid.UUID` is serialized to a string.
- :class:`dataclasses.dataclass` is passed to
:func:`dataclasses.asdict`.
- :class:`~markupsafe.Markup` (or any object with a ``__html__``
method) will call the ``__html__`` method to get a string.
In order to support more data types, override the :meth:`default`
method.
Assign a subclass of this to :attr:`flask.Flask.json_encoder` or
:attr:`flask.Blueprint.json_encoder` to override the default.
"""
def default(self, o):
"""Implement this method in a subclass such that it returns a
serializable object for ``o``, or calls the base implementation (to
raise a :exc:`TypeError`).
For example, to support arbitrary iterators, you could implement
default like this::
def default(self, o):
try:
iterable = iter(o)
except TypeError:
pass
else:
return list(iterable)
return JSONEncoder.default(self, o)
"""Convert ``o`` to a JSON serializable type. See
:meth:`json.JSONEncoder.default`. Python does not support
overriding how basic types like ``str`` or ``list`` are
serialized, they are handled before this method.
"""
if isinstance(o, datetime):
return http_date(o.utctimetuple())
@ -88,14 +51,17 @@ class JSONEncoder(_json.JSONEncoder):
return dataclasses.asdict(o)
if hasattr(o, "__html__"):
return str(o.__html__())
return _json.JSONEncoder.default(self, o)
return super().default(self, o)
class JSONDecoder(_json.JSONDecoder):
"""The default JSON decoder. This one does not change the behavior from
the default simplejson decoder. Consult the :mod:`json` documentation
for more information. This decoder is not only used for the load
functions of this module but also :attr:`~flask.Request`.
"""The default JSON decoder.
This does not change any behavior from the built-in
:class:`json.JSONDecoder`.
Assign a subclass of this to :attr:`flask.Flask.json_decoder` or
:attr:`flask.Blueprint.json_decoder` to override the default.
"""
@ -106,13 +72,9 @@ def _dump_arg_defaults(kwargs, app=None):
if app:
bp = app.blueprints.get(request.blueprint) if request else None
kwargs.setdefault(
"cls", bp.json_encoder if bp and bp.json_encoder else app.json_encoder
)
if not app.config["JSON_AS_ASCII"]:
kwargs.setdefault("ensure_ascii", False)
cls = bp.json_encoder if bp and bp.json_encoder else app.json_encoder
kwargs.setdefault("cls", cls)
kwargs.setdefault("ensure_ascii", app.config["JSON_AS_ASCII"])
kwargs.setdefault("sort_keys", app.config["JSON_SORT_KEYS"])
else:
kwargs.setdefault("sort_keys", True)
@ -126,223 +88,235 @@ def _load_arg_defaults(kwargs, app=None):
if app:
bp = app.blueprints.get(request.blueprint) if request else None
kwargs.setdefault(
"cls", bp.json_decoder if bp and bp.json_decoder else app.json_decoder
)
cls = bp.json_decoder if bp and bp.json_decoder else app.json_decoder
kwargs.setdefault("cls", cls)
else:
kwargs.setdefault("cls", JSONDecoder)
def detect_encoding(data):
"""Detect which UTF codec was used to encode the given bytes.
The latest JSON standard (:rfc:`8259`) suggests that only UTF-8 is
accepted. Older documents allowed 8, 16, or 32. 16 and 32 can be big
or little endian. Some editors or libraries may prepend a BOM.
:param data: Bytes in unknown UTF encoding.
:return: UTF encoding name
"""
head = data[:4]
if head[:3] == codecs.BOM_UTF8:
return "utf-8-sig"
if b"\x00" not in head:
return "utf-8"
if head in (codecs.BOM_UTF32_BE, codecs.BOM_UTF32_LE):
return "utf-32"
if head[:2] in (codecs.BOM_UTF16_BE, codecs.BOM_UTF16_LE):
return "utf-16"
if len(head) == 4:
if head[:3] == b"\x00\x00\x00":
return "utf-32-be"
if head[::2] == b"\x00\x00":
return "utf-16-be"
if head[1:] == b"\x00\x00\x00":
return "utf-32-le"
if head[1::2] == b"\x00\x00":
return "utf-16-le"
if len(head) == 2:
return "utf-16-be" if head.startswith(b"\x00") else "utf-16-le"
return "utf-8"
def dumps(obj, app=None, **kwargs):
"""Serialize ``obj`` to a JSON-formatted string. If there is an
app context pushed, use the current app's configured encoder
(:attr:`~flask.Flask.json_encoder`), or fall back to the default
:class:`JSONEncoder`.
"""Serialize an object to a string of JSON.
Takes the same arguments as the built-in :func:`json.dumps`, and
does some extra configuration based on the application. If the
simplejson package is installed, it is preferred.
Takes the same arguments as the built-in :func:`json.dumps`, with
some defaults from application configuration.
:param obj: Object to serialize to JSON.
:param app: App instance to use to configure the JSON encoder.
Uses ``current_app`` if not given, and falls back to the default
encoder when not in an app context.
:param kwargs: Extra arguments passed to :func:`json.dumps`.
:param app: Use this app's config instead of the active app context
or defaults.
:param kwargs: Extra arguments passed to func:`json.dumps`.
.. versionchanged:: 2.0
``encoding`` is deprecated and will be removed in 2.1.
.. versionchanged:: 1.0.3
``app`` can be passed directly, rather than requiring an app
context for configuration.
"""
_dump_arg_defaults(kwargs, app=app)
encoding = kwargs.pop("encoding", None)
rv = _json.dumps(obj, **kwargs)
if encoding is not None and isinstance(rv, str):
rv = rv.encode(encoding)
if encoding is not None:
warnings.warn(
"'encoding' is deprecated and will be removed in 2.1.",
DeprecationWarning,
stacklevel=2,
)
if isinstance(rv, str):
return rv.encode(encoding)
return rv
def dump(obj, fp, app=None, **kwargs):
"""Like :func:`dumps` but writes into a file object."""
"""Serialize an object to JSON written to a file object.
Takes the same arguments as the built-in :func:`json.dump`, with
some defaults from application configuration.
:param obj: Object to serialize to JSON.
:param fp: File object to write JSON to.
:param app: Use this app's config instead of the active app context
or defaults.
:param kwargs: Extra arguments passed to func:`json.dump`.
.. versionchanged:: 2.0
Writing to a binary file, and the ``encoding`` argument, is
deprecated and will be removed in 2.1.
"""
_dump_arg_defaults(kwargs, app=app)
encoding = kwargs.pop("encoding", None)
if encoding is not None:
fp = _wrap_writer_for_text(fp, encoding)
show_warning = encoding is not None
try:
fp.write("")
except TypeError:
show_warning = True
fp = io.TextIOWrapper(fp, encoding or "utf-8")
if show_warning:
warnings.warn(
"Writing to a binary file, and the 'encoding' argument, is"
" deprecated and will be removed in 2.1.",
DeprecationWarning,
stacklevel=2,
)
_json.dump(obj, fp, **kwargs)
def loads(s, app=None, **kwargs):
"""Deserialize an object from a JSON-formatted string ``s``. If
there is an app context pushed, use the current app's configured
decoder (:attr:`~flask.Flask.json_decoder`), or fall back to the
default :class:`JSONDecoder`.
"""Deserialize an object from a string of JSON.
Takes the same arguments as the built-in :func:`json.loads`, and
does some extra configuration based on the application. If the
simplejson package is installed, it is preferred.
Takes the same arguments as the built-in :func:`json.loads`, with
some defaults from application configuration.
:param s: JSON string to deserialize.
:param app: App instance to use to configure the JSON decoder.
Uses ``current_app`` if not given, and falls back to the default
encoder when not in an app context.
:param kwargs: Extra arguments passed to :func:`json.dumps`.
:param app: Use this app's config instead of the active app context
or defaults.
:param kwargs: Extra arguments passed to func:`json.dump`.
.. versionchanged:: 2.0
``encoding`` is deprecated and will be removed in 2.1. The data
must be a string or UTF-8 bytes.
.. versionchanged:: 1.0.3
``app`` can be passed directly, rather than requiring an app
context for configuration.
"""
_load_arg_defaults(kwargs, app=app)
if isinstance(s, bytes):
encoding = kwargs.pop("encoding", None)
if encoding is None:
encoding = detect_encoding(s)
s = s.decode(encoding)
encoding = kwargs.pop("encoding", None)
if encoding is not None:
warnings.warn(
"'encoding' is deprecated and will be removed in 2.1. The"
" data must be a string or UTF-8 bytes.",
DeprecationWarning,
stacklevel=2,
)
if isinstance(s, bytes):
s = s.decode(encoding)
return _json.loads(s, **kwargs)
def load(fp, app=None, **kwargs):
"""Like :func:`loads` but reads from a file object."""
"""Deserialize an object from JSON read from a file object.
Takes the same arguments as the built-in :func:`json.load`, with
some defaults from application configuration.
:param fp: File object to read JSON from.
:param app: Use this app's config instead of the active app context
or defaults.
:param kwargs: Extra arguments passed to func:`json.load`.
.. versionchanged:: 2.0
``encoding`` is deprecated and will be removed in 2.1. The file
must be text mode, or binary mode with UTF-8 bytes.
"""
_load_arg_defaults(kwargs, app=app)
fp = _wrap_reader_for_text(fp, kwargs.pop("encoding", None) or "utf-8")
encoding = kwargs.pop("encoding", None)
if encoding is not None:
warnings.warn(
"'encoding' is deprecated and will be removed in 2.1. The"
" file must be text mode, or binary mode with UTF-8 bytes.",
DeprecationWarning,
stacklevel=2,
)
if isinstance(fp.read(0), bytes):
fp = io.TextIOWrapper(fp, encoding)
return _json.load(fp, **kwargs)
_htmlsafe_map = str.maketrans(
{"<": "\\u003c", ">": "\\u003e", "&": "\\u0026", "'": "\\u0027"}
)
def htmlsafe_dumps(obj, **kwargs):
"""Works exactly like :func:`dumps` but is safe for use in ``<script>``
tags. It accepts the same arguments and returns a JSON string. Note that
this is available in templates through the ``|tojson`` filter which will
also mark the result as safe. Due to how this function escapes certain
characters this is safe even if used outside of ``<script>`` tags.
"""Serialize an object to a string of JSON, replacing HTML-unsafe
characters with Unicode escapes. Otherwise behaves the same as
:func:`dumps`.
The following characters are escaped in strings:
This is available in templates as the ``|tojson`` filter, which will
also mark the result with ``|safe``.
- ``<``
- ``>``
- ``&``
- ``'``
This makes it safe to embed such strings in any place in HTML with the
notable exception of double quoted attributes. In that case single
quote your attributes or HTML escape it in addition.
The returned string is safe to render in HTML documents and
``<script>`` tags. The exception is in HTML attributes that are
double quoted; either use single quotes or the ``|forceescape``
filter.
.. versionchanged:: 0.10
This function's return value is now always safe for HTML usage, even
if outside of script tags or if used in XHTML. This rule does not
hold true when using this function in HTML attributes that are double
quoted. Always single quote attributes if you use the ``|tojson``
filter. Alternatively use ``|tojson|forceescape``.
Single quotes are escaped, making this safe to use in HTML,
``<script>`` tags, and single-quoted attributes without further
escaping.
"""
rv = (
dumps(obj, **kwargs)
.replace("<", "\\u003c")
.replace(">", "\\u003e")
.replace("&", "\\u0026")
.replace("'", "\\u0027")
)
if not _slash_escape:
rv = rv.replace("\\/", "/")
return rv
return dumps(obj, **kwargs).translate(_htmlsafe_map)
def htmlsafe_dump(obj, fp, **kwargs):
"""Like :func:`htmlsafe_dumps` but writes into a file object."""
fp.write(str(htmlsafe_dumps(obj, **kwargs)))
"""Serialize an object to JSON written to a file object, replacing
HTML-unsafe characters with Unicode escapes. See
:func:`htmlsafe_dumps` and :func:`dumps`.
"""
fp.write(htmlsafe_dumps(obj, **kwargs))
def jsonify(*args, **kwargs):
"""This function wraps :func:`dumps` to add a few enhancements that make
life easier. It turns the JSON output into a :class:`~flask.Response`
object with the :mimetype:`application/json` mimetype. For convenience, it
also converts multiple arguments into an array or multiple keyword arguments
into a dict. This means that both ``jsonify(1,2,3)`` and
``jsonify([1,2,3])`` serialize to ``[1,2,3]``.
"""Serialize data to JSON and wrap it in a :class:`~flask.Response`
with the :mimetype:`application/json` mimetype.
For clarity, the JSON serialization behavior has the following differences
from :func:`dumps`:
Uses :func:`dumps` to serialize the data, but ``args`` and
``kwargs`` are treated as data rather than arguments to
:func:`json.dumps`.
1. Single argument: Passed straight through to :func:`dumps`.
2. Multiple arguments: Converted to an array before being passed to
:func:`dumps`.
3. Multiple keyword arguments: Converted to a dict before being passed to
:func:`dumps`.
4. Both args and kwargs: Behavior undefined and will throw an exception.
1. Single argument: Treated as a single value.
2. Multiple arguments: Treated as a list of values.
``jsonify(1, 2, 3)`` is the same as ``jsonify([1, 2, 3])``.
3. Keyword arguments: Treated as a dict of values.
``jsonify(data=data, errors=errors)`` is the same as
``jsonify({"data": data, "errors": errors})``.
4. Passing both arguments and keyword arguments is not allowed as
it's not clear what should happen.
Example usage::
.. code-block:: python
from flask import jsonify
@app.route('/_get_current_user')
@app.route("/users/me")
def get_current_user():
return jsonify(username=g.user.username,
email=g.user.email,
id=g.user.id)
return jsonify(
username=g.user.username,
email=g.user.email,
id=g.user.id,
)
This will send a JSON response like this to the browser::
Will return a JSON response like this:
.. code-block:: javascript
{
"username": "admin",
"email": "admin@localhost",
"id": 42
"username": "admin",
"email": "admin@localhost",
"id": 42
}
The default output omits indents and spaces after separators. In
debug mode or if :data:`JSONIFY_PRETTYPRINT_REGULAR` is ``True``,
the output will be formatted to be easier to read.
.. versionchanged:: 0.11
Added support for serializing top-level arrays. This introduces
a security risk in ancient browsers. See :ref:`security-json`
for details.
This function's response will be pretty printed if the
``JSONIFY_PRETTYPRINT_REGULAR`` config parameter is set to True or the
Flask app is running in debug mode. Compressed (not pretty) formatting
currently means no indents and no spaces after separators.
a security risk in ancient browsers. See :ref:`security-json`.
.. versionadded:: 0.2
"""
indent = None
separators = (",", ":")

8
src/flask/json/tag.py
View file

@ -45,7 +45,7 @@ from base64 import b64encode
from datetime import datetime
from uuid import UUID
from jinja2 import Markup
from markupsafe import Markup
from werkzeug.http import http_date
from werkzeug.http import parse_date
@ -167,9 +167,9 @@ class TagBytes(JSONTag):
class TagMarkup(JSONTag):
"""Serialize anything matching the :class:`~flask.Markup` API by
"""Serialize anything matching the :class:`~markupsafe.Markup` API by
having a ``__html__`` method to the result of that method. Always
deserializes to an instance of :class:`~flask.Markup`."""
deserializes to an instance of :class:`~markupsafe.Markup`."""
__slots__ = ()
key = " m"
@ -222,7 +222,7 @@ class TaggedJSONSerializer:
* :class:`dict`
* :class:`tuple`
* :class:`bytes`
* :class:`~flask.Markup`
* :class:`~markupsafe.Markup`
* :class:`~uuid.UUID`
* :class:`~datetime.datetime`
"""

22
tests/test_helpers.py
View file

@ -13,7 +13,6 @@ from werkzeug.http import parse_cache_control_header
from werkzeug.http import parse_options_header
import flask
from flask import json
from flask.helpers import get_debug_flag
from flask.helpers import get_env
@ -64,26 +63,6 @@ class FixedOffset(datetime.tzinfo):
class TestJSON:
@pytest.mark.parametrize(
"value", (1, "t", True, False, None, [], [1, 2, 3], {}, {"foo": "🐍"})
)
@pytest.mark.parametrize(
"encoding",
(
"utf-8",
"utf-8-sig",
"utf-16-le",
"utf-16-be",
"utf-16",
"utf-32-le",
"utf-32-be",
"utf-32",
),
)
def test_detect_encoding(self, value, encoding):
data = json.dumps(value).encode(encoding)
assert json.loads(data) == value
@pytest.mark.parametrize("debug", (True, False))
def test_bad_request_debug_message(self, app, client, debug):
app.config["DEBUG"] = debug
@ -253,7 +232,6 @@ class TestJSON:
render = flask.render_template_string
rv = flask.json.htmlsafe_dumps("</script>")
assert rv == '"\\u003c/script\\u003e"'
assert type(rv) is str
rv = render('{{ "</script>"|tojson }}')
assert rv == '"\\u003c/script\\u003e"'
rv = render('{{ "<\0/script>"|tojson }}')

4
tox.ini
View file

@ -1,7 +1,7 @@
[tox]
envlist =
py{38,37,36,py3}
py38-{simplejson,devel,lowest}
py38-{devel,lowest}
style
docs
skip_missing_interpreters = true
@ -24,8 +24,6 @@ deps =
devel: https://github.com/pallets/itsdangerous/archive/master.tar.gz
devel: https://github.com/pallets/click/archive/master.tar.gz
simplejson: simplejson
commands =
pip install -q -e examples/tutorial[test]
pip install -q -e examples/javascript[test]