Merge tag 'v1.21.0rc1' into develop

Synapse 1.21.0rc1 (2020-10-01)
==============================

Features
--------

- Require the user to confirm that their password should be reset after clicking the email confirmation link. ([\#8004](matrix-org/synapse#8004))
- Add an admin API `GET /_synapse/admin/v1/event_reports` to read entries of table `event_reports`. Contributed by @dklimpel. ([\#8217](matrix-org/synapse#8217))
- Consolidate the SSO error template across all configuration. ([\#8248](matrix-org/synapse#8248), [\#8405](matrix-org/synapse#8405))
- Add a configuration option to specify a whitelist of domains that a user can be redirected to after validating their email or phone number. ([\#8275](matrix-org/synapse#8275), [\#8417](matrix-org/synapse#8417))
- Add experimental support for sharding event persister. ([\#8294](matrix-org/synapse#8294), [\#8387](matrix-org/synapse#8387), [\#8396](matrix-org/synapse#8396), [\#8419](matrix-org/synapse#8419))
- Add the room topic and avatar to the room details admin API. ([\#8305](matrix-org/synapse#8305))
- Add an admin API for querying rooms where a user is a member. Contributed by @dklimpel. ([\#8306](matrix-org/synapse#8306))
- Add `uk.half-shot.msc2778.login.application_service` login type to allow appservices to login. ([\#8320](matrix-org/synapse#8320))
- Add a configuration option that allows existing users to log in with OpenID Connect. Contributed by @BBBSnowball and @OmmyZhang. ([\#8345](matrix-org/synapse#8345))
- Add prometheus metrics for replication requests. ([\#8406](matrix-org/synapse#8406))
- Support passing additional single sign-on parameters to the client. ([\#8413](matrix-org/synapse#8413))
- Add experimental reporting of metrics on expensive rooms for state-resolution. ([\#8420](matrix-org/synapse#8420))
- Add experimental prometheus metric to track numbers of "large" rooms for state resolutiom. ([\#8425](matrix-org/synapse#8425))
- Add prometheus metrics to track federation delays. ([\#8430](matrix-org/synapse#8430))

Bugfixes
--------

- Fix a bug in the media repository where remote thumbnails with the same size but different crop methods would overwrite each other. Contributed by @deepbluev7. ([\#7124](matrix-org/synapse#7124))
- Fix inconsistent handling of non-existent push rules, and stop tracking the `enabled` state of removed push rules. ([\#7796](matrix-org/synapse#7796))
- Fix a longstanding bug when storing a media file with an empty `upload_name`. ([\#7905](matrix-org/synapse#7905))
- Fix messages not being sent over federation until an event is sent into the same room. ([\#8230](matrix-org/synapse#8230), [\#8247](matrix-org/synapse#8247), [\#8258](matrix-org/synapse#8258), [\#8272](matrix-org/synapse#8272), [\#8322](matrix-org/synapse#8322))
- Fix a longstanding bug where files that could not be thumbnailed would result in an Internal Server Error. ([\#8236](matrix-org/synapse#8236), [\#8435](matrix-org/synapse#8435))
- Upgrade minimum version of `canonicaljson` to version 1.4.0, to fix an unicode encoding issue. ([\#8262](matrix-org/synapse#8262))
- Fix longstanding bug which could lead to incomplete database upgrades on SQLite. ([\#8265](matrix-org/synapse#8265))
- Fix stack overflow when stderr is redirected to the logging system, and the logging system encounters an error. ([\#8268](matrix-org/synapse#8268))
- Fix a bug which cause the logging system to report errors, if `DEBUG` was enabled and no `context` filter was applied. ([\#8278](matrix-org/synapse#8278))
- Fix edge case where push could get delayed for a user until a later event was pushed. ([\#8287](matrix-org/synapse#8287))
- Fix fetching malformed events from remote servers. ([\#8324](matrix-org/synapse#8324))
- Fix `UnboundLocalError` from occuring when appservices send a malformed register request. ([\#8329](matrix-org/synapse#8329))
- Don't send push notifications to expired user accounts. ([\#8353](matrix-org/synapse#8353))
- Fix a regression in v1.19.0 with reactivating users through the admin API. ([\#8362](matrix-org/synapse#8362))
- Fix a bug where during device registration the length of the device name wasn't limited. ([\#8364](matrix-org/synapse#8364))
- Include `guest_access` in the fields that are checked for null bytes when updating `room_stats_state`. Broke in v1.7.2. ([\#8373](matrix-org/synapse#8373))
- Fix theoretical race condition where events are not sent down `/sync` if the synchrotron worker is restarted without restarting other workers. ([\#8374](matrix-org/synapse#8374))
- Fix a bug which could cause errors in rooms with malformed membership events, on servers using sqlite. ([\#8385](matrix-org/synapse#8385))
- Fix "Re-starting finished log context" warning when receiving an event we already had over federation. ([\#8398](matrix-org/synapse#8398))
- Fix incorrect handling of timeouts on outgoing HTTP requests. ([\#8400](matrix-org/synapse#8400))
- Fix a regression in v1.20.0 in the `synapse_port_db` script regarding the `ui_auth_sessions_ips` table. ([\#8410](matrix-org/synapse#8410))
- Remove unnecessary 3PID registration check when resetting password via an email address. Bug introduced in v0.34.0rc2. ([\#8414](matrix-org/synapse#8414))

Improved Documentation
----------------------

- Add `/_synapse/client` to the reverse proxy documentation. ([\#8227](matrix-org/synapse#8227))
- Add note to the reverse proxy settings documentation about disabling Apache's mod_security2. Contributed by Julian Fietkau (@jfietkau). ([\#8375](matrix-org/synapse#8375))
- Improve description of `server_name` config option in `homserver.yaml`. ([\#8415](matrix-org/synapse#8415))

Deprecations and Removals
-------------------------

- Drop support for `prometheus_client` older than 0.4.0. ([\#8426](matrix-org/synapse#8426))

Internal Changes
----------------

- Fix tests on distros which disable TLSv1.0. Contributed by @danc86. ([\#8208](matrix-org/synapse#8208))
- Simplify the distributor code to avoid unnecessary work. ([\#8216](matrix-org/synapse#8216))
- Remove the `populate_stats_process_rooms_2` background job and restore functionality to `populate_stats_process_rooms`. ([\#8243](matrix-org/synapse#8243))
- Clean up type hints for `PaginationConfig`. ([\#8250](matrix-org/synapse#8250), [\#8282](matrix-org/synapse#8282))
- Track the latest event for every destination and room for catch-up after federation outage. ([\#8256](matrix-org/synapse#8256))
- Fix non-user visible bug in implementation of `MultiWriterIdGenerator.get_current_token_for_writer`. ([\#8257](matrix-org/synapse#8257))
- Switch to the JSON implementation from the standard library. ([\#8259](matrix-org/synapse#8259))
- Add type hints to `synapse.util.async_helpers`. ([\#8260](matrix-org/synapse#8260))
- Simplify tests that mock asynchronous functions. ([\#8261](matrix-org/synapse#8261))
- Add type hints to `StreamToken` and `RoomStreamToken` classes. ([\#8279](matrix-org/synapse#8279))
- Change `StreamToken.room_key` to be a `RoomStreamToken` instance. ([\#8281](matrix-org/synapse#8281))
- Refactor notifier code to correctly use the max event stream position. ([\#8288](matrix-org/synapse#8288))
- Use slotted classes where possible. ([\#8296](matrix-org/synapse#8296))
- Support testing the local Synapse checkout against the [Complement homeserver test suite](https://github.com/matrix-org/complement/). ([\#8317](matrix-org/synapse#8317))
- Update outdated usages of `metaclass` to python 3 syntax. ([\#8326](matrix-org/synapse#8326))
- Move lint-related dependencies to package-extra field, update CONTRIBUTING.md to utilise this. ([\#8330](matrix-org/synapse#8330), [\#8377](matrix-org/synapse#8377))
- Use the `admin_patterns` helper in additional locations. ([\#8331](matrix-org/synapse#8331))
- Fix test logging to allow braces in log output. ([\#8335](matrix-org/synapse#8335))
- Remove `__future__` imports related to Python 2 compatibility. ([\#8337](matrix-org/synapse#8337))
- Simplify `super()` calls to Python 3 syntax. ([\#8344](matrix-org/synapse#8344))
- Fix bad merge from `release-v1.20.0` branch to `develop`. ([\#8354](matrix-org/synapse#8354))
- Factor out a `_send_dummy_event_for_room` method. ([\#8370](matrix-org/synapse#8370))
- Improve logging of state resolution. ([\#8371](matrix-org/synapse#8371))
- Add type annotations to `SimpleHttpClient`. ([\#8372](matrix-org/synapse#8372))
- Refactor ID generators to use `async with` syntax. ([\#8383](matrix-org/synapse#8383))
- Add `EventStreamPosition` type. ([\#8388](matrix-org/synapse#8388))
- Create a mechanism for marking tests "logcontext clean". ([\#8399](matrix-org/synapse#8399))
- A pair of tiny cleanups in the federation request code. ([\#8401](matrix-org/synapse#8401))
- Add checks on startup that PostgreSQL sequences are consistent with their associated tables. ([\#8402](matrix-org/synapse#8402))
- Do not include appservice users when calculating the total MAU for a server. ([\#8404](matrix-org/synapse#8404))
- Typing fixes for `synapse.handlers.federation`. ([\#8422](matrix-org/synapse#8422))
- Various refactors to simplify stream token handling. ([\#8423](matrix-org/synapse#8423))
- Make stream token serializing/deserializing async. ([\#8427](matrix-org/synapse#8427))

Author: richvdh

CHANGES.md

@@ -48,7 +48,7 @@
 except ImportError:
     pass
 
-__version__ = "1.20.1"
+__version__ = "1.21.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when