TODO: commit message

Merge branch 'gwmngilfen-next-patch-21674' into 'next'
Add two flavours of change for the mautrix-signal patch See merge request famedly/conduit!197
2021-10-10 13:51:57 +02:00 · 2021-09-15 20:31:21 +00:00 · 2021-09-15 21:16:25 +01:00 · 2021-09-15 21:16:25 +01:00 · 2021-09-15 07:05:33 +00:00 · 2021-09-14 20:44:11 +00:00
36 changed files with 1131 additions and 1202 deletions
@@ -3,7 +3,6 @@ stages:
  - build docker image
  - test
  - upload artifacts
-  - publish

 variables:
  GIT_SUBMODULE_STRATEGY: recursive
@@ -76,12 +75,12 @@ build:release:cargo:aarch64-unknown-linux-gnu:
  extends: .build-cargo-shared-settings
  variables:
    TARGET: "aarch64-unknown-linux-gnu"
-    NEEDED_PACKAGES: "build-essential gcc-8-aarch64-linux-gnu g++-aarch64-linux-gnu libc6-dev-arm64-cross"
+    NEEDED_PACKAGES: "build-essential gcc-10-aarch64-linux-gnu g++-aarch64-linux-gnu libc6-dev-arm64-cross"
    CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER: aarch64-linux-gnu-gcc
    CC_aarch64_unknown_linux_gnu: aarch64-linux-gnu-gcc
    CXX_aarch64_unknown_linux_gnu: aarch64-linux-gnu-g++
-    TARGET_CC: "/usr/bin/aarch64-linux-gnu-gcc-8"
-    TARGET_AR: "/usr/bin/aarch64-linux-gnu-gcc-ar-8"
+    TARGET_CC: "/usr/bin/aarch64-linux-gnu-gcc-10"
+    TARGET_AR: "/usr/bin/aarch64-linux-gnu-gcc-ar-10"
  artifacts:
    name: "conduit-aarch64-unknown-linux-gnu"
    paths:
@@ -223,6 +222,7 @@ build:docker:next:
    - >
      /kaniko/executor
      $KANIKO_CACHE_ARGS
+      --force
      --context $CI_PROJECT_DIR
      --build-arg CREATED=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
      --build-arg VERSION=$(grep -m1 -o '[0-9].[0-9].[0-9]' Cargo.toml)
@@ -370,17 +370,3 @@ publish:package:
    - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file conduit-x86_64-unknown-linux-gnu.deb "${BASE_URL}/conduit-x86_64-unknown-linux-gnu.deb"'


-pages:
-  stage: "publish"
-  image:
-    name: "squidfunk/mkdocs-material"
-    entrypoint: [""]
-  tags: ["docker"]
-  needs: []
-  rules:
-    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-  script:
-    - "mkdocs build --site-dir public"
-  artifacts:
-    paths:
-      - "public"
@@ -0,0 +1,86 @@
+# Setting up Appservices
+
+## Getting help
+
+If you run into any problems while setting up an Appservice, write an email to `timo@koesters.xyz`, ask us in `#conduit:matrix.org` or [open an issue on GitLab](https://gitlab.com/famedly/conduit/-/issues/new).
+
+## Set up the appservice - general instructions
+
+Follow whatever instructions are given by the appservice. This usually includes
+downloading, changing its config (setting domain, homeserver url, port etc.)
+and later starting it.
+
+At some point the appservice guide should ask you to add a registration yaml
+file to the homeserver. In Synapse you would do this by adding the path to the
+homeserver.yaml, but in Conduit you can do this from within Matrix:
+
+First, go into the #admins room of your homeserver. The first person that
+registered on the homeserver automatically joins it. Then send a message into
+the room like this:
+
+    @conduit:your.server.name: register_appservice
+    ```
+    paste
+    the
+    contents
+    of
+    the
+    yaml
+    registration
+    here
+    ```
+
+You can confirm it worked by sending a message like this:
+`@conduit:your.server.name: list_appservices`
+
+The @conduit bot should answer with `Appservices (1): your-bridge`
+
+Then you are done. Conduit will send messages to the appservices and the
+appservice can send requests to the homeserver. You don't need to restart
+Conduit, but if it doesn't work, restarting while the appservice is running
+could help.
+
+## Appservice-specific instructions
+
+### Tested appservices
+
+These appservices have been tested and work with Conduit without any extra steps:
+
+- [matrix-appservice-discord](https://github.com/Half-Shot/matrix-appservice-discord)
+- [mautrix-hangouts](https://github.com/mautrix/hangouts/)
+- [mautrix-telegram](https://github.com/mautrix/telegram/)
+
+### [mautrix-signal](https://github.com/mautrix/signal)
+
+There are a few things you need to do, in order for the Signal bridge (at least
+up to version `0.2.0`) to work. How you do this depends on whether you use
+Docker or `virtualenv` to run it. In either case you need to modify
+[portal.py](https://github.com/mautrix/signal/blob/master/mautrix_signal/portal.py).
+Do this **before** following the bridge installation guide.
+
+1. **Create a copy of `portal.py`**. Go to
+   [portal.py](https://github.com/mautrix/signal/blob/master/mautrix_signal/portal.py)
+at [mautrix-signal](https://github.com/mautrix/signal) (make sure you change to
+the correct commit/version of mautrix-signal you're using) and copy its
+content. Create a new `portal.py` on your system and paste the content in.
+2. **Patch the copy**. Exact line numbers may be slightly different, look nearby if they don't match: 
+  - [Line 1020](https://github.com/mautrix/signal/blob/4ea831536f154aba6419d13292479eb383ea3308/mautrix_signal/portal.py#L1020)
+    ```diff
+    --- levels.users[self.main_intent.mxid] = 9001 if is_initial else 100
+    +++ levels.users[self.main_intent.mxid] = 100 if is_initial else 100
+    ```
+  - [Between lines 1041 and 1042](https://github.com/mautrix/signal/blob/4ea831536f154aba6419d13292479eb383ea3308/mautrix_signal/portal.py#L1041-L1042) add a new line:
+    ```diff
+        "type": str(EventType.ROOM_POWER_LEVELS),
+    +++ "state_key": "",
+        "content": power_levels.serialize(),
+    ```
+3. **Deploy the patch**. This is different depending on how you have `mautrix-signal` deployed:
+  - [*If using virtualenv*] Copy your patched `portal.py` to `./lib/python3.7/site-packages/mautrix_signal/portal.py` (the exact version of Python may be different on your system).
+  -  [*If using Docker*] Map the patched `portal.py` into the `mautrix-signal` container:
+    
+    ```yaml
+    volumes:
+      - ./your/path/on/host/portal.py:/usr/lib/python3.9/site-packages/mautrix_signal/portal.py
+    ```
+4. Now continue with the [bridge installation instructions ](https://docs.mau.fi/bridges/index.html) and the general bridge notes above.
@@ -0,0 +1,11 @@
+Install docker:
+
+```
+$ sudo apt install docker
+$ sudo usermod -aG docker $USER
+$ exec sudo su -l $USER
+$ sudo systemctl start docker
+$ cargo install cross
+$ cross build --release --target armv7-unknown-linux-musleabihf
+```
+The cross-compiled binary is at target/armv7-unknown-linux-musleabihf/release/conduit
@@ -254,6 +254,7 @@ dependencies = [
 "opentelemetry-jaeger",
 "parking_lot",
 "rand 0.8.4",
+ "rayon",
 "regex",
 "reqwest",
 "ring",
@@ -1747,6 +1748,31 @@ dependencies = [
 "rand_core 0.6.3",
 ]

+[[package]]
+name = "rayon"
+version = "1.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c06aca804d41dbc8ba42dfd964f0d01334eceb64314b9ecf7c5fad5188a06d90"
+dependencies = [
+ "autocfg",
+ "crossbeam-deque",
+ "either",
+ "rayon-core",
+]
+
+[[package]]
+name = "rayon-core"
+version = "1.9.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d78120e2c850279833f1dd3582f730c4ab53ed95aeaaaa862a2a5c71b1656d8e"
+dependencies = [
+ "crossbeam-channel",
+ "crossbeam-deque",
+ "crossbeam-utils 0.8.5",
+ "lazy_static",
+ "num_cpus",
+]
+
 [[package]]
 name = "redox_syscall"
 version = "0.2.10"
@@ -1968,7 +1994,7 @@ dependencies = [
 [[package]]
 name = "ruma"
 version = "0.4.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "assign",
 "js_int",
@@ -1989,7 +2015,7 @@ dependencies = [
 [[package]]
 name = "ruma-api"
 version = "0.18.3"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "bytes",
 "http",
@@ -2005,7 +2031,7 @@ dependencies = [
 [[package]]
 name = "ruma-api-macros"
 version = "0.18.3"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "proc-macro-crate",
 "proc-macro2",
@@ -2016,7 +2042,7 @@ dependencies = [
 [[package]]
 name = "ruma-appservice-api"
 version = "0.4.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "ruma-api",
 "ruma-common",
@@ -2030,7 +2056,7 @@ dependencies = [
 [[package]]
 name = "ruma-client-api"
 version = "0.12.2"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "assign",
 "bytes",
@@ -2050,7 +2076,7 @@ dependencies = [
 [[package]]
 name = "ruma-common"
 version = "0.6.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "indexmap",
 "js_int",
@@ -2065,7 +2091,7 @@ dependencies = [
 [[package]]
 name = "ruma-events"
 version = "0.24.5"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "indoc",
 "js_int",
@@ -2081,7 +2107,7 @@ dependencies = [
 [[package]]
 name = "ruma-events-macros"
 version = "0.24.5"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "proc-macro-crate",
 "proc-macro2",
@@ -2092,7 +2118,7 @@ dependencies = [
 [[package]]
 name = "ruma-federation-api"
 version = "0.3.1"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "js_int",
 "ruma-api",
@@ -2107,7 +2133,7 @@ dependencies = [
 [[package]]
 name = "ruma-identifiers"
 version = "0.20.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "paste",
 "rand 0.8.4",
@@ -2121,7 +2147,7 @@ dependencies = [
 [[package]]
 name = "ruma-identifiers-macros"
 version = "0.20.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "quote",
 "ruma-identifiers-validation",
@@ -2131,12 +2157,12 @@ dependencies = [
 [[package]]
 name = "ruma-identifiers-validation"
 version = "0.5.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"

 [[package]]
 name = "ruma-identity-service-api"
 version = "0.3.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "js_int",
 "ruma-api",
@@ -2149,7 +2175,7 @@ dependencies = [
 [[package]]
 name = "ruma-push-gateway-api"
 version = "0.3.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "js_int",
 "ruma-api",
@@ -2164,7 +2190,7 @@ dependencies = [
 [[package]]
 name = "ruma-serde"
 version = "0.5.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "bytes",
 "form_urlencoded",
@@ -2178,7 +2204,7 @@ dependencies = [
 [[package]]
 name = "ruma-serde-macros"
 version = "0.5.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "proc-macro-crate",
 "proc-macro2",
@@ -2189,7 +2215,7 @@ dependencies = [
 [[package]]
 name = "ruma-signatures"
 version = "0.9.0"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "base64 0.13.0",
 "ed25519-dalek",
@@ -2206,7 +2232,7 @@ dependencies = [
 [[package]]
 name = "ruma-state-res"
 version = "0.4.1"
-source = "git+https://github.com/ruma/ruma?rev=a6a1224652912a957b09f136ec5da2686be6e0e2#a6a1224652912a957b09f136ec5da2686be6e0e2"
+source = "git+https://github.com/timokoesters/ruma?rev=6e9823de284967a41100db0e3134319f4b6a0cfa#6e9823de284967a41100db0e3134319f4b6a0cfa"
 dependencies = [
 "itertools 0.10.1",
 "js_int",
@@ -19,8 +19,8 @@ rocket = { version = "0.5.0-rc.1", features = ["tls"] } # Used to handle request

 # Used for matrix spec type definitions and helpers
 #ruma = { version = "0.4.0", features = ["compat", "rand", "appservice-api-c", "client-api", "federation-api", "push-gateway-api-c", "state-res", "unstable-pre-spec", "unstable-exhaustive-types"] }
-ruma = { git = "https://github.com/ruma/ruma", rev = "a6a1224652912a957b09f136ec5da2686be6e0e2", features = ["compat", "rand", "appservice-api-c", "client-api", "federation-api", "push-gateway-api-c", "state-res", "unstable-pre-spec", "unstable-exhaustive-types"] }
-#ruma = { git = "https://github.com/timokoesters/ruma", rev = "50c1db7e0a3a21fc794b0cce3b64285a4c750c71", features = ["compat", "rand", "appservice-api-c", "client-api", "federation-api", "push-gateway-api-c", "state-res", "unstable-pre-spec", "unstable-exhaustive-types"] }
+#ruma = { git = "https://github.com/ruma/ruma", rev = "a6a1224652912a957b09f136ec5da2686be6e0e2", features = ["compat", "rand", "appservice-api-c", "client-api", "federation-api", "push-gateway-api-c", "state-res", "unstable-pre-spec", "unstable-exhaustive-types"] }
+ruma = { git = "https://github.com/timokoesters/ruma", rev = "6e9823de284967a41100db0e3134319f4b6a0cfa", features = ["compat", "rand", "appservice-api-c", "client-api", "federation-api", "push-gateway-api-c", "state-res", "unstable-pre-spec", "unstable-exhaustive-types"] }
 #ruma = { path = "../ruma/crates/ruma", features = ["compat", "rand", "appservice-api-c", "client-api", "federation-api", "push-gateway-api-c", "state-res", "unstable-pre-spec", "unstable-exhaustive-types"] }

 # Used for long polling and federation sender, should be the same as rocket::tokio
@@ -79,6 +79,7 @@ num_cpus = "1.13.0"
 threadpool = "1.8.1"
 heed = { git = "https://github.com/timokoesters/heed.git", rev = "f6f825da7fb2c758867e05ad973ef800a6fe1d5d", optional = true }
 thread_local = "1.1.3"
+rayon = "1.5.1"

 [features]
 default = ["conduit_bin", "backend_sqlite"]
@@ -1,16 +0,0 @@
-[book]
-title = "Conduit Docs"
-author = "The Conduit contributors"
-description = "Conduit is a simple, fast and reliable chat server for the Matrix protocol"
-language = "en"
-src = "docs"
-
-[rust]
-edition = "2018"
-
-[build]
-build-dir = "public"
-create-missing = true
-
-[output.html.search]
-limit-results = 15
@@ -1,16 +0,0 @@
-# Summary
-
-[Home](index.md)
-
- [Installation](installation.md)
-    - [Prerequisites](installation/prerequisites.md)
-    - [From Binaries](installation/manual.md)
-    - [From Packages](installation/packages.md)
-    - [With Docker](installation/docker.md)
- [Configuration](configuration.md)
-    - [Config options](configuration/configuration.md)
-    - [Appservices](configuration/appservices.md)
- [Contribute](contribute.md)
-    - [Basics](development/basics.md)
-    - [Cross compilation](development/cross-compilation.md)
-    - [Tests & CI](development/tests-ci.md)
@@ -1 +0,0 @@
-# Configuration
@@ -1,2 +0,0 @@
-# Configuring Conduit
-
@@ -1,220 +0,0 @@
-# Configuring Conduit
-
-Conduit can be configured via a config file (conventionally called Conduit.toml) or environment variables. If a config
-file exists and environment variables are set, environment variables overwrite config options.
-
-You absolutely need to set the environment variable `CONDUIT_CONFIG_FILE` to either point to a config file (
-e.g. `CONDUIT_CONFIG_FILE=/etc/conduit/Conduit.toml`) or to an empty string (`CONDUIT_CONFIG_FILE=''`) if you want to
-configure Conduit with just environment variables.
-
-## Mandatory config options
-
-Mandatory variables must be configured in order for Conduit to run properly.
-
-### Server Name
-
- Config file key: `server_name`
- Envirnoment variable: `CONDUIT_SERVER_NAME`
- Default value: _None, you will need to choose your own._
-
-The server_name is the name of this server. It is used as a suffix for user and room ids. Example: If you set it
-to `conduit.rs`, your usernames will look like `@somebody:conduit.rs`.
-
-The Conduit server needs to be reachable at https://your.server.name/ on port 443 (client-server) and 8448 (
-server-server) OR you can create /.well-known files to redirect requests. See
-the [Client-Server specs](https://matrix.org/docs/spec/client_server/latest#get-well-known-matrix-client) and
-the [Server-Server specs](https://matrix.org/docs/spec/server_server/r0.1.4#get-well-known-matrix-server) for more
-information.
-
-### Database Path
-
- Config file key: `database_path`
- Envirnoment variable: `CONDUIT_DATABASE_PATH`
- Default value: _None, but many people like to use `/var/lib/conduit/`_.
-
-A **directory** where Conduit stores its database and media files. This directory must exist, have enough free space and
-be readable and writable by the user Conduit is running as.
-
-What does _enough free space_ mean? It heavily on the amount of messages your Conduit server will see and the amount and
-size of media files users on your Conduit server send. As a rule of thumb, you should have at least 10 GB of free space
-left. You should be comfortable for quite some time with 50 GB.
-
-### TCP Port
-
- Config file key: `port`
- Environment variable: `CONDUIT_PORT`
- Default value: _None, but many people like to use `6167`_.
-
-The TCP port Conduit will listen on for connections. The port needs to be free (no other program is listeing on it).
-
-Conduit does currently (2021-09) not offer HTTPS by itself. Only unencrypted HTTP requests will be accepted on this
-port. Unless you know what you are doing, this port should not be exposed to the internet. Instead, use a reverse proxy
-capable of doing TLS to offer your Conduit server to the internet via HTTPS. See [TODO] for example configurations.
-
-## Optional configuration options
-
-These config options come with defaults and don't need to be configured for Conduit to run. That said, you should still
-check them to make sure that your Conduit server behaves like you want it to do.
-
-### Maximum request size
-
- Config file key: `max_request_size`
- Environment variable: `CONDUIT_MAX_REQUEST_SIZE`
- Default value: `20_000_000` (~= 20 MB)
-
-The maximum size in bytes for incoming requests to Conduit. You can use underscores to improve readability.
-
-This will effectively limit the size for images, videos and other files users on your Conduit server can send.
-
-### Allow Registration?
-
- Config file key: `allow_registration`
- Environment variable: `CONDUIT_ALLOW_REGISTRATION`
- Default value: `true`
- Possible values: `true`, `false`
-
-It this is set to `false`, no new users can register accounts on your Conduit server. Already registered users will not
-be affected from this setting and can continue to user your server.
-
-The first user to ever register on your Conduit server will be considered the admin account and is automatically invited
-into the admin room.
-
-### Allow Encryption?
-
- Config file key: `allow_encryption`
- Environment variable: `CONDUIT_ALLOW_ENCRYPTION`
- Default value: `true`
- Possible values: `true`, `false`
-
-If this is set to `false`, Conduit disables the ability for users to create encrypted chats. Existing encrypted chats
-may continue to work.
-
-### Allow federation?
-
- Config file key: `allow_federation`
- Environment variable: `CONDUIT_ALLOW_FEDERATION`
- Default value: `false`
- Possible values: `true`, `false`
-
-Federation means that users from different Matrix servers can chat with each other. E.g. `@mathew:matrix.org` can chat
-with `@timo:conduit.rs`.
-
-If this option is set to `false`, users on your Conduit server can only talk with other users on your Conduit server.
-
-Federation with other servers needs to happen over HTTPS, so make sure you have set up a reverse proxy.
-
-### Jaeger Tracing
-
- Config file key: `allow_jaeger`
- Environment variable: `CONDUIT_ALLOW_JAEGER`
- Default value: `false`
- Possible values: `true`, `false`
-
-Enable Jaeger to support monitoring and troubleshooting through Jaeger.
-
-If you don't know what Jaeger is, you can safely leave this set to `false`.
-
-### Trusted servers
-
- Config file key: `trusted_servers`
- Environment variable: `CONDUIT_TRUSTED_SERVERS`
- Default value: `[]`
- Possible values: JSON-Array of server domains, e.g. `["matrix.org"]` or `["matrix.org", "conduit.rs"]`.
-
-Matrix servers have so-called "server keys", which authenticate messages from their users. Because your Conduit server
-might not know the server keys from every server it encounters, it can ask a _trusted server_ for them. This speeds
-things up for rooms with people from a lot of different servers.
-
-You should only set this to include trustworthy servers. Most people consider a good default to be `["matrix.org"]`.
-
-Only relevant if you have federation enabled.
-
-### Limit amount of concurrent requests
-
- Config file key: `max_concurrent_requests`
- Environment variable: `CONDUIT_MAX_CONCURRENT_REQUESTS`
- Default value: `100`
- Suggested values: `1` - `1000` (u16)
-
-How many requests Conduit can make at the same time. This affects federation with other Matrix servers, push
-notifications and app_services.
-
-// TODO Timo: When does it make sense to change this?
-
-### Configure logging
-
- Config file key: `log`
- Environment variable: `CONDUIT_LOG`
- Default value: `info,state_res=warn,rocket=off,_=off,sled=off`
-
-Configures which kind of messages Conduit logs.
-
-> // TODO: Better and more thorough explanation
-
-### Worker threads
-
- Config file key: `workers`
- Environment variable: `CONDUIT_WORKERS`
- Default value: cpu core count * 2
- Possible values: // TODO
-
-> // TODO: Which thing exactly threads? What not?
-
-### Listening address
-
- Config file key: `address`
- Environment variable: `CONDUIT_ADDRESS`
- Default value: `127.0.0.1`
- Possible values: Valid IP addresses.
-
-Which IP address conduit is listening on. 127.0.0.1 means that Conduit can only be accessed from the same server or
-through a reverse proxy on that server. If you want it to be accessible from any network interface (which you should
-not, because other matrix servers should talk to your Conduit via a reverse proxy and not directly), you can set it
-to `0.0.0.0`.
-
-### Database cache capacity
-
- Config file key: `db_cache_capacity_mb`
- Environment variable: `CONDUIT_DB_CACHE_CAPACITY_MB`
- Default value: `200`
- Possible values: `true`, `false`
-
-The total amount of memory (RAM) that the database cache will be able to use.
-
-> // TODO: this needs clearification: In RAM or on disk and for what exactly?
-
-### PDU cache capacity
-
- Config file key: `pdu_cache_capacity`
- Environment variable: `CONDUIT_PDU_CACHE_CAPACITY`
- Default value: `100_000`
- Suggested values: `1_000` - `1_000_000` (u32)
-
-The total capacity (read: number of items) the pdu cache can hold in memory. Setting this to a lower number may slow
-Conduit down, as it must fetch more data from the database. Increasing it will mean that Conduit will start to use more
-memory as the cache slowly fills up.
-
-### SQLite WAL clean interval
-
- Config file key: `sqlite_wal_clean_second_interval`
- Environment variable: `CONDUIT_SQLITE_WAL_CLEAN_SECOND_INTERVAL`
- Default value: `60` (every 60 seconds)
- Suggested values: `1` - `3600` (u32)
-
-How often the WAL file should be cleaned up. The WAL file will be written to until cleaned up, after which it restarts
-writing from the beginning.
-
-The file's size will correspond to how long it could write to it in one go. (e.g. if conduit writes 100MB of data to the
-database inbetween that period, the file will grow to 100MB). You can read more about that in
-the [SQLite Docs](https://www.sqlite.org/draft/wal.html).
-
-Reducing this down too much can offset the benefits of using a WAL at all. However, having this too high can result in a
-large WAL file.
-
-Only relevant when using SQLite as the database.
-
-### Still undocumented config options
-
- `tracing_flame`
- `proxy`
- `jwt_secret`
@@ -1 +0,0 @@
-# Contribute
@@ -1 +0,0 @@
-# Basics
@@ -1,13 +0,0 @@
-# Cross compilation
-
-Install docker:
-
-```bash
-sudo apt install docker
-sudo usermod -aG docker $USER
-exec sudo su -l $USER
-sudo systemctl start docker
-cargo install cross
-cross build --release --target armv7-unknown-linux-musleabihf
-```
-The cross-compiled binary is at target/armv7-unknown-linux-musleabihf/release/conduit
@@ -1 +0,0 @@
-# Tests & CI
@@ -1,52 +0,0 @@
-# Conduit
-
-Conduit is a simple, fast and reliable chat server for the [Matrix] protocol written in [Rust].
-
-----
-> Note: This project is work-in-progress. Do *not* rely on it yet.
-
-## What is Matrix?
-
-[Matrix] is an open network for secure and decentralized
-communication. It allows you to chat with friends even if they are using
-another servers and client. You can even use bridges to communicate with users
-outside of Matrix, like a community on Discord or your family on Hangouts.
-
-## Why Conduit?
-
-Conduit is an open-source server implementation of the [Matrix
-Specification] with a focus on easy setup and low
-system requirements, making it very easy to set up.
-
-Other server implementations try to be extremely scalable, which makes sense if
-the goal is to support millions of users on a single instance, but makes
-smaller deployments a lot more inefficient. Conduit tries to keep it simple but
-takes full advantage of that, for example by using an in-memory database for
-[huge performance gains](https://github.com/timokoesters/romeo-and-juliet-benchmark).
-
-The future for Conduit in peer-to-peer Matrix (every client contains a server)
-is also bright.
-
-Conduit tries to be reliable by using the Rust programming language and paying
-close attention to error handling to make sure that evil clients, misbehaving
-servers or even a partially broken database will not cause the whole server to
-stop working.
-
-## Chat with us!
-
-We have a room on Matrix: [#conduit:matrix.org](https://matrix.to/#/#conduit:matrix.org)
-
-You can also contact us using:
- Matrix: [@timo:koesters.xyz](https://matrix.to/#/@timo:koesters.xyz)
- Email: [conduit@koesters.xyz](mailto:conduit@koesters.xyz)
-
-
-## Donate
-
-Liberapay: <https://liberapay.com/timokoesters/>\
-Bitcoin: `bc1qnnykf986tw49ur7wx9rpw2tevpsztvar5x8w4n`
-
-
-[Matrix]: https://matrix.org/
-[Rust]: https://rust-lang.org
-[Matrix Specification]: https://matrix.org/docs/spec
@@ -1 +0,0 @@
-# Installation
@@ -1,2 +0,0 @@
-:::warning Test
-:::
@@ -1,12 +0,0 @@
-# Distribution packages
-
-## Debian / Ubuntu
-
-[@paul:luon.net](https://matrix.to/#/@paul:luon.net) plans to package Conduit for Debian as soon as it reaches 1.0.
-Until it is available in the official repos, you can install the development version of it manually:
-
-```bash
-sudo apt-get install ca-certificates
-wget --https-only -O /tmp/conduit.deb https://gitlab.com/famedly/conduit/-/jobs/artifacts/master/raw/conduit-x86_64-unknown-linux-gnu.deb?job=build:cargo-deb:x86_64-unknown-linux-gnu
-sudo dpkg -i /tmp/conduit.deb
-```
@@ -1,31 +0,0 @@
-# Prerequisites for running Conduit
-
-You'll need:
-
- A domain. Commonly cost about $10/year.
- A Linux server with a stable internet connection, at least 500 MB of RAM and some disk space for messages and
-  attachments. Commonly start at $5/month.
- Some basic knowledge about using a shell, SSH and configuring and protecting a server.
-  
-  
-## A word of caution:
-
-Don't underestimate the toll of administrating your own server.
-Conduit can't protect your conversations if your server gets compromised or deleted.
-
-Make sure that you got:
-
- Automatic security updates
-    - On Ubuntu/Debian: Set up unattended-upgrades
-    - On RHEL/CentOS: Have a look at yum-cron
- A firewall blocking all but the needed incoming ports
-    - ufw is an easy interface for the linux firewall
- Protection against automatic attacks
-    - fail2ban scans logs and bans IPs which try to brute force their way into your server.
-    - Disable ssh login for root and switch from password to key based authentication.
- Automated backups
-    - Most VPS hosting companies offer whole server backups for a small fee.
-    - Or run your own backup with something like borg.
- A way to get notified if your disk fills up.
-    - If you send too much cat videos to your friends, Conduit might at some point become unable to
-      store any important messages.
@@ -1,40 +0,0 @@
-# About Matrix Homeservers
-
-Matrix homeservers manage its users chats. Every Matrix username includes the homserver it belongs to:
-`@alice:matrix.org` means that the `matrix.org` homeserver hosts a user called `@alice`.
-Every time someone chats with Alice, the `matrix.org` homeserver stores these messages.
-When `@alice:matrix.org` talks with `@adelaide:matrix.org`, that's easy. Both users use the same server.
-
-But how can `@bob:vector.tld`, who uses the `vector.tld` homeserver, exchange messages with `@alice:matrix.org`?
-This is where it get's a bit more complicated.
-
-## Matrix Homeserver discovery
-
-The Matrix specification specifies multiple ways how servers can discover and then talk to each other.
-Let's look at the most common one:
-
-### .well-known files
-
-At first, the only information a server has about a user (e.g. `@bob:vector.tld`) is its homeserver name: `vector.tld`.
-It then makes a HTTP GET request to `https://vector.tld/.well-known/matrix/server`.
-In the ideal case, this file contains a content like this: 
-
-```json
-{
-  "m.server": "matrix.vector.tld:443"
-}
-```
-
-This translates to: The matrix homeserver software for users with a username ending on `vector.tld`
-can be found at the address `matrix.vector.tld` at port 443 (which is the common port for HTTPS).
-
-The homeserver on it's quest to find `@bob:vector.tld` now contacts `matrix.vector.tld:443` and is then
-able to exchange chat messages with it.
-
-
-### Why so complicated?
-
-Organizations often don't want to run their Matrix server on the same machine that hosts their website,
-but `@foo:matrix.evil.corp` usernames are ugly and everyone wants to be `@foo:evil.corp`.
-
-To solve that problem, Matrix implements this extra step via a .well-known file or a DNS entry.
@@ -1,20 +0,0 @@
-site_name: Conduit Docs
-site_description: Conduit is a simple, fast and reliable chat server for the Matrix protocol
-theme:
-  name: material
-repo_url: https://gitlab.com/famedly/conduit
-nav:
-  - Home: index.md
-  - Installation:
-      - Prerequisites: installation/prerequisites.md
-      - Distribution Packages: installation/packages.md
-      - Manual: installation/manual.md
-      - Docker: installation/docker.md
-  - Configuration:
-      - Conduit.toml: configuration/conduit.toml.md
-      - Appservices: configuration/appservices.md
-  - Development:
-      - Basics: development/basics.md
-      - Cross compilation: development/cross-compilation.md
-      - Tests & CI: development/tests-ci.md
-
@@ -28,6 +28,7 @@ use ruma::{
    identifiers::RoomName,
    push, RoomAliasId, RoomId, RoomVersionId, UserId,
 };
+use serde_json::value::RawValue;
 use tracing::info;

 use register::RegistrationKind;
@@ -279,7 +280,10 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomCreate,
-                content: serde_json::to_value(content).expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&content).expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -294,16 +298,19 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomMember,
-                content: serde_json::to_value(member::MemberEventContent {
-                    membership: member::MembershipState::Join,
-                    displayname: None,
-                    avatar_url: None,
-                    is_direct: None,
-                    third_party_invite: None,
-                    blurhash: None,
-                    reason: None,
-                })
-                .expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&member::MemberEventContent {
+                        membership: member::MembershipState::Join,
+                        displayname: None,
+                        avatar_url: None,
+                        is_direct: None,
+                        third_party_invite: None,
+                        blurhash: None,
+                        reason: None,
+                    })
+                    .expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some(conduit_user.to_string()),
                redacts: None,
@@ -322,13 +329,16 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomPowerLevels,
-                content: serde_json::to_value(
-                    ruma::events::room::power_levels::PowerLevelsEventContent {
-                        users,
-                        ..Default::default()
-                    },
+                content: RawValue::from_string(
+                    serde_json::to_string(
+                        &ruma::events::room::power_levels::PowerLevelsEventContent {
+                            users,
+                            ..Default::default()
+                        },
+                    )
+                    .expect("event is valid, we just created it"),
                )
-                .expect("event is valid, we just created it"),
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -343,10 +353,13 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomJoinRules,
-                content: serde_json::to_value(join_rules::JoinRulesEventContent::new(
-                    join_rules::JoinRule::Invite,
-                ))
-                .expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&join_rules::JoinRulesEventContent::new(
+                        join_rules::JoinRule::Invite,
+                    ))
+                    .expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -361,12 +374,13 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomHistoryVisibility,
-                content: serde_json::to_value(
-                    history_visibility::HistoryVisibilityEventContent::new(
+                content: RawValue::from_string(
+                    serde_json::to_string(&history_visibility::HistoryVisibilityEventContent::new(
                        history_visibility::HistoryVisibility::Shared,
-                    ),
+                    ))
+                    .expect("event is valid, we just created it"),
                )
-                .expect("event is valid, we just created it"),
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -381,10 +395,13 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomGuestAccess,
-                content: serde_json::to_value(guest_access::GuestAccessEventContent::new(
-                    guest_access::GuestAccess::Forbidden,
-                ))
-                .expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&guest_access::GuestAccessEventContent::new(
+                        guest_access::GuestAccess::Forbidden,
+                    ))
+                    .expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -402,8 +419,11 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomName,
-                content: serde_json::to_value(name::NameEventContent::new(Some(room_name)))
-                    .expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&name::NameEventContent::new(Some(room_name)))
+                        .expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -417,10 +437,13 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomTopic,
-                content: serde_json::to_value(topic::TopicEventContent {
-                    topic: format!("Manage {}", db.globals.server_name()),
-                })
-                .expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&topic::TopicEventContent {
+                        topic: format!("Manage {}", db.globals.server_name()),
+                    })
+                    .expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -439,11 +462,14 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomCanonicalAlias,
-                content: serde_json::to_value(canonical_alias::CanonicalAliasEventContent {
-                    alias: Some(alias.clone()),
-                    alt_aliases: Vec::new(),
-                })
-                .expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&canonical_alias::CanonicalAliasEventContent {
+                        alias: Some(alias.clone()),
+                        alt_aliases: Vec::new(),
+                    })
+                    .expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -460,16 +486,19 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomMember,
-                content: serde_json::to_value(member::MemberEventContent {
-                    membership: member::MembershipState::Invite,
-                    displayname: None,
-                    avatar_url: None,
-                    is_direct: None,
-                    third_party_invite: None,
-                    blurhash: None,
-                    reason: None,
-                })
-                .expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&member::MemberEventContent {
+                        membership: member::MembershipState::Invite,
+                        displayname: None,
+                        avatar_url: None,
+                        is_direct: None,
+                        third_party_invite: None,
+                        blurhash: None,
+                        reason: None,
+                    })
+                    .expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some(user_id.to_string()),
                redacts: None,
@@ -482,16 +511,19 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomMember,
-                content: serde_json::to_value(member::MemberEventContent {
-                    membership: member::MembershipState::Join,
-                    displayname: Some(displayname),
-                    avatar_url: None,
-                    is_direct: None,
-                    third_party_invite: None,
-                    blurhash: None,
-                    reason: None,
-                })
-                .expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&member::MemberEventContent {
+                        membership: member::MembershipState::Join,
+                        displayname: Some(displayname),
+                        avatar_url: None,
+                        is_direct: None,
+                        third_party_invite: None,
+                        blurhash: None,
+                        reason: None,
+                    })
+                    .expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some(user_id.to_string()),
                redacts: None,
@@ -506,11 +538,11 @@ pub async fn register_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomMessage,
-                content: serde_json::to_value(message::MessageEventContent::text_html(
+                content: RawValue::from_string(serde_json::to_string(&message::MessageEventContent::text_html(
                        "## Thank you for trying out Conduit!\n\nConduit is currently in Beta. This means you can join and participate in most Matrix rooms, but not all features are supported and you might run into bugs from time to time.\n\nHelpful links:\n> Website: https://conduit.rs\n> Git and Documentation: https://gitlab.com/famedly/conduit\n> Report issues: https://gitlab.com/famedly/conduit/-/issues\n\nHere are some rooms you can join (by typing the command):\n\nConduit room (Ask questions and get notified on updates):\n`/join #conduit:fachschaften.org`\n\nConduit lounge (Off-topic, only Conduit users are allowed to join)\n`/join #conduit-lounge:conduit.rs`".to_owned(),
                        "<h2>Thank you for trying out Conduit!</h2>\n<p>Conduit is currently in Beta. This means you can join and participate in most Matrix rooms, but not all features are supported and you might run into bugs from time to time.</p>\n<p>Helpful links:</p>\n<blockquote>\n<p>Website: https://conduit.rs<br>Git and Documentation: https://gitlab.com/famedly/conduit<br>Report issues: https://gitlab.com/famedly/conduit/-/issues</p>\n</blockquote>\n<p>Here are some rooms you can join (by typing the command):</p>\n<p>Conduit room (Ask questions and get notified on updates):<br><code>/join #conduit:fachschaften.org</code></p>\n<p>Conduit lounge (Off-topic, only Conduit users are allowed to join)<br><code>/join #conduit-lounge:conduit.rs</code></p>\n".to_owned(),
                ))
-                .expect("event is valid, we just created it"),
+                .expect("event is valid, we just created it")).expect("string is valid"),
                unsigned: None,
                state_key: None,
                redacts: None,
@@ -721,7 +753,10 @@ pub async fn deactivate_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomMember,
-                content: serde_json::to_value(event).expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&event).expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some(sender_user.to_string()),
                redacts: None,
@@ -20,7 +20,6 @@ use ruma::{
        room::{avatar, canonical_alias, guest_access, history_visibility, name, topic},
        EventType,
    },
-    serde::Raw,
    ServerName, UInt,
 };
 use tracing::{info, warn};
@@ -217,76 +216,68 @@ pub(crate) async fn get_public_rooms_filtered_helper(
        }
    }

-    let mut all_rooms =
-        db.rooms
-            .public_rooms()
-            .map(|room_id| {
-                let room_id = room_id?;
+    let mut all_rooms = db
+        .rooms
+        .public_rooms()
+        .map(|room_id| {
+            let room_id = room_id?;

-                let chunk = PublicRoomsChunk {
-                    aliases: Vec::new(),
-                    canonical_alias: db
-                        .rooms
-                        .room_state_get(&room_id, &EventType::RoomCanonicalAlias, "")?
-                        .map_or(Ok::<_, Error>(None), |s| {
-                            Ok(serde_json::from_value::<
-                                Raw<canonical_alias::CanonicalAliasEventContent>,
-                            >(s.content.clone())
-                            .expect("from_value::<Raw<..>> can never fail")
-                            .deserialize()
+            let chunk = PublicRoomsChunk {
+                aliases: Vec::new(),
+                canonical_alias: db
+                    .rooms
+                    .room_state_get(&room_id, &EventType::RoomCanonicalAlias, "")?
+                    .map_or(Ok::<_, Error>(None), |s| {
+                        Ok(
+                            serde_json::from_str::<canonical_alias::CanonicalAliasEventContent>(
+                                s.content.get(),
+                            )
                            .map_err(|_| {
                                Error::bad_database("Invalid canonical alias event in database.")
                            })?
-                            .alias)
-                        })?,
-                    name: db
-                        .rooms
-                        .room_state_get(&room_id, &EventType::RoomName, "")?
-                        .map_or(Ok::<_, Error>(None), |s| {
-                            Ok(serde_json::from_value::<Raw<name::NameEventContent>>(
-                                s.content.clone(),
-                            )
-                            .expect("from_value::<Raw<..>> can never fail")
-                            .deserialize()
-                            .map_err(|_| {
-                                Error::bad_database("Invalid room name event in database.")
-                            })?
-                            .name)
-                        })?,
-                    num_joined_members: db
-                        .rooms
-                        .room_joined_count(&room_id)?
-                        .unwrap_or_else(|| {
-                            warn!("Room {} has no member count", room_id);
-                            0
-                        })
-                        .try_into()
-                        .expect("user count should not be that big"),
-                    topic: db
-                        .rooms
-                        .room_state_get(&room_id, &EventType::RoomTopic, "")?
-                        .map_or(Ok::<_, Error>(None), |s| {
-                            Ok(Some(
-                                serde_json::from_value::<Raw<topic::TopicEventContent>>(
-                                    s.content.clone(),
-                                )
-                                .expect("from_value::<Raw<..>> can never fail")
-                                .deserialize()
+                            .alias,
+                        )
+                    })?,
+                name: db
+                    .rooms
+                    .room_state_get(&room_id, &EventType::RoomName, "")?
+                    .map_or(Ok::<_, Error>(None), |s| {
+                        Ok(
+                            serde_json::from_str::<name::NameEventContent>(s.content.get())
+                                .map_err(|_| {
+                                    Error::bad_database("Invalid room name event in database.")
+                                })?
+                                .name,
+                        )
+                    })?,
+                num_joined_members: db
+                    .rooms
+                    .room_joined_count(&room_id)?
+                    .unwrap_or_else(|| {
+                        warn!("Room {} has no member count", room_id);
+                        0
+                    })
+                    .try_into()
+                    .expect("user count should not be that big"),
+                topic: db
+                    .rooms
+                    .room_state_get(&room_id, &EventType::RoomTopic, "")?
+                    .map_or(Ok::<_, Error>(None), |s| {
+                        Ok(Some(
+                            serde_json::from_str::<topic::TopicEventContent>(s.content.get())
                                .map_err(|_| {
                                    Error::bad_database("Invalid room topic event in database.")
                                })?
                                .topic,
-                            ))
-                        })?,
-                    world_readable: db
-                        .rooms
-                        .room_state_get(&room_id, &EventType::RoomHistoryVisibility, "")?
-                        .map_or(Ok::<_, Error>(false), |s| {
-                            Ok(serde_json::from_value::<
-                                Raw<history_visibility::HistoryVisibilityEventContent>,
-                            >(s.content.clone())
-                            .expect("from_value::<Raw<..>> can never fail")
-                            .deserialize()
+                        ))
+                    })?,
+                world_readable: db
+                    .rooms
+                    .room_state_get(&room_id, &EventType::RoomHistoryVisibility, "")?
+                    .map_or(Ok::<_, Error>(false), |s| {
+                        Ok(serde_json::from_str::<
+                                history_visibility::HistoryVisibilityEventContent,
+                            >(s.content.get())
                            .map_err(|_| {
                                Error::bad_database(
                                    "Invalid room history visibility event in database.",
@@ -294,80 +285,74 @@ pub(crate) async fn get_public_rooms_filtered_helper(
                            })?
                            .history_visibility
                                == history_visibility::HistoryVisibility::WorldReadable)
-                        })?,
-                    guest_can_join: db
-                        .rooms
-                        .room_state_get(&room_id, &EventType::RoomGuestAccess, "")?
-                        .map_or(Ok::<_, Error>(false), |s| {
-                            Ok(
-                            serde_json::from_value::<Raw<guest_access::GuestAccessEventContent>>(
-                                s.content.clone(),
+                    })?,
+                guest_can_join: db
+                    .rooms
+                    .room_state_get(&room_id, &EventType::RoomGuestAccess, "")?
+                    .map_or(Ok::<_, Error>(false), |s| {
+                        Ok(
+                            serde_json::from_str::<guest_access::GuestAccessEventContent>(
+                                s.content.get(),
                            )
-                            .expect("from_value::<Raw<..>> can never fail")
-                            .deserialize()
                            .map_err(|_| {
                                Error::bad_database("Invalid room guest access event in database.")
                            })?
                            .guest_access
                                == guest_access::GuestAccess::CanJoin,
                        )
-                        })?,
-                    avatar_url: db
-                        .rooms
-                        .room_state_get(&room_id, &EventType::RoomAvatar, "")?
-                        .map(|s| {
-                            Ok::<_, Error>(
-                                serde_json::from_value::<Raw<avatar::AvatarEventContent>>(
-                                    s.content.clone(),
-                                )
-                                .expect("from_value::<Raw<..>> can never fail")
-                                .deserialize()
+                    })?,
+                avatar_url: db
+                    .rooms
+                    .room_state_get(&room_id, &EventType::RoomAvatar, "")?
+                    .map(|s| {
+                        Ok::<_, Error>(
+                            serde_json::from_str::<avatar::AvatarEventContent>(s.content.get())
                                .map_err(|_| {
                                    Error::bad_database("Invalid room avatar event in database.")
                                })?
                                .url,
-                            )
-                        })
-                        .transpose()?
-                        // url is now an Option<String> so we must flatten
-                        .flatten(),
-                    room_id,
-                };
-                Ok(chunk)
-            })
-            .filter_map(|r: Result<_>| r.ok()) // Filter out buggy rooms
-            .filter(|chunk| {
-                if let Some(query) = filter
-                    .generic_search_term
-                    .as_ref()
-                    .map(|q| q.to_lowercase())
-                {
-                    if let Some(name) = &chunk.name {
-                        if name.as_str().to_lowercase().contains(&query) {
-                            return true;
-                        }
+                        )
+                    })
+                    .transpose()?
+                    // url is now an Option<String> so we must flatten
+                    .flatten(),
+                room_id,
+            };
+            Ok(chunk)
+        })
+        .filter_map(|r: Result<_>| r.ok()) // Filter out buggy rooms
+        .filter(|chunk| {
+            if let Some(query) = filter
+                .generic_search_term
+                .as_ref()
+                .map(|q| q.to_lowercase())
+            {
+                if let Some(name) = &chunk.name {
+                    if name.as_str().to_lowercase().contains(&query) {
+                        return true;
                    }
-
-                    if let Some(topic) = &chunk.topic {
-                        if topic.to_lowercase().contains(&query) {
-                            return true;
-                        }
-                    }
-
-                    if let Some(canonical_alias) = &chunk.canonical_alias {
-                        if canonical_alias.as_str().to_lowercase().contains(&query) {
-                            return true;
-                        }
-                    }
-
-                    false
-                } else {
-                    // No search term
-                    true
                }
-            })
-            // We need to collect all, so we can sort by member count
-            .collect::<Vec<_>>();
+
+                if let Some(topic) = &chunk.topic {
+                    if topic.to_lowercase().contains(&query) {
+                        return true;
+                    }
+                }
+
+                if let Some(canonical_alias) = &chunk.canonical_alias {
+                    if canonical_alias.as_str().to_lowercase().contains(&query) {
+                        return true;
+                    }
+                }
+
+                false
+            } else {
+                // No search term
+                true
+            }
+        })
+        // We need to collect all, so we can sort by member count
+        .collect::<Vec<_>>();

    all_rooms.sort_by(|l, r| r.num_joined_members.cmp(&l.num_joined_members));

@@ -5,6 +5,7 @@ use crate::{
    server_server, utils, ConduitResult, Database, Error, Result, Ruma,
 };
 use member::{MemberEventContent, MembershipState};
+use rayon::prelude::*;
 use ruma::{
    api::{
        client::{
@@ -15,7 +16,10 @@ use ruma::{
                unban_user, IncomingThirdPartySigned,
            },
        },
-        federation::{self, membership::create_invite},
+        federation::{
+            self,
+            membership::{create_invite, create_join_event},
+        },
    },
    events::{
        pdu::Pdu,
@@ -26,13 +30,13 @@ use ruma::{
    state_res::{self, RoomVersion},
    uint, EventId, RoomId, RoomVersionId, ServerName, UserId,
 };
+use serde_json::value::RawValue;
 use std::{
-    collections::{hash_map::Entry, BTreeMap, HashMap, HashSet},
+    collections::{BTreeMap, HashMap, HashSet},
    convert::{TryFrom, TryInto},
    sync::{Arc, RwLock},
-    time::{Duration, Instant},
 };
-use tracing::{debug, error, warn};
+use tracing::{error, warn};

 #[cfg(feature = "conduit_bin")]
 use rocket::{get, post};
@@ -204,7 +208,7 @@ pub async fn kick_user_route(
 ) -> ConduitResult<kick_user::Response> {
    let sender_user = body.sender_user.as_ref().expect("user is authenticated");

-    let mut event = serde_json::from_value::<Raw<ruma::events::room::member::MemberEventContent>>(
+    let mut event = serde_json::from_str::<Raw<ruma::events::room::member::MemberEventContent>>(
        db.rooms
            .room_state_get(
                &body.room_id,
@@ -216,7 +220,7 @@ pub async fn kick_user_route(
                "Cannot kick member that's not in the room.",
            ))?
            .content
-            .clone(),
+            .get(),
    )
    .expect("Raw::from_value always works")
    .deserialize()
@@ -238,7 +242,10 @@ pub async fn kick_user_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomMember,
-            content: serde_json::to_value(event).expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&event).expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some(body.user_id.to_string()),
            redacts: None,
@@ -290,12 +297,9 @@ pub async fn ban_user_route(
                reason: None,
            }),
            |event| {
-                let mut event = serde_json::from_value::<Raw<member::MemberEventContent>>(
-                    event.content.clone(),
-                )
-                .expect("Raw::from_value always works")
-                .deserialize()
-                .map_err(|_| Error::bad_database("Invalid member event in database."))?;
+                let mut event =
+                    serde_json::from_str::<member::MemberEventContent>(event.content.get())
+                        .map_err(|_| Error::bad_database("Invalid member event in database."))?;
                event.membership = ruma::events::room::member::MembershipState::Ban;
                Ok(event)
            },
@@ -314,7 +318,10 @@ pub async fn ban_user_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomMember,
-            content: serde_json::to_value(event).expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&event).expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some(body.user_id.to_string()),
            redacts: None,
@@ -346,7 +353,7 @@ pub async fn unban_user_route(
 ) -> ConduitResult<unban_user::Response> {
    let sender_user = body.sender_user.as_ref().expect("user is authenticated");

-    let mut event = serde_json::from_value::<Raw<ruma::events::room::member::MemberEventContent>>(
+    let mut event = serde_json::from_str::<ruma::events::room::member::MemberEventContent>(
        db.rooms
            .room_state_get(
                &body.room_id,
@@ -358,10 +365,8 @@ pub async fn unban_user_route(
                "Cannot unban a user who is not banned.",
            ))?
            .content
-            .clone(),
+            .get(),
    )
-    .expect("from_value::<Raw<..>> can never fail")
-    .deserialize()
    .map_err(|_| Error::bad_database("Invalid member event in database."))?;

    event.membership = ruma::events::room::member::MembershipState::Leave;
@@ -379,7 +384,10 @@ pub async fn unban_user_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomMember,
-            content: serde_json::to_value(event).expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&event).expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some(body.user_id.to_string()),
            redacts: None,
@@ -563,6 +571,7 @@ async fn join_room_by_id_helper(
                    },
                )
                .await;
+            warn!("Make join done");

            make_join_response_and_server = make_join_response.map(|r| (r, remote_server));

@@ -658,93 +667,193 @@ async fn join_room_by_id_helper(
            )
            .await?;

+        warn!("Send join done");
+
        db.rooms.get_or_create_shortroomid(room_id, &db.globals)?;

        let pdu = PduEvent::from_id_val(&event_id, join_event.clone())
            .map_err(|_| Error::BadServerResponse("Invalid join event PDU."))?;

-        let mut state = HashMap::new();
-        let pub_key_map = RwLock::new(BTreeMap::new());
+        let pub_key_map = Arc::new(RwLock::new(BTreeMap::new()));
+        let missing_servers = Arc::new(RwLock::new(BTreeMap::new()));

-        server_server::fetch_join_signing_keys(
-            &send_join_response,
-            &room_version,
-            &pub_key_map,
-            db,
-        )
-        .await?;
-
-        for result in send_join_response
-            .room_state
-            .state
-            .iter()
-            .map(|pdu| validate_and_add_event_id(pdu, &room_version, &pub_key_map, db))
-        {
-            let (event_id, value) = match result {
-                Ok(t) => t,
-                Err(_) => continue,
-            };
-
-            let pdu = PduEvent::from_id_val(&event_id, value.clone()).map_err(|e| {
-                warn!("{:?}: {}", value, e);
-                Error::BadServerResponse("Invalid PDU in send_join response.")
-            })?;
-
-            db.rooms.add_pdu_outlier(&event_id, &value)?;
-            if let Some(state_key) = &pdu.state_key {
-                let shortstatekey =
-                    db.rooms
-                        .get_or_create_shortstatekey(&pdu.kind, state_key, &db.globals)?;
-                state.insert(shortstatekey, pdu.event_id.clone());
-            }
-        }
-
-        let incoming_shortstatekey = db.rooms.get_or_create_shortstatekey(
-            &pdu.kind,
-            pdu.state_key
-                .as_ref()
-                .expect("Pdu is a membership state event"),
-            &db.globals,
-        )?;
-
-        state.insert(incoming_shortstatekey, pdu.event_id.clone());
+        let create_join_event::RoomState {
+            state: mut room_state_state,
+            auth_chain: mut room_state_auth_chain,
+        } = send_join_response.room_state;

        let create_shortstatekey = db
            .rooms
            .get_shortstatekey(&EventType::RoomCreate, "")?
            .expect("Room exists");

-        if state.get(&create_shortstatekey).is_none() {
+        let mut saw_create_event = false;
+
+        warn!("Parsing send join response state");
+        const CHUNK_SIZE: usize = 500;
+        let mut parsed_state = room_state_state
+            .par_chunks_mut(CHUNK_SIZE)
+            .filter_map(|pdus| {
+                let mut r = HashMap::with_capacity(CHUNK_SIZE);
+                for pdu in pdus {
+                    let (id, value) = get_event_id(&pdu, &room_version).ok()?;
+                    r.insert(id, value);
+                }
+
+                let mut missing_servers = missing_servers.write().unwrap();
+                let mut pub_key_map = pub_key_map.write().unwrap();
+                for (_, value) in &r {
+                    server_server::get_server_keys_from_cache(
+                        &value,
+                        &mut missing_servers,
+                        &mut pub_key_map,
+                        &db,
+                    )
+                    .ok()?;
+                }
+
+                Some(r)
+            })
+            .collect::<Vec<_>>();
+
+        warn!("Parsing send join response auth chain");
+        let mut parsed_chain = room_state_auth_chain
+            .par_chunks_mut(CHUNK_SIZE)
+            .filter_map(|pdus| {
+                let mut r = HashMap::with_capacity(CHUNK_SIZE);
+                for pdu in pdus {
+                    let (id, value) = get_event_id(&pdu, &room_version).ok()?;
+                    r.insert(id, value);
+                }
+
+                let mut missing_servers = missing_servers.write().unwrap();
+                let mut pub_key_map = pub_key_map.write().unwrap();
+                for (_, value) in &r {
+                    server_server::get_server_keys_from_cache(
+                        &value,
+                        &mut missing_servers,
+                        &mut pub_key_map,
+                        &db,
+                    )
+                    .ok()?;
+                }
+
+                Some(r)
+            })
+            .collect::<Vec<_>>();
+
+        warn!("Fetching send join signing keys");
+        server_server::fetch_join_signing_keys(missing_servers, &pub_key_map, db).await?;
+
+        warn!("Validating state");
+        parsed_state.par_iter_mut().for_each(|chunk| {
+            let mut bad_events = Vec::new();
+            for (event_id, value) in chunk.iter_mut() {
+                if let Err(e) = ruma::signatures::verify_event(
+                    &*pub_key_map.read().unwrap(),
+                    &value,
+                    &room_version,
+                ) {
+                    warn!("Event {} failed verification {:?} {}", event_id, value, e);
+                    bad_events.push(event_id.clone());
+                    continue;
+                }
+
+                value.insert(
+                    "event_id".to_owned(),
+                    CanonicalJsonValue::String(event_id.as_str().to_owned()),
+                );
+            }
+            for id in bad_events {
+                chunk.remove(&id);
+            }
+        });
+
+        warn!("Inserting state");
+        db.rooms
+            .add_pdu_outlier_batch(&mut parsed_state.iter().flatten())?;
+
+        warn!("Compressing state");
+        let state = parsed_state
+            .iter()
+            .flatten()
+            .map(|(event_id, value)| {
+                let kind = if let Some(s) = value.get("type").and_then(|s| s.as_str()) {
+                    s
+                } else {
+                    warn!("Event {} has no type: {:?}", event_id, value);
+                    return Ok(None);
+                };
+
+                if let Some(state_key) = value.get("state_key").and_then(|s| s.as_str()) {
+                    let shortstatekey = db.rooms.get_or_create_shortstatekey(
+                        &EventType::from(kind),
+                        state_key,
+                        &db.globals,
+                    )?;
+
+                    if shortstatekey == create_shortstatekey {
+                        saw_create_event = true;
+                    }
+
+                    Ok(Some(db.rooms.compress_state_event(
+                        shortstatekey,
+                        &event_id,
+                        &db.globals,
+                    )?))
+                } else {
+                    Ok(None)
+                }
+            })
+            .filter_map(|r| r.transpose())
+            .collect::<Result<HashSet<_>>>()?;
+
+        if !saw_create_event {
            return Err(Error::BadServerResponse("State contained no create event."));
        }

-        db.rooms.force_state(
+        warn!("Validating chain");
+        parsed_chain.par_iter_mut().for_each(|chunk| {
+            let mut bad_events = Vec::new();
+            for (event_id, value) in chunk.iter_mut() {
+                if let Err(e) = ruma::signatures::verify_event(
+                    &*pub_key_map.read().unwrap(),
+                    &value,
+                    &room_version,
+                ) {
+                    warn!("Event {} failed verification {:?} {}", event_id, value, e);
+                    bad_events.push(event_id.clone());
+                    continue;
+                }
+
+                value.insert(
+                    "event_id".to_owned(),
+                    CanonicalJsonValue::String(event_id.as_str().to_owned()),
+                );
+            }
+            for id in bad_events {
+                chunk.remove(&id);
+            }
+        });
+
+        warn!("Inserting chain");
+        db.rooms
+            .add_pdu_outlier_batch(&mut parsed_chain.iter().flatten())?;
+
+        warn!("Forcing state of room");
+        db.rooms.force_state_new(
            room_id,
-            state
-                .into_iter()
-                .map(|(k, id)| db.rooms.compress_state_event(k, &id, &db.globals))
-                .collect::<Result<HashSet<_>>>()?,
+            state,
+            &mut parsed_state.iter().flat_map(|m| m.values()),
            db,
        )?;

-        for result in send_join_response
-            .room_state
-            .auth_chain
-            .iter()
-            .map(|pdu| validate_and_add_event_id(pdu, &room_version, &pub_key_map, db))
-        {
-            let (event_id, value) = match result {
-                Ok(t) => t,
-                Err(_) => continue,
-            };
-
-            db.rooms.add_pdu_outlier(&event_id, &value)?;
-        }
-
        // We append to state before appending the pdu, so we don't have a moment in time with the
        // pdu without it's state. This is okay because append_pdu can't fail.
+        warn!("Appending join event to state");
        let statehashid = db.rooms.append_to_state(&pdu, &db.globals)?;

+        warn!("Adding join event to db");
        db.rooms.append_pdu(
            &pdu,
            utils::to_canonical_object(&pdu).expect("Pdu is valid canonical object"),
@@ -752,6 +861,7 @@ async fn join_room_by_id_helper(
            db,
        )?;

+        warn!("Updating room state to join event");
        // We set the room state after inserting the pdu, so that we never have a moment in time
        // where events in the current room state do not exist
        db.rooms.set_room_state(room_id, statehashid)?;
@@ -769,7 +879,10 @@ async fn join_room_by_id_helper(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomMember,
-                content: serde_json::to_value(event).expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&event).expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some(sender_user.to_string()),
                redacts: None,
@@ -788,16 +901,15 @@ async fn join_room_by_id_helper(
    Ok(join_room_by_id::Response::new(room_id.clone()).into())
 }

-fn validate_and_add_event_id(
+fn get_event_id(
    pdu: &Raw<Pdu>,
    room_version: &RoomVersionId,
-    pub_key_map: &RwLock<BTreeMap<String, BTreeMap<String, String>>>,
-    db: &Database,
 ) -> Result<(EventId, CanonicalJsonObject)> {
-    let mut value = serde_json::from_str::<CanonicalJsonObject>(pdu.json().get()).map_err(|e| {
-        error!("Invalid PDU in server response: {:?}: {:?}", pdu, e);
+    let value = serde_json::from_str::<CanonicalJsonObject>(pdu.json().get()).map_err(|e| {
+        warn!("Invalid PDU in server response: {:?}: {:?}", pdu, e);
        Error::BadServerResponse("Invalid PDU in server response")
    })?;
+
    let event_id = EventId::try_from(&*format!(
        "${}",
        ruma::signatures::reference_hash(&value, room_version)
@@ -805,49 +917,6 @@ fn validate_and_add_event_id(
    ))
    .expect("ruma's reference hashes are valid event ids");

-    let back_off = |id| match db.globals.bad_event_ratelimiter.write().unwrap().entry(id) {
-        Entry::Vacant(e) => {
-            e.insert((Instant::now(), 1));
-        }
-        Entry::Occupied(mut e) => *e.get_mut() = (Instant::now(), e.get().1 + 1),
-    };
-
-    if let Some((time, tries)) = db
-        .globals
-        .bad_event_ratelimiter
-        .read()
-        .unwrap()
-        .get(&event_id)
-    {
-        // Exponential backoff
-        let mut min_elapsed_duration = Duration::from_secs(30) * (*tries) * (*tries);
-        if min_elapsed_duration > Duration::from_secs(60 * 60 * 24) {
-            min_elapsed_duration = Duration::from_secs(60 * 60 * 24);
-        }
-
-        if time.elapsed() < min_elapsed_duration {
-            debug!("Backing off from {}", event_id);
-            return Err(Error::BadServerResponse("bad event, still backing off"));
-        }
-    }
-
-    if let Err(e) = ruma::signatures::verify_event(
-        &*pub_key_map
-            .read()
-            .map_err(|_| Error::bad_database("RwLock is poisoned."))?,
-        &value,
-        room_version,
-    ) {
-        warn!("Event {} failed verification {:?} {}", event_id, pdu, e);
-        back_off(event_id);
-        return Err(Error::BadServerResponse("Event failed verification."));
-    }
-
-    value.insert(
-        "event_id".to_owned(),
-        CanonicalJsonValue::String(event_id.as_str().to_owned()),
-    );
-
    Ok((event_id, value))
 }

@@ -884,7 +953,7 @@ pub(crate) async fn invite_helper<'a>(
            let create_event_content = create_event
                .as_ref()
                .map(|create_event| {
-                    serde_json::from_value::<Raw<CreateEventContent>>(create_event.content.clone())
+                    serde_json::from_str::<Raw<CreateEventContent>>(create_event.content.get())
                        .expect("Raw::from_value always works.")
                        .deserialize()
                        .map_err(|e| {
@@ -910,16 +979,19 @@ pub(crate) async fn invite_helper<'a>(
            let room_version =
                RoomVersion::new(&room_version_id).expect("room version is supported");

-            let content = serde_json::to_value(MemberEventContent {
-                avatar_url: None,
-                displayname: None,
-                is_direct: Some(is_direct),
-                membership: MembershipState::Invite,
-                third_party_invite: None,
-                blurhash: None,
-                reason: None,
-            })
-            .expect("member event is valid value");
+            let content = RawValue::from_string(
+                serde_json::to_string(&MemberEventContent {
+                    avatar_url: None,
+                    displayname: None,
+                    is_direct: Some(is_direct),
+                    membership: MembershipState::Invite,
+                    third_party_invite: None,
+                    blurhash: None,
+                    reason: None,
+                })
+                .expect("member event is valid value"),
+            )
+            .expect("string is valid");

            let state_key = user_id.to_string();
            let kind = EventType::RoomMember;
@@ -946,7 +1018,7 @@ pub(crate) async fn invite_helper<'a>(
                unsigned.insert("prev_content".to_owned(), prev_pdu.content.clone());
                unsigned.insert(
                    "prev_sender".to_owned(),
-                    serde_json::to_value(&prev_pdu.sender).expect("UserId::to_value always works"),
+                    serde_json::from_str(prev_pdu.sender.as_str()).expect("UserId is valid string"),
                );
            }

@@ -959,6 +1031,7 @@ pub(crate) async fn invite_helper<'a>(
                    .expect("time is valid"),
                kind,
                content,
+                parsed_content: RwLock::new(None),
                state_key: Some(state_key),
                prev_events,
                depth,
@@ -967,11 +1040,20 @@ pub(crate) async fn invite_helper<'a>(
                    .map(|(_, pdu)| pdu.event_id.clone())
                    .collect(),
                redacts: None,
-                unsigned,
+                unsigned: if unsigned.is_empty() {
+                    None
+                } else {
+                    Some(
+                        RawValue::from_string(
+                            serde_json::to_string(&unsigned).expect("to_string always works"),
+                        )
+                        .expect("string is valid"),
+                    )
+                },
                hashes: ruma::events::pdu::EventHash {
                    sha256: "aaa".to_owned(),
                },
-                signatures: BTreeMap::new(),
+                signatures: None,
            };

            let auth_check = state_res::auth_check(
@@ -1092,14 +1174,13 @@ pub(crate) async fn invite_helper<'a>(
            "Could not accept incoming PDU as timeline event.",
        ))?;

-        for server in db
+        let servers = db
            .rooms
            .room_servers(room_id)
            .filter_map(|r| r.ok())
-            .filter(|server| &**server != db.globals.server_name())
-        {
-            db.sending.send_pdu(&server, &pdu_id)?;
-        }
+            .filter(|server| &**server != db.globals.server_name());
+
+        db.sending.send_pdu(servers, &pdu_id)?;

        return Ok(());
    }
@@ -1117,16 +1198,19 @@ pub(crate) async fn invite_helper<'a>(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomMember,
-            content: serde_json::to_value(member::MemberEventContent {
-                membership: member::MembershipState::Invite,
-                displayname: db.users.displayname(user_id)?,
-                avatar_url: db.users.avatar_url(user_id)?,
-                is_direct: Some(is_direct),
-                third_party_invite: None,
-                blurhash: db.users.blurhash(user_id)?,
-                reason: None,
-            })
-            .expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&member::MemberEventContent {
+                    membership: member::MembershipState::Invite,
+                    displayname: db.users.displayname(user_id)?,
+                    avatar_url: db.users.avatar_url(user_id)?,
+                    is_direct: Some(is_direct),
+                    third_party_invite: None,
+                    blurhash: db.users.blurhash(user_id)?,
+                    reason: None,
+                })
+                .expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some(user_id.to_string()),
            redacts: None,
@@ -10,8 +10,8 @@ use ruma::{
        federation::{self, query::get_profile_information::v1::ProfileField},
    },
    events::EventType,
-    serde::Raw,
 };
+use serde_json::value::RawValue;
 use std::{convert::TryInto, sync::Arc};

 #[cfg(feature = "conduit_bin")]
@@ -45,29 +45,30 @@ pub async fn set_displayname_route(
            Ok::<_, Error>((
                PduBuilder {
                    event_type: EventType::RoomMember,
-                    content: serde_json::to_value(ruma::events::room::member::MemberEventContent {
-                        displayname: body.displayname.clone(),
-                        ..serde_json::from_value::<Raw<_>>(
-                            db.rooms
-                                .room_state_get(
-                                    &room_id,
-                                    &EventType::RoomMember,
-                                    &sender_user.to_string(),
-                                )?
-                                .ok_or_else(|| {
-                                    Error::bad_database(
-                                        "Tried to send displayname update for user not in the \
+                    content: RawValue::from_string(
+                        serde_json::to_string(&ruma::events::room::member::MemberEventContent {
+                            displayname: body.displayname.clone(),
+                            ..serde_json::from_str(
+                                db.rooms
+                                    .room_state_get(
+                                        &room_id,
+                                        &EventType::RoomMember,
+                                        &sender_user.to_string(),
+                                    )?
+                                    .ok_or_else(|| {
+                                        Error::bad_database(
+                                            "Tried to send displayname update for user not in the \
                                     room.",
-                                    )
-                                })?
-                                .content
-                                .clone(),
-                        )
-                        .expect("from_value::<Raw<..>> can never fail")
-                        .deserialize()
-                        .map_err(|_| Error::bad_database("Database contains invalid PDU."))?
-                    })
-                    .expect("event is valid, we just created it"),
+                                        )
+                                    })?
+                                    .content
+                                    .get(),
+                            )
+                            .map_err(|_| Error::bad_database("Database contains invalid PDU."))?
+                        })
+                        .expect("event is valid, we just created it"),
+                    )
+                    .expect("string is valid"),
                    unsigned: None,
                    state_key: Some(sender_user.to_string()),
                    redacts: None,
@@ -190,29 +191,30 @@ pub async fn set_avatar_url_route(
            Ok::<_, Error>((
                PduBuilder {
                    event_type: EventType::RoomMember,
-                    content: serde_json::to_value(ruma::events::room::member::MemberEventContent {
-                        avatar_url: body.avatar_url.clone(),
-                        ..serde_json::from_value::<Raw<_>>(
-                            db.rooms
-                                .room_state_get(
-                                    &room_id,
-                                    &EventType::RoomMember,
-                                    &sender_user.to_string(),
-                                )?
-                                .ok_or_else(|| {
-                                    Error::bad_database(
-                                        "Tried to send displayname update for user not in the \
+                    content: RawValue::from_string(
+                        serde_json::to_string(&ruma::events::room::member::MemberEventContent {
+                            avatar_url: body.avatar_url.clone(),
+                            ..serde_json::from_str(
+                                db.rooms
+                                    .room_state_get(
+                                        &room_id,
+                                        &EventType::RoomMember,
+                                        &sender_user.to_string(),
+                                    )?
+                                    .ok_or_else(|| {
+                                        Error::bad_database(
+                                            "Tried to send displayname update for user not in the \
                                     room.",
-                                    )
-                                })?
-                                .content
-                                .clone(),
-                        )
-                        .expect("from_value::<Raw<..>> can never fail")
-                        .deserialize()
-                        .map_err(|_| Error::bad_database("Database contains invalid PDU."))?
-                    })
-                    .expect("event is valid, we just created it"),
+                                        )
+                                    })?
+                                    .content
+                                    .get(),
+                            )
+                            .map_err(|_| Error::bad_database("Database contains invalid PDU."))?
+                        })
+                        .expect("event is valid, we just created it"),
+                    )
+                    .expect("string is valid"),
                    unsigned: None,
                    state_key: Some(sender_user.to_string()),
                    redacts: None,
@@ -8,6 +8,7 @@ use ruma::{

 #[cfg(feature = "conduit_bin")]
 use rocket::put;
+use serde_json::value::RawValue;

 /// # `PUT /_matrix/client/r0/rooms/{roomId}/redact/{eventId}/{txnId}`
 ///
@@ -38,10 +39,13 @@ pub async fn redact_event_route(
    let event_id = db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomRedaction,
-            content: serde_json::to_value(redaction::RedactionEventContent {
-                reason: body.reason.clone(),
-            })
-            .expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&redaction::RedactionEventContent {
+                    reason: body.reason.clone(),
+                })
+                .expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: None,
            redacts: Some(body.event_id.clone()),
@@ -11,9 +11,9 @@ use ruma::{
        room::{guest_access, history_visibility, join_rules, member, name, topic},
        EventType,
    },
-    serde::Raw,
    RoomAliasId, RoomId, RoomVersionId,
 };
+use serde_json::value::RawValue;
 use std::{cmp::max, collections::BTreeMap, convert::TryFrom, sync::Arc};
 use tracing::{info, warn};

@@ -101,7 +101,10 @@ pub async fn create_room_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomCreate,
-            content: serde_json::to_value(content).expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&content).expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some("".to_owned()),
            redacts: None,
@@ -116,16 +119,19 @@ pub async fn create_room_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomMember,
-            content: serde_json::to_value(member::MemberEventContent {
-                membership: member::MembershipState::Join,
-                displayname: db.users.displayname(sender_user)?,
-                avatar_url: db.users.avatar_url(sender_user)?,
-                is_direct: Some(body.is_direct),
-                third_party_invite: None,
-                blurhash: db.users.blurhash(sender_user)?,
-                reason: None,
-            })
-            .expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&member::MemberEventContent {
+                    membership: member::MembershipState::Join,
+                    displayname: db.users.displayname(sender_user)?,
+                    avatar_url: db.users.avatar_url(sender_user)?,
+                    is_direct: Some(body.is_direct),
+                    third_party_invite: None,
+                    blurhash: db.users.blurhash(sender_user)?,
+                    reason: None,
+                })
+                .expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some(sender_user.to_string()),
            redacts: None,
@@ -180,7 +186,8 @@ pub async fn create_room_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomPowerLevels,
-            content: power_levels_content,
+            content: RawValue::from_string(power_levels_content.to_string())
+                .expect("string is valid"),
            unsigned: None,
            state_key: Some("".to_owned()),
            redacts: None,
@@ -196,13 +203,16 @@ pub async fn create_room_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomCanonicalAlias,
-                content: serde_json::to_value(
-                    ruma::events::room::canonical_alias::CanonicalAliasEventContent {
-                        alias: Some(room_alias_id.clone()),
-                        alt_aliases: vec![],
-                    },
+                content: RawValue::from_string(
+                    serde_json::to_string(
+                        &ruma::events::room::canonical_alias::CanonicalAliasEventContent {
+                            alias: Some(room_alias_id.clone()),
+                            alt_aliases: vec![],
+                        },
+                    )
+                    .expect("We checked that alias earlier, it must be fine"),
                )
-                .expect("We checked that alias earlier, it must be fine"),
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -220,17 +230,15 @@ pub async fn create_room_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomJoinRules,
-            content: match preset {
-                create_room::RoomPreset::PublicChat => serde_json::to_value(
-                    join_rules::JoinRulesEventContent::new(join_rules::JoinRule::Public),
-                )
+            content: RawValue::from_string(
+                serde_json::to_string(&join_rules::JoinRulesEventContent::new(match preset {
+                    create_room::RoomPreset::PublicChat => join_rules::JoinRule::Public,
+                    // according to spec "invite" is the default
+                    _ => join_rules::JoinRule::Invite,
+                }))
                .expect("event is valid, we just created it"),
-                // according to spec "invite" is the default
-                _ => serde_json::to_value(join_rules::JoinRulesEventContent::new(
-                    join_rules::JoinRule::Invite,
-                ))
-                .expect("event is valid, we just created it"),
-            },
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some("".to_owned()),
            redacts: None,
@@ -245,10 +253,13 @@ pub async fn create_room_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomHistoryVisibility,
-            content: serde_json::to_value(history_visibility::HistoryVisibilityEventContent::new(
-                history_visibility::HistoryVisibility::Shared,
-            ))
-            .expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&history_visibility::HistoryVisibilityEventContent::new(
+                    history_visibility::HistoryVisibility::Shared,
+                ))
+                .expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some("".to_owned()),
            redacts: None,
@@ -263,18 +274,14 @@ pub async fn create_room_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomGuestAccess,
-            content: match preset {
-                create_room::RoomPreset::PublicChat => {
-                    serde_json::to_value(guest_access::GuestAccessEventContent::new(
-                        guest_access::GuestAccess::Forbidden,
-                    ))
-                    .expect("event is valid, we just created it")
-                }
-                _ => serde_json::to_value(guest_access::GuestAccessEventContent::new(
-                    guest_access::GuestAccess::CanJoin,
-                ))
+            content: RawValue::from_string(
+                serde_json::to_string(&guest_access::GuestAccessEventContent::new(match preset {
+                    create_room::RoomPreset::PublicChat => guest_access::GuestAccess::Forbidden,
+                    _ => guest_access::GuestAccess::CanJoin,
+                }))
                .expect("event is valid, we just created it"),
-            },
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some("".to_owned()),
            redacts: None,
@@ -306,8 +313,11 @@ pub async fn create_room_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomName,
-                content: serde_json::to_value(name::NameEventContent::new(Some(name.clone())))
-                    .expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&name::NameEventContent::new(Some(name.clone())))
+                        .expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -323,10 +333,13 @@ pub async fn create_room_route(
        db.rooms.build_and_append_pdu(
            PduBuilder {
                event_type: EventType::RoomTopic,
-                content: serde_json::to_value(topic::TopicEventContent {
-                    topic: topic.clone(),
-                })
-                .expect("event is valid, we just created it"),
+                content: RawValue::from_string(
+                    serde_json::to_string(&topic::TopicEventContent {
+                        topic: topic.clone(),
+                    })
+                    .expect("event is valid, we just created it"),
+                )
+                .expect("string is valid"),
                unsigned: None,
                state_key: Some("".to_owned()),
                redacts: None,
@@ -477,11 +490,14 @@ pub async fn upgrade_room_route(
    let tombstone_event_id = db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomTombstone,
-            content: serde_json::to_value(ruma::events::room::tombstone::TombstoneEventContent {
-                body: "This room has been replaced".to_string(),
-                replacement_room: replacement_room.clone(),
-            })
-            .expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&ruma::events::room::tombstone::TombstoneEventContent {
+                    body: "This room has been replaced".to_string(),
+                    replacement_room: replacement_room.clone(),
+                })
+                .expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some("".to_owned()),
            redacts: None,
@@ -505,15 +521,13 @@ pub async fn upgrade_room_route(
    let state_lock = mutex_state.lock().await;

    // Get the old room federations status
-    let federate = serde_json::from_value::<Raw<ruma::events::room::create::CreateEventContent>>(
+    let federate = serde_json::from_str::<ruma::events::room::create::CreateEventContent>(
        db.rooms
            .room_state_get(&body.room_id, &EventType::RoomCreate, "")?
            .ok_or_else(|| Error::bad_database("Found room without m.room.create event."))?
            .content
-            .clone(),
+            .get(),
    )
-    .expect("Raw::from_value always works")
-    .deserialize()
    .map_err(|_| Error::bad_database("Invalid room event in database."))?
    .federate;

@@ -533,8 +547,11 @@ pub async fn upgrade_room_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomCreate,
-            content: serde_json::to_value(create_event_content)
-                .expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&create_event_content)
+                    .expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some("".to_owned()),
            redacts: None,
@@ -549,16 +566,19 @@ pub async fn upgrade_room_route(
    db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomMember,
-            content: serde_json::to_value(member::MemberEventContent {
-                membership: member::MembershipState::Join,
-                displayname: db.users.displayname(sender_user)?,
-                avatar_url: db.users.avatar_url(sender_user)?,
-                is_direct: None,
-                third_party_invite: None,
-                blurhash: db.users.blurhash(sender_user)?,
-                reason: None,
-            })
-            .expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&member::MemberEventContent {
+                    membership: member::MembershipState::Join,
+                    displayname: db.users.displayname(sender_user)?,
+                    avatar_url: db.users.avatar_url(sender_user)?,
+                    is_direct: None,
+                    third_party_invite: None,
+                    blurhash: db.users.blurhash(sender_user)?,
+                    reason: None,
+                })
+                .expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some(sender_user.to_string()),
            redacts: None,
@@ -585,7 +605,7 @@ pub async fn upgrade_room_route(
    // Replicate transferable state events to the new room
    for event_type in transferable_state_events {
        let event_content = match db.rooms.room_state_get(&body.room_id, &event_type, "")? {
-            Some(v) => v.content.clone(),
+            Some(v) => v.content,
            None => continue, // Skipping missing events.
        };

@@ -612,15 +632,13 @@ pub async fn upgrade_room_route(

    // Get the old room power levels
    let mut power_levels_event_content =
-        serde_json::from_value::<Raw<ruma::events::room::power_levels::PowerLevelsEventContent>>(
+        serde_json::from_str::<ruma::events::room::power_levels::PowerLevelsEventContent>(
            db.rooms
                .room_state_get(&body.room_id, &EventType::RoomPowerLevels, "")?
                .ok_or_else(|| Error::bad_database("Found room without m.room.create event."))?
                .content
-                .clone(),
+                .get(),
        )
-        .expect("database contains invalid PDU")
-        .deserialize()
        .map_err(|_| Error::bad_database("Invalid room event in database."))?;

    // Setting events_default and invite to the greater of 50 and users_default + 1
@@ -635,8 +653,11 @@ pub async fn upgrade_room_route(
    let _ = db.rooms.build_and_append_pdu(
        PduBuilder {
            event_type: EventType::RoomPowerLevels,
-            content: serde_json::to_value(power_levels_event_content)
-                .expect("event is valid, we just created it"),
+            content: RawValue::from_string(
+                serde_json::to_string(&power_levels_event_content)
+                    .expect("event is valid, we just created it"),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some("".to_owned()),
            redacts: None,
@@ -112,7 +112,7 @@ pub async fn get_state_events_route(
            db.rooms
                .room_state_get(&body.room_id, &EventType::RoomHistoryVisibility, "")?
                .map(|event| {
-                    serde_json::from_value::<HistoryVisibilityEventContent>(event.content.clone())
+                    serde_json::from_str::<HistoryVisibilityEventContent>(event.content.get())
                        .map_err(|_| {
                            Error::bad_database(
                                "Invalid room history visibility event in database.",
@@ -164,7 +164,7 @@ pub async fn get_state_events_for_key_route(
            db.rooms
                .room_state_get(&body.room_id, &EventType::RoomHistoryVisibility, "")?
                .map(|event| {
-                    serde_json::from_value::<HistoryVisibilityEventContent>(event.content.clone())
+                    serde_json::from_str::<HistoryVisibilityEventContent>(event.content.get())
                        .map_err(|_| {
                            Error::bad_database(
                                "Invalid room history visibility event in database.",
@@ -190,7 +190,7 @@ pub async fn get_state_events_for_key_route(
        ))?;

    Ok(get_state_events_for_key::Response {
-        content: serde_json::from_value(event.content.clone())
+        content: serde_json::from_str(event.content.get())
            .map_err(|_| Error::bad_database("Invalid event content in database"))?,
    }
    .into())
@@ -220,7 +220,7 @@ pub async fn get_state_events_for_empty_key_route(
            db.rooms
                .room_state_get(&body.room_id, &EventType::RoomHistoryVisibility, "")?
                .map(|event| {
-                    serde_json::from_value::<HistoryVisibilityEventContent>(event.content.clone())
+                    serde_json::from_str::<HistoryVisibilityEventContent>(event.content.get())
                        .map_err(|_| {
                            Error::bad_database(
                                "Invalid room history visibility event in database.",
@@ -246,7 +246,7 @@ pub async fn get_state_events_for_empty_key_route(
        ))?;

    Ok(get_state_events_for_key::Response {
-        content: serde_json::from_value(event.content.clone())
+        content: serde_json::from_str(event.content.get())
            .map_err(|_| Error::bad_database("Invalid event content in database"))?,
    }
    .into())
@@ -287,9 +287,9 @@ async fn sync_helper(
                    .filter_map(|pdu| pdu.ok()) // Ignore all broken pdus
                    .filter(|(_, pdu)| pdu.kind == EventType::RoomMember)
                    .map(|(_, pdu)| {
-                        let content = serde_json::from_value::<
+                        let content = serde_json::from_str::<
                            ruma::events::room::member::MemberEventContent,
-                        >(pdu.content.clone())
+                        >(pdu.content.get())
                        .map_err(|_| Error::bad_database("Invalid member event in database."))?;

                        if let Some(state_key) = &pdu.state_key {
@@ -371,11 +371,9 @@ async fn sync_helper(
                    sender_user.as_str(),
                )?
                .and_then(|pdu| {
-                    serde_json::from_value::<Raw<ruma::events::room::member::MemberEventContent>>(
-                        pdu.content.clone(),
+                    serde_json::from_str::<ruma::events::room::member::MemberEventContent>(
+                        pdu.content.get(),
                    )
-                    .expect("Raw::from_value always works")
-                    .deserialize()
                    .map_err(|_| Error::bad_database("Invalid PDU in database."))
                    .ok()
                });
@@ -432,11 +430,9 @@ async fn sync_helper(
                            continue;
                        }

-                        let new_membership = serde_json::from_value::<
-                            Raw<ruma::events::room::member::MemberEventContent>,
-                        >(state_event.content.clone())
-                        .expect("Raw::from_value always works")
-                        .deserialize()
+                        let new_membership = serde_json::from_str::<
+                            ruma::events::room::member::MemberEventContent,
+                        >(state_event.content.get())
                        .map_err(|_| Error::bad_database("Invalid PDU in database."))?
                        .membership;

@@ -9,6 +9,7 @@ use ruma::{
    events::{room::message, EventType},
    UserId,
 };
+use serde_json::value::RawValue;
 use tokio::sync::{MutexGuard, RwLock, RwLockReadGuard};
 use tracing::warn;

@@ -66,8 +67,11 @@ impl Admin {
                    .build_and_append_pdu(
                        PduBuilder {
                            event_type: EventType::RoomMessage,
-                            content: serde_json::to_value(message)
-                                .expect("event is valid, we just created it"),
+                            content: RawValue::from_string(
+                                serde_json::to_string(&message)
+                                    .expect("event is valid, we just created it"),
+                            )
+                            .expect("string is valid"),
                            unsigned: None,
                            state_key: None,
                            redacts: None,
@@ -9,8 +9,10 @@ use ruma::{
        },
        IncomingResponse, OutgoingRequest, SendAccessToken,
    },
-    events::{room::power_levels::PowerLevelsEventContent, AnySyncRoomEvent, EventType},
-    identifiers::RoomName,
+    events::{
+        room::{name::NameEventContent, power_levels::PowerLevelsEventContent},
+        AnySyncRoomEvent, EventType,
+    },
    push::{Action, PushConditionRoomCtx, PushFormat, Ruleset, Tweak},
    serde::Raw,
    uint, RoomId, UInt, UserId,
@@ -181,7 +183,7 @@ pub async fn send_push_notice(
        .rooms
        .room_state_get(&pdu.room_id, &EventType::RoomPowerLevels, "")?
        .map(|ev| {
-            serde_json::from_value(ev.content.clone())
+            serde_json::from_str(ev.content.get())
                .map_err(|_| Error::bad_database("invalid m.room.power_levels event"))
        })
        .transpose()?
@@ -318,16 +320,18 @@ async fn send_notice(

        let user_name = db.users.displayname(&event.sender)?;
        notifi.sender_display_name = user_name.as_deref();
-        let room_name = db
-            .rooms
-            .room_state_get(&event.room_id, &EventType::RoomName, "")?
-            .map(|pdu| match pdu.content.get("name") {
-                Some(serde_json::Value::String(s)) => {
-                    Some(Box::<RoomName>::try_from(&**s).expect("room name is valid"))
-                }
-                _ => None,
-            })
-            .flatten();
+
+        let room_name = if let Some(room_name_pdu) =
+            db.rooms
+                .room_state_get(&event.room_id, &EventType::RoomName, "")?
+        {
+            serde_json::from_str::<NameEventContent>(room_name_pdu.content.get())
+                .map_err(|_| Error::bad_database("Invalid room name event in database."))?
+                .name
+        } else {
+            None
+        };
+
        notifi.room_name = room_name.as_deref();

        send_request(
@@ -2,6 +2,7 @@ mod edus;

 pub use edus::RoomEdus;
 use member::MembershipState;
+use serde_json::value::RawValue;

 use crate::{pdu::PduBuilder, server_server, utils, Database, Error, PduEvent, Result};
 use lru_cache::LruCache;
@@ -96,7 +97,7 @@ pub struct Rooms {
    /// RoomId + EventId -> Parent PDU EventId.
    pub(super) referencedevents: Arc<dyn Tree>,

-    pub(super) pdu_cache: Mutex<LruCache<EventId, Arc<PduEvent>>>,
+    pub(super) pdu_cache: Mutex<LruCache<EventId, Vec<u8>>>,
    pub(super) shorteventid_cache: Mutex<LruCache<u64, Arc<EventId>>>,
    pub(super) auth_chain_cache: Mutex<LruCache<Vec<u64>, Arc<HashSet<u64>>>>,
    pub(super) eventidshort_cache: Mutex<LruCache<EventId, u64>>,
@@ -137,7 +138,7 @@ impl Rooms {
    pub fn state_full(
        &self,
        shortstatehash: u64,
-    ) -> Result<HashMap<(EventType, String), Arc<PduEvent>>> {
+    ) -> Result<HashMap<(EventType, String), PduEvent>> {
        let full_state = self
            .load_shortstatehash_info(shortstatehash)?
            .pop()
@@ -199,7 +200,7 @@ impl Rooms {
        shortstatehash: u64,
        event_type: &EventType,
        state_key: &str,
-    ) -> Result<Option<Arc<PduEvent>>> {
+    ) -> Result<Option<PduEvent>> {
        self.state_get_id(shortstatehash, event_type, state_key)?
            .map_or(Ok(None), |event_id| self.get_pdu(&event_id))
    }
@@ -243,8 +244,8 @@ impl Rooms {
        kind: &EventType,
        sender: &UserId,
        state_key: Option<&str>,
-        content: &serde_json::Value,
-    ) -> Result<StateMap<Arc<PduEvent>>> {
+        content: &serde_json::value::RawValue,
+    ) -> Result<StateMap<PduEvent>> {
        let shortstatehash =
            if let Some(current_shortstatehash) = self.current_shortstatehash(room_id)? {
                current_shortstatehash
@@ -252,7 +253,12 @@ impl Rooms {
                return Ok(HashMap::new());
            };

-        let auth_events = state_res::auth_types_for_event(kind, sender, state_key, content);
+        let auth_events = state_res::auth_types_for_event(
+            kind,
+            sender,
+            state_key,
+            &serde_json::from_str(content.get()).expect("RawValue is valid json"),
+        );

        let mut sauthevents = auth_events
            .into_iter()
@@ -329,6 +335,75 @@ impl Rooms {
            .transpose()
    }

+    /// Force the creation of a new StateHash and insert it into the db. This version should be
+    /// used when creating new rooms.
+    ///
+    /// Whatever `state` is supplied to `force_state` becomes the new current room state snapshot.
+    #[tracing::instrument(skip(self, new_state_ids_compressed, state_values, db))]
+    pub fn force_state_new(
+        &self,
+        room_id: &RoomId,
+        new_state_ids_compressed: HashSet<CompressedStateEvent>,
+        state_values: &mut dyn Iterator<Item = &CanonicalJsonObject>,
+        db: &Database,
+    ) -> Result<()> {
+        let state_hash = self.calculate_hash(
+            &new_state_ids_compressed
+                .iter()
+                .map(|bytes| &bytes[..])
+                .collect::<Vec<_>>(),
+        );
+
+        let (new_shortstatehash, _) =
+            self.get_or_create_shortstatehash(&state_hash, &db.globals)?;
+
+        self.save_state_from_diff(
+            new_shortstatehash,
+            new_state_ids_compressed.clone(),
+            HashSet::new(),
+            2, // every state change is 2 event changes on average
+            Vec::new(),
+        )?;
+
+        for pdu in state_values {
+            if pdu.get("type").and_then(|val| val.as_str()) == Some("m.room.member") {
+                let content = pdu
+                    .get("content")
+                    .and_then(|o| o.as_object())
+                    .ok_or_else(|| Error::bad_database("Invalid content in pdu."))?;
+                if let Some(membership) = content.get("membership").and_then(|membership| {
+                    serde_json::from_str::<member::MembershipState>(
+                        &serde_json::to_string(membership).expect("json is valid string"),
+                    )
+                    .ok()
+                }) {
+                    if let Some(state_key) = pdu
+                        .get("state_key")
+                        .and_then(|o| o.as_str())
+                        .and_then(|state_key| UserId::try_from(state_key).ok())
+                    {
+                        if let Some(sender) = pdu
+                            .get("sender")
+                            .and_then(|o| o.as_str())
+                            .and_then(|state_key| UserId::try_from(state_key).ok())
+                        {
+                            self.update_membership(
+                                room_id, &state_key, membership, &sender, None, db, false,
+                            )?;
+                        }
+                    }
+                }
+            }
+        }
+
+        self.update_joined_count(room_id, db)?;
+
+        self.roomid_shortstatehash
+            .insert(room_id.as_bytes(), &new_shortstatehash.to_be_bytes())?;
+
+        Ok(())
+    }
+
    /// Force the creation of a new StateHash and insert it into the db.
    ///
    /// Whatever `state` is supplied to `force_state` becomes the new current room state snapshot.
@@ -393,29 +468,28 @@ impl Rooms {
        }) {
            if let Some(pdu) = self.get_pdu_json(&event_id)? {
                if pdu.get("type").and_then(|val| val.as_str()) == Some("m.room.member") {
-                    if let Ok(pdu) = serde_json::from_value::<PduEvent>(
-                        serde_json::to_value(&pdu).expect("CanonicalJsonObj is a valid JsonValue"),
-                    ) {
-                        if let Some(membership) =
-                            pdu.content.get("membership").and_then(|membership| {
-                                serde_json::from_value::<member::MembershipState>(
-                                    membership.clone(),
-                                )
-                                .ok()
-                            })
+                    let content = pdu
+                        .get("content")
+                        .and_then(|o| o.as_object())
+                        .ok_or_else(|| Error::bad_database("Invalid content in pdu."))?;
+                    if let Some(membership) = content.get("membership").and_then(|membership| {
+                        serde_json::from_str::<member::MembershipState>(
+                            &serde_json::to_string(membership).expect("json is valid string"),
+                        )
+                        .ok()
+                    }) {
+                        if let Some(state_key) = pdu
+                            .get("state_key")
+                            .and_then(|o| o.as_str())
+                            .and_then(|state_key| UserId::try_from(state_key).ok())
                        {
-                            if let Some(state_key) = pdu
-                                .state_key
+                            if let Some(sender) = pdu
+                                .get("sender")
+                                .and_then(|o| o.as_str())
                                .and_then(|state_key| UserId::try_from(state_key).ok())
                            {
                                self.update_membership(
-                                    room_id,
-                                    &state_key,
-                                    membership,
-                                    &pdu.sender,
-                                    None,
-                                    db,
-                                    false,
+                                    room_id, &state_key, membership, &sender, None, db, false,
                                )?;
                            }
                        }
@@ -915,7 +989,7 @@ impl Rooms {
    pub fn room_state_full(
        &self,
        room_id: &RoomId,
-    ) -> Result<HashMap<(EventType, String), Arc<PduEvent>>> {
+    ) -> Result<HashMap<(EventType, String), PduEvent>> {
        if let Some(current_shortstatehash) = self.current_shortstatehash(room_id)? {
            self.state_full(current_shortstatehash)
        } else {
@@ -945,7 +1019,7 @@ impl Rooms {
        room_id: &RoomId,
        event_type: &EventType,
        state_key: &str,
-    ) -> Result<Option<Arc<PduEvent>>> {
+    ) -> Result<Option<PduEvent>> {
        if let Some(current_shortstatehash) = self.current_shortstatehash(room_id)? {
            self.state_get(current_shortstatehash, event_type, state_key)
        } else {
@@ -1073,9 +1147,11 @@ impl Rooms {
    ///
    /// Checks the `eventid_outlierpdu` Tree if not found in the timeline.
    #[tracing::instrument(skip(self))]
-    pub fn get_pdu(&self, event_id: &EventId) -> Result<Option<Arc<PduEvent>>> {
+    pub fn get_pdu(&self, event_id: &EventId) -> Result<Option<PduEvent>> {
        if let Some(p) = self.pdu_cache.lock().unwrap().get_mut(event_id) {
-            return Ok(Some(Arc::clone(p)));
+            return Ok(Some(
+                serde_json::from_slice(p).expect("pdus in cache are valid"),
+            ));
        }

        if let Some(pdu) = self
@@ -1092,18 +1168,13 @@ impl Rooms {
                    })?))
                },
            )?
-            .map(|pdu| {
-                serde_json::from_slice(&pdu)
-                    .map_err(|_| Error::bad_database("Invalid PDU in db."))
-                    .map(Arc::new)
-            })
-            .transpose()?
        {
-            self.pdu_cache
-                .lock()
-                .unwrap()
-                .insert(event_id.clone(), Arc::clone(&pdu));
-            Ok(Some(pdu))
+            let parsed =
+                serde_json::from_slice(&pdu).map_err(|_| Error::bad_database("Invalid PDU in db."));
+
+            self.pdu_cache.lock().unwrap().insert(event_id.clone(), pdu);
+
+            parsed
        } else {
            Ok(None)
        }
@@ -1228,6 +1299,23 @@ impl Rooms {
        )
    }

+    /// Append the PDU as an outlier.
+    ///
+    /// Any event given to this will be processed (state-res) on another thread.
+    #[tracing::instrument(skip(self, batch))]
+    pub fn add_pdu_outlier_batch(
+        &self,
+        batch: &mut dyn Iterator<Item = (&EventId, &CanonicalJsonObject)>,
+    ) -> Result<()> {
+        self.eventid_outlierpdu
+            .insert_batch(&mut batch.map(|(event_id, pdu)| {
+                (
+                    event_id.as_bytes().to_vec(),
+                    serde_json::to_vec(&pdu).expect("CanonicalJsonObject is valid"),
+                )
+            }))
+    }
+
    #[tracing::instrument(skip(self))]
    pub fn mark_event_soft_failed(&self, event_id: &EventId) -> Result<()> {
        self.softfailedeventids.insert(event_id.as_bytes(), &[])
@@ -1329,7 +1417,7 @@ impl Rooms {
            .rooms
            .room_state_get(&pdu.room_id, &EventType::RoomPowerLevels, "")?
            .map(|ev| {
-                serde_json::from_value(ev.content.clone())
+                serde_json::from_str(ev.content.get())
                    .map_err(|_| Error::bad_database("invalid m.room.power_levels event"))
            })
            .transpose()?
@@ -1408,8 +1496,11 @@ impl Rooms {
                    let target_user_id = UserId::try_from(state_key.clone())
                        .expect("This state_key was previously validated");

+                    let content = serde_json::from_str::<serde_json::Value>(pdu.content.get())
+                        .map_err(|_| Error::bad_database("Invalid content in pdu."))?;
+
                    let membership = serde_json::from_value::<member::MembershipState>(
-                        pdu.content
+                        content
                            .get("membership")
                            .ok_or(Error::BadRequest(
                                ErrorKind::InvalidParam,
@@ -1447,7 +1538,10 @@ impl Rooms {
                }
            }
            EventType::RoomMessage => {
-                if let Some(body) = pdu.content.get("body").and_then(|b| b.as_str()) {
+                let content = serde_json::from_str::<serde_json::Value>(pdu.content.get())
+                    .map_err(|_| Error::bad_database("Invalid content in pdu."))?;
+
+                if let Some(body) = content.get("body").and_then(|b| b.as_str()) {
                    let mut batch = body
                        .split_terminator(|c: char| !c.is_alphanumeric())
                        .filter(|s| !s.is_empty())
@@ -1961,7 +2055,7 @@ impl Rooms {
        let create_event_content = create_event
            .as_ref()
            .map(|create_event| {
-                serde_json::from_value::<Raw<CreateEventContent>>(create_event.content.clone())
+                serde_json::from_str::<Raw<CreateEventContent>>(create_event.content.get())
                    .expect("Raw::from_value always works.")
                    .deserialize()
                    .map_err(|e| {
@@ -2000,7 +2094,10 @@ impl Rooms {
        let mut unsigned = unsigned.unwrap_or_default();
        if let Some(state_key) = &state_key {
            if let Some(prev_pdu) = self.room_state_get(room_id, &event_type, state_key)? {
-                unsigned.insert("prev_content".to_owned(), prev_pdu.content.clone());
+                unsigned.insert(
+                    "prev_content".to_owned(),
+                    serde_json::from_str(prev_pdu.content.get()).expect("string is valid json"),
+                );
                unsigned.insert(
                    "prev_sender".to_owned(),
                    serde_json::to_value(&prev_pdu.sender).expect("UserId::to_value always works"),
@@ -2017,6 +2114,7 @@ impl Rooms {
                .expect("time is valid"),
            kind: event_type,
            content,
+            parsed_content: RwLock::new(None),
            state_key,
            prev_events,
            depth,
@@ -2025,11 +2123,20 @@ impl Rooms {
                .map(|(_, pdu)| pdu.event_id.clone())
                .collect(),
            redacts,
-            unsigned,
+            unsigned: if unsigned.is_empty() {
+                None
+            } else {
+                Some(
+                    RawValue::from_string(
+                        serde_json::to_string(&unsigned).expect("to_string always works"),
+                    )
+                    .expect("string is valid"),
+                )
+            },
            hashes: ruma::events::pdu::EventHash {
                sha256: "aaa".to_owned(),
            },
-            signatures: BTreeMap::new(),
+            signatures: None,
        };

        let auth_check = state_res::auth_check(
@@ -2104,13 +2211,12 @@ impl Rooms {
        // where events in the current room state do not exist
        self.set_room_state(room_id, statehashid)?;

-        for server in self
+        let servers = self
            .room_servers(room_id)
            .filter_map(|r| r.ok())
-            .filter(|server| &**server != db.globals.server_name())
-        {
-            db.sending.send_pdu(&server, &pdu_id)?;
-        }
+            .filter(|server| &**server != db.globals.server_name());
+
+        db.sending.send_pdu(servers, &pdu_id)?;

        for appservice in db.appservice.all()? {
            if self.appservice_in_room(room_id, &appservice, db)? {
@@ -2206,7 +2312,7 @@ impl Rooms {
                let mut pdu = serde_json::from_slice::<PduEvent>(&v)
                    .map_err(|_| Error::bad_database("PDU in db is invalid."))?;
                if pdu.sender != user_id {
-                    pdu.unsigned.remove("transaction_id");
+                    pdu.remove_transaction_id()?;
                }
                Ok((pdu_id, pdu))
            }))
@@ -2243,7 +2349,7 @@ impl Rooms {
                let mut pdu = serde_json::from_slice::<PduEvent>(&v)
                    .map_err(|_| Error::bad_database("PDU in db is invalid."))?;
                if pdu.sender != user_id {
-                    pdu.unsigned.remove("transaction_id");
+                    pdu.remove_transaction_id()?;
                }
                Ok((pdu_id, pdu))
            }))
@@ -2280,7 +2386,7 @@ impl Rooms {
                let mut pdu = serde_json::from_slice::<PduEvent>(&v)
                    .map_err(|_| Error::bad_database("PDU in db is invalid."))?;
                if pdu.sender != user_id {
-                    pdu.unsigned.remove("transaction_id");
+                    pdu.remove_transaction_id()?;
                }
                Ok((pdu_id, pdu))
            }))
@@ -2349,11 +2455,9 @@ impl Rooms {
                    if let Some(predecessor) = self
                        .room_state_get(room_id, &EventType::RoomCreate, "")?
                        .and_then(|create| {
-                            serde_json::from_value::<
-                                Raw<ruma::events::room::create::CreateEventContent>,
-                            >(create.content.clone())
-                            .expect("Raw::from_value always works")
-                            .deserialize()
+                            serde_json::from_str::<ruma::events::room::create::CreateEventContent>(
+                                create.content.get(),
+                            )
                            .ok()
                        })
                        .and_then(|content| content.predecessor)
@@ -2701,17 +2805,15 @@ impl Rooms {
            );
            let state_lock = mutex_state.lock().await;

-            let mut event = serde_json::from_value::<Raw<member::MemberEventContent>>(
+            let mut event = serde_json::from_str::<member::MemberEventContent>(
                self.room_state_get(room_id, &EventType::RoomMember, &user_id.to_string())?
                    .ok_or(Error::BadRequest(
                        ErrorKind::BadState,
                        "Cannot leave a room you are not a member of.",
                    ))?
                    .content
-                    .clone(),
+                    .get(),
            )
-            .expect("from_value::<Raw<..>> can never fail")
-            .deserialize()
            .map_err(|_| Error::bad_database("Invalid member event in database."))?;

            event.membership = member::MembershipState::Leave;
@@ -2719,8 +2821,10 @@ impl Rooms {
            self.build_and_append_pdu(
                PduBuilder {
                    event_type: EventType::RoomMember,
-                    content: serde_json::to_value(event)
-                        .expect("event is valid, we just created it"),
+                    content: RawValue::from_string(
+                        serde_json::to_string(&event).expect("event is valid, we just created it"),
+                    )
+                    .expect("string is valid"),
                    unsigned: None,
                    state_key: Some(user_id.to_string()),
                    redacts: None,
@@ -84,7 +84,7 @@ pub enum SendingEventType {
 pub struct Sending {
    /// The state for a given state hash.
    pub(super) servername_educount: Arc<dyn Tree>, // EduCount: Count of last EDU sync
-    pub(super) servernameevent_data: Arc<dyn Tree>, // ServernamEvent = (+ / $)SenderKey / ServerName / UserId + PduId / Id (for edus), Data = EDU content
+    pub(super) servernameevent_data: Arc<dyn Tree>, // ServernameEvent = (+ / $)SenderKey / ServerName / UserId + PduId / Id (for edus), Data = EDU content
    pub(super) servercurrentevent_data: Arc<dyn Tree>, // ServerCurrentEvents = (+ / $)ServerName / UserId + PduId / Id (for edus), Data = EDU content
    pub(super) maximum_requests: Arc<Semaphore>,
    pub sender: mpsc::UnboundedSender<(Vec<u8>, Vec<u8>)>,
@@ -423,13 +423,23 @@ impl Sending {
        Ok(())
    }

-    #[tracing::instrument(skip(self, server, pdu_id))]
-    pub fn send_pdu(&self, server: &ServerName, pdu_id: &[u8]) -> Result<()> {
-        let mut key = server.as_bytes().to_vec();
-        key.push(0xff);
-        key.extend_from_slice(pdu_id);
-        self.servernameevent_data.insert(&key, &[])?;
-        self.sender.unbounded_send((key, vec![])).unwrap();
+    #[tracing::instrument(skip(self, servers, pdu_id))]
+    pub fn send_pdu<I: Iterator<Item = Box<ServerName>>>(
+        &self,
+        servers: I,
+        pdu_id: &[u8],
+    ) -> Result<()> {
+        let mut batch = servers.map(|server| {
+            let mut key = server.as_bytes().to_vec();
+            key.push(0xff);
+            key.extend_from_slice(pdu_id);
+
+            self.sender.unbounded_send((key.clone(), vec![])).unwrap();
+
+            (key, Vec::new())
+        });
+
+        self.servernameevent_data.insert_batch(&mut batch)?;

        Ok(())
    }
@@ -563,8 +573,14 @@ impl Sending {

                for pdu in pdus {
                    // Redacted events are not notification targets (we don't send push for them)
-                    if pdu.unsigned.get("redacted_because").is_some() {
-                        continue;
+                    if let Some(unsigned) = &pdu.unsigned {
+                        if let Ok(unsigned) =
+                            serde_json::from_str::<serde_json::Value>(unsigned.get())
+                        {
+                            if unsigned.get("redacted_because").is_some() {
+                                continue;
+                            }
+                        }
                    }

                    let userid =
@@ -6,15 +6,15 @@ use ruma::{
        AnySyncStateEvent, EventType, StateEvent,
    },
    serde::{CanonicalJsonObject, CanonicalJsonValue, Raw},
-    state_res, EventId, MilliSecondsSinceUnixEpoch, RoomId, RoomVersionId, ServerName,
-    ServerSigningKeyId, UInt, UserId,
+    state_res, EventId, MilliSecondsSinceUnixEpoch, RoomId, RoomVersionId, ServerName, UInt,
+    UserId,
 };
 use serde::{Deserialize, Serialize};
-use serde_json::json;
-use std::{cmp::Ordering, collections::BTreeMap, convert::TryFrom};
+use serde_json::{json, value::RawValue};
+use std::{cmp::Ordering, collections::BTreeMap, convert::TryFrom, sync::RwLock};
 use tracing::warn;

-#[derive(Clone, Deserialize, Serialize, Debug)]
+#[derive(Deserialize, Serialize, Debug)]
 pub struct PduEvent {
    pub event_id: EventId,
    pub room_id: RoomId,
@@ -22,7 +22,9 @@ pub struct PduEvent {
    pub origin_server_ts: UInt,
    #[serde(rename = "type")]
    pub kind: EventType,
-    pub content: serde_json::Value,
+    pub content: Box<serde_json::value::RawValue>,
+    #[serde(skip)]
+    pub parsed_content: RwLock<Option<serde_json::Value>>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub state_key: Option<String>,
    pub prev_events: Vec<EventId>,
@@ -30,16 +32,17 @@ pub struct PduEvent {
    pub auth_events: Vec<EventId>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub redacts: Option<EventId>,
-    #[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
-    pub unsigned: BTreeMap<String, serde_json::Value>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub unsigned: Option<Box<serde_json::value::RawValue>>,
    pub hashes: EventHash,
-    pub signatures: BTreeMap<Box<ServerName>, BTreeMap<ServerSigningKeyId, String>>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub signatures: Option<Box<serde_json::value::RawValue>>, // BTreeMap<Box<ServerName>, BTreeMap<ServerSigningKeyId, String>>
 }

 impl PduEvent {
    #[tracing::instrument(skip(self))]
    pub fn redact(&mut self, reason: &PduEvent) -> crate::Result<()> {
-        self.unsigned.clear();
+        self.unsigned = None;

        let allowed: &[&str] = match self.kind {
            EventType::RoomMember => &["membership"],
@@ -59,10 +62,9 @@ impl PduEvent {
            _ => &[],
        };

-        let old_content = self
-            .content
-            .as_object_mut()
-            .ok_or_else(|| Error::bad_database("PDU in db has invalid content."))?;
+        let mut old_content =
+            serde_json::from_str::<BTreeMap<String, serde_json::Value>>(self.content.get())
+                .map_err(|_| Error::bad_database("PDU in db has invalid content."))?;

        let mut new_content = serde_json::Map::new();

@@ -72,12 +74,29 @@ impl PduEvent {
            }
        }

-        self.unsigned.insert(
-            "redacted_because".to_owned(),
-            serde_json::to_value(reason).expect("to_value(PduEvent) always works"),
-        );
+        self.unsigned = Some(RawValue::from_string(serde_json::to_string(&json!({
+            "redacted_because": serde_json::to_value(reason).expect("to_value(PduEvent) always works")
+        })).expect("to string always works")).expect("string is valid"));

-        self.content = new_content.into();
+        self.content = RawValue::from_string(
+            serde_json::to_string(&new_content).expect("to string always works"),
+        )
+        .expect("string is valid");
+
+        Ok(())
+    }
+
+    pub fn remove_transaction_id(&mut self) -> crate::Result<()> {
+        if let Some(unsigned) = &self.unsigned {
+            let mut unsigned =
+                serde_json::from_str::<BTreeMap<String, Box<RawValue>>>(unsigned.get())
+                    .map_err(|_| Error::bad_database("Invalid unsigned in pdu event"))?;
+            unsigned.remove("transaction_id");
+            self.unsigned = Some(
+                RawValue::from_string(serde_json::to_string(&unsigned).expect("unsigned is valid"))
+                    .expect("string is valid"),
+            );
+        }

        Ok(())
    }
@@ -265,8 +284,14 @@ impl state_res::Event for PduEvent {
        &self.kind
    }

-    fn content(&self) -> &serde_json::Value {
-        &self.content
+    fn content(&self) -> serde_json::Value {
+        self.parsed_content
+            .write()
+            .unwrap()
+            .get_or_insert_with(|| {
+                serde_json::to_value(&self.content).expect("content is valid json")
+            })
+            .clone()
    }

    fn origin_server_ts(&self) -> MilliSecondsSinceUnixEpoch {
@@ -298,11 +323,17 @@ impl state_res::Event for PduEvent {
    }

    fn signatures(&self) -> BTreeMap<Box<ServerName>, BTreeMap<ruma::ServerSigningKeyId, String>> {
-        self.signatures.clone()
+        self.signatures
+            .as_ref()
+            .map(|raw| serde_json::from_str(raw.get()).expect("string is valid signatures json"))
+            .unwrap_or_default()
    }

-    fn unsigned(&self) -> &BTreeMap<String, serde_json::Value> {
-        &self.unsigned
+    fn unsigned(&self) -> BTreeMap<String, serde_json::Value> {
+        self.unsigned
+            .as_ref()
+            .map(|raw| serde_json::from_str(raw.get()).expect("string is valid json"))
+            .unwrap_or_default()
    }
 }

@@ -352,7 +383,7 @@ pub(crate) fn gen_event_id_canonical_json(
 pub struct PduBuilder {
    #[serde(rename = "type")]
    pub event_type: EventType,
-    pub content: serde_json::Value,
+    pub content: Box<serde_json::value::RawValue>,
    pub unsigned: Option<BTreeMap<String, serde_json::Value>>,
    pub state_key: Option<String>,
    pub redacts: Option<EventId>,
@@ -363,8 +394,12 @@ impl From<AnyInitialStateEvent> for PduBuilder {
    fn from(event: AnyInitialStateEvent) -> Self {
        Self {
            event_type: EventType::from(event.event_type()),
-            content: serde_json::value::to_value(event.content())
-                .expect("AnyStateEventContent came from JSON and can thus turn back into JSON."),
+            content: RawValue::from_string(
+                serde_json::to_string(&event.content()).expect(
+                    "AnyStateEventContent came from JSON and can thus turn back into JSON.",
+                ),
+            )
+            .expect("string is valid"),
            unsigned: None,
            state_key: Some(event.state_key().to_owned()),
            redacts: None,
@@ -39,7 +39,6 @@ use ruma::{
    },
    directory::{IncomingFilter, IncomingRoomNetwork},
    events::{
-        pdu::Pdu,
        receipt::{ReceiptEvent, ReceiptEventContent},
        room::{
            create::CreateEventContent,
@@ -55,7 +54,9 @@ use ruma::{
    uint, EventId, MilliSecondsSinceUnixEpoch, RoomId, RoomVersionId, ServerName,
    ServerSigningKeyId,
 };
+use serde_json::value::RawValue;
 use std::{
+    cell::RefCell,
    collections::{btree_map, hash_map, BTreeMap, BTreeSet, HashMap, HashSet},
    convert::{TryFrom, TryInto},
    fmt::Debug,
@@ -1095,7 +1096,7 @@ pub(crate) async fn handle_incoming_pdu<'a>(
            let start_time = Instant::now();
            let event_id = pdu.event_id.clone();
            if let Err(e) = upgrade_outlier_to_timeline_pdu(
-                pdu,
+                &pdu,
                json,
                &create_event,
                origin,
@@ -1119,7 +1120,7 @@ pub(crate) async fn handle_incoming_pdu<'a>(
    }

    upgrade_outlier_to_timeline_pdu(
-        incoming_pdu,
+        &incoming_pdu,
        val,
        &create_event,
        origin,
@@ -1139,8 +1140,7 @@ fn handle_outlier_pdu<'a>(
    value: BTreeMap<String, CanonicalJsonValue>,
    db: &'a Database,
    pub_key_map: &'a RwLock<BTreeMap<String, BTreeMap<String, String>>>,
-) -> AsyncRecursiveType<'a, StdResult<(Arc<PduEvent>, BTreeMap<String, CanonicalJsonValue>), String>>
-{
+) -> AsyncRecursiveType<'a, StdResult<(PduEvent, BTreeMap<String, CanonicalJsonValue>), String>> {
    Box::pin(async move {
        // TODO: For RoomVersion6 we must check that Raw<..> is canonical do we anywhere?: https://matrix.org/docs/spec/rooms/v6#canonical-json

@@ -1153,14 +1153,13 @@ fn handle_outlier_pdu<'a>(
        // 2. Check signatures, otherwise drop
        // 3. check content hash, redact if doesn't match

-        let create_event_content =
-            serde_json::from_value::<Raw<CreateEventContent>>(create_event.content.clone())
-                .expect("Raw::from_value always works.")
-                .deserialize()
-                .map_err(|e| {
-                    warn!("Invalid create event: {}", e);
-                    "Invalid create event in db.".to_owned()
-                })?;
+        let create_event_content = serde_json::from_str::<CreateEventContent>(
+            create_event.content.get(),
+        )
+        .map_err(|e| {
+            warn!("Invalid create event: {}", e);
+            "Invalid create event in db.".to_owned()
+        })?;

        let room_version_id = &create_event_content.room_version;
        let room_version = RoomVersion::new(room_version_id).expect("room version is supported");
@@ -1241,7 +1240,7 @@ fn handle_outlier_pdu<'a>(
                    .expect("all auth events have state keys"),
            )) {
                hash_map::Entry::Vacant(v) => {
-                    v.insert(auth_event.clone());
+                    v.insert(auth_event);
                }
                hash_map::Entry::Occupied(_) => {
                    return Err(
@@ -1253,11 +1252,7 @@ fn handle_outlier_pdu<'a>(
        }

        // The original create event must be in the auth events
-        if auth_events
-            .get(&(EventType::RoomCreate, "".to_owned()))
-            .map(|a| a.as_ref())
-            != Some(create_event)
-        {
+        if auth_events.get(&(EventType::RoomCreate, "".to_owned())) != Some(create_event) {
            return Err("Incoming event refers to wrong create event.".to_owned());
        }

@@ -1268,7 +1263,7 @@ fn handle_outlier_pdu<'a>(
            db.rooms
                .get_pdu(&incoming_pdu.auth_events[0])
                .map_err(|e| e.to_string())?
-                .filter(|maybe_create| **maybe_create == *create_event)
+                .filter(|maybe_create| maybe_create == create_event)
        } else {
            None
        };
@@ -1276,7 +1271,7 @@ fn handle_outlier_pdu<'a>(
        if !state_res::event_auth::auth_check(
            &room_version,
            &incoming_pdu,
-            previous_create,
+            previous_create.as_ref(),
            None::<PduEvent>, // TODO: third party invite
            |k, s| auth_events.get(&(k.clone(), s.to_owned())),
        )
@@ -1293,13 +1288,13 @@ fn handle_outlier_pdu<'a>(
            .map_err(|_| "Failed to add pdu as outlier.".to_owned())?;
        debug!("Added pdu as outlier.");

-        Ok((Arc::new(incoming_pdu), val))
+        Ok((incoming_pdu, val))
    })
 }

 #[tracing::instrument(skip(incoming_pdu, val, create_event, origin, db, room_id, pub_key_map))]
 async fn upgrade_outlier_to_timeline_pdu(
-    incoming_pdu: Arc<PduEvent>,
+    incoming_pdu: &PduEvent,
    val: BTreeMap<String, CanonicalJsonValue>,
    create_event: &PduEvent,
    origin: &ServerName,
@@ -1320,13 +1315,10 @@ async fn upgrade_outlier_to_timeline_pdu(
    }

    let create_event_content =
-        serde_json::from_value::<Raw<CreateEventContent>>(create_event.content.clone())
-            .expect("Raw::from_value always works.")
-            .deserialize()
-            .map_err(|e| {
-                warn!("Invalid create event: {}", e);
-                "Invalid create event in db.".to_owned()
-            })?;
+        serde_json::from_str::<CreateEventContent>(create_event.content.get()).map_err(|e| {
+            warn!("Invalid create event: {}", e);
+            "Invalid create event in db.".to_owned()
+        })?;

    let room_version_id = &create_event_content.room_version;
    let room_version = RoomVersion::new(room_version_id).expect("room version is supported");
@@ -1434,16 +1426,20 @@ async fn upgrade_outlier_to_timeline_pdu(
                fork_states.push(state);
            }

+            let event_cache = RefCell::new(HashMap::new());
            state_at_incoming_event = match state_res::resolve(
                room_version_id,
                &fork_states,
                auth_chain_sets,
-                |id| {
-                    let res = db.rooms.get_pdu(id);
-                    if let Err(e) = &res {
-                        error!("LOOK AT ME Failed to fetch event: {}", e);
+                |id| match event_cache.borrow_mut().entry(id.clone()) {
+                    hash_map::Entry::Vacant(v) => {
+                        let res = db.rooms.get_pdu(id);
+                        if let Err(e) = &res {
+                            error!("LOOK AT ME Failed to fetch event: {}", e);
+                        }
+                        v.insert(res.ok().flatten().map(Arc::new)).clone()
                    }
-                    res.ok().flatten()
+                    hash_map::Entry::Occupied(o) => o.get().as_ref().map(Arc::clone),
                },
            ) {
                Ok(new_state) => Some(
@@ -1554,7 +1550,7 @@ async fn upgrade_outlier_to_timeline_pdu(
        db.rooms
            .get_pdu(&incoming_pdu.auth_events[0])
            .map_err(|e| e.to_string())?
-            .filter(|maybe_create| **maybe_create == *create_event)
+            .filter(|maybe_create| maybe_create == create_event)
    } else {
        None
    };
@@ -1562,7 +1558,7 @@ async fn upgrade_outlier_to_timeline_pdu(
    let check_result = state_res::event_auth::auth_check(
        &room_version,
        &incoming_pdu,
-        previous_create.as_deref(),
+        previous_create.as_ref(),
        None::<PduEvent>, // TODO: third party invite
        |k, s| {
            db.rooms
@@ -1646,7 +1642,7 @@ async fn upgrade_outlier_to_timeline_pdu(
    let soft_fail = !state_res::event_auth::auth_check(
        &room_version,
        &incoming_pdu,
-        previous_create.as_deref(),
+        previous_create.as_ref(),
        None::<PduEvent>,
        |k, s| auth_events.get(&(k.clone(), s.to_owned())),
    )
@@ -1791,16 +1787,20 @@ async fn upgrade_outlier_to_timeline_pdu(
                .collect::<Result<Vec<_>>>()
                .map_err(|_| "Failed to get_statekey_from_short.".to_owned())?;

+            let event_cache = RefCell::new(HashMap::new());
            let state = match state_res::resolve(
                room_version_id,
                fork_states,
                auth_chain_sets,
-                |id| {
-                    let res = db.rooms.get_pdu(id);
-                    if let Err(e) = &res {
-                        error!("LOOK AT ME Failed to fetch event: {}", e);
+                |id| match event_cache.borrow_mut().entry(id.clone()) {
+                    hash_map::Entry::Vacant(v) => {
+                        let res = db.rooms.get_pdu(id);
+                        if let Err(e) = &res {
+                            error!("LOOK AT ME Failed to fetch event: {}", e);
+                        }
+                        v.insert(res.ok().flatten().map(Arc::new)).clone()
                    }
-                    res.ok().flatten()
+                    hash_map::Entry::Occupied(o) => o.get().as_ref().map(Arc::clone),
                },
            ) {
                Ok(new_state) => new_state,
@@ -1873,7 +1873,7 @@ pub(crate) fn fetch_and_handle_outliers<'a>(
    create_event: &'a PduEvent,
    room_id: &'a RoomId,
    pub_key_map: &'a RwLock<BTreeMap<String, BTreeMap<String, String>>>,
-) -> AsyncRecursiveType<'a, Vec<(Arc<PduEvent>, Option<BTreeMap<String, CanonicalJsonValue>>)>> {
+) -> AsyncRecursiveType<'a, Vec<(PduEvent, Option<BTreeMap<String, CanonicalJsonValue>>)>> {
    Box::pin(async move {
        let back_off = |id| match db.globals.bad_event_ratelimiter.write().unwrap().entry(id) {
            hash_map::Entry::Vacant(e) => {
@@ -2669,13 +2669,10 @@ pub fn create_join_event_template_route(
    let create_event_content = create_event
        .as_ref()
        .map(|create_event| {
-            serde_json::from_value::<Raw<CreateEventContent>>(create_event.content.clone())
-                .expect("Raw::from_value always works.")
-                .deserialize()
-                .map_err(|e| {
-                    warn!("Invalid create event: {}", e);
-                    Error::bad_database("Invalid create event in db.")
-                })
+            serde_json::from_str::<CreateEventContent>(create_event.content.get()).map_err(|e| {
+                warn!("Invalid create event: {}", e);
+                Error::bad_database("Invalid create event in db.")
+            })
        })
        .transpose()?;

@@ -2702,16 +2699,19 @@ pub fn create_join_event_template_route(
        ));
    }

-    let content = serde_json::to_value(MemberEventContent {
-        avatar_url: None,
-        blurhash: None,
-        displayname: None,
-        is_direct: None,
-        membership: MembershipState::Join,
-        third_party_invite: None,
-        reason: None,
-    })
-    .expect("member event is valid value");
+    let content = RawValue::from_string(
+        serde_json::to_string(&MemberEventContent {
+            avatar_url: None,
+            blurhash: None,
+            displayname: None,
+            is_direct: None,
+            membership: MembershipState::Join,
+            third_party_invite: None,
+            reason: None,
+        })
+        .expect("member event is valid value"),
+    )
+    .expect("string is valid");

    let state_key = body.user_id.to_string();
    let kind = EventType::RoomMember;
@@ -2738,7 +2738,7 @@ pub fn create_join_event_template_route(
        unsigned.insert("prev_content".to_owned(), prev_pdu.content.clone());
        unsigned.insert(
            "prev_sender".to_owned(),
-            serde_json::to_value(&prev_pdu.sender).expect("UserId::to_value always works"),
+            serde_json::from_str(prev_pdu.sender.as_str()).expect("UserId is valid string"),
        );
    }

@@ -2751,6 +2751,7 @@ pub fn create_join_event_template_route(
            .expect("time is valid"),
        kind,
        content,
+        parsed_content: RwLock::new(None),
        state_key: Some(state_key),
        prev_events,
        depth,
@@ -2759,17 +2760,26 @@ pub fn create_join_event_template_route(
            .map(|(_, pdu)| pdu.event_id.clone())
            .collect(),
        redacts: None,
-        unsigned,
+        unsigned: if unsigned.is_empty() {
+            None
+        } else {
+            Some(
+                RawValue::from_string(
+                    serde_json::to_string(&unsigned).expect("to_string always works"),
+                )
+                .expect("string is valid"),
+            )
+        },
        hashes: ruma::events::pdu::EventHash {
            sha256: "aaa".to_owned(),
        },
-        signatures: BTreeMap::new(),
+        signatures: None,
    };

    let auth_check = state_res::auth_check(
        &room_version,
        &pdu,
-        create_prev_event.as_deref(),
+        create_prev_event,
        None::<PduEvent>, // TODO: third_party_invite
        |k, s| auth_events.get(&(k.clone(), s.to_owned())),
    )
@@ -2880,14 +2890,13 @@ async fn create_join_event(
        db,
    )?;

-    for server in db
+    let servers = db
        .rooms
        .room_servers(room_id)
        .filter_map(|r| r.ok())
-        .filter(|server| &**server != db.globals.server_name())
-    {
-        db.sending.send_pdu(&server, &pdu_id)?;
-    }
+        .filter(|server| &**server != db.globals.server_name());
+
+    db.sending.send_pdu(servers, &pdu_id)?;

    db.flush()?;

@@ -3280,45 +3289,13 @@ pub(crate) async fn fetch_required_signing_keys(

 // Gets a list of servers for which we don't have the signing key yet. We go over
 // the PDUs and either cache the key or add it to the list that needs to be retrieved.
-fn get_server_keys_from_cache(
-    pdu: &Raw<Pdu>,
+pub fn get_server_keys_from_cache(
+    pdu: &CanonicalJsonObject,
    servers: &mut BTreeMap<Box<ServerName>, BTreeMap<ServerSigningKeyId, QueryCriteria>>,
-    room_version: &RoomVersionId,
    pub_key_map: &mut RwLockWriteGuard<'_, BTreeMap<String, BTreeMap<String, String>>>,
    db: &Database,
 ) -> Result<()> {
-    let value = serde_json::from_str::<CanonicalJsonObject>(pdu.json().get()).map_err(|e| {
-        error!("Invalid PDU in server response: {:?}: {:?}", pdu, e);
-        Error::BadServerResponse("Invalid PDU in server response")
-    })?;
-
-    let event_id = EventId::try_from(&*format!(
-        "${}",
-        ruma::signatures::reference_hash(&value, room_version)
-            .expect("ruma can calculate reference hashes")
-    ))
-    .expect("ruma's reference hashes are valid event ids");
-
-    if let Some((time, tries)) = db
-        .globals
-        .bad_event_ratelimiter
-        .read()
-        .unwrap()
-        .get(&event_id)
-    {
-        // Exponential backoff
-        let mut min_elapsed_duration = Duration::from_secs(30) * (*tries) * (*tries);
-        if min_elapsed_duration > Duration::from_secs(60 * 60 * 24) {
-            min_elapsed_duration = Duration::from_secs(60 * 60 * 24);
-        }
-
-        if time.elapsed() < min_elapsed_duration {
-            debug!("Backing off from {}", event_id);
-            return Err(Error::BadServerResponse("bad event, still backing off"));
-        }
-    }
-
-    let signatures = value
+    let signatures = pdu
        .get("signatures")
        .ok_or(Error::BadServerResponse(
            "No signatures in server response pdu.",
@@ -3370,30 +3347,11 @@ fn get_server_keys_from_cache(
 }

 pub(crate) async fn fetch_join_signing_keys(
-    event: &create_join_event::v2::Response,
-    room_version: &RoomVersionId,
+    servers: Arc<RwLock<BTreeMap<Box<ServerName>, BTreeMap<ServerSigningKeyId, QueryCriteria>>>>,
    pub_key_map: &RwLock<BTreeMap<String, BTreeMap<String, String>>>,
    db: &Database,
 ) -> Result<()> {
-    let mut servers =
-        BTreeMap::<Box<ServerName>, BTreeMap<ServerSigningKeyId, QueryCriteria>>::new();
-
-    {
-        let mut pkm = pub_key_map
-            .write()
-            .map_err(|_| Error::bad_database("RwLock is poisoned."))?;
-
-        // Try to fetch keys, failure is okay
-        // Servers we couldn't find in the cache will be added to `servers`
-        for pdu in &event.room_state.state {
-            let _ = get_server_keys_from_cache(pdu, &mut servers, room_version, &mut pkm, db);
-        }
-        for pdu in &event.room_state.auth_chain {
-            let _ = get_server_keys_from_cache(pdu, &mut servers, room_version, &mut pkm, db);
-        }
-
-        drop(pkm);
-    }
+    let mut servers = std::mem::take(&mut *servers.write().unwrap());

    if servers.is_empty() {
        // We had all keys locally
Author	SHA1	Message	Date
Timo Kösters	bd1172835e	TODO: commit message	2021-10-10 13:51:57 +02:00
Timo Kösters	47f3263396	Merge branch 'gwmngilfen-next-patch-21674' into 'next' Add two flavours of change for the mautrix-signal patch See merge request famedly/conduit!197	2021-09-15 20:31:21 +00:00
Greg Sutcliffe	d38f9b5f01	Move Generic instructions for APPSERVICES above notes for specific bridges and tidy up.	2021-09-15 21:16:25 +01:00
Greg Sutcliffe	fa616342b6	Add two flavours of change for the mautrix-signal patch	2021-09-15 21:16:25 +01:00
Timo Kösters	34fa7fee05	Merge branch 'ci-fix-kaniko-i-hope' into 'next' fix(ci): Convince kaniko that it is indeed running in a container by --force-ing it. See merge request famedly/conduit!205	2021-09-15 07:05:33 +00:00
Jonas Zohren	e8d998cedf	fix(ci): Convince kaniko that it is indeed running in a container by --force-ing it.	2021-09-14 20:44:11 +00:00
Timo Kösters	565fe3a7dd	Merge branch 'faster-sending' into 'next' improvement: batch inserts for inserting pdus See merge request famedly/conduit!204	2021-09-14 18:35:12 +00:00
Timo Kösters	5c02dc7830	improvement: batch inserts for inserting pdus	2021-09-14 14:33:46 +02:00
Timo Kösters	159e22e450	Merge branch 'ci-fix-aarch64-build' into 'next' fix(ci): Fix aarch64 build See merge request famedly/conduit!203	2021-09-14 12:09:44 +00:00
Jonas Zohren	ab472e9b7c	fix(ci): Fix aarch64 build gcc-8-aarch64-linux-gnu is not available in debian 11 (which the rust image uses), so update to gcc-10 Signed-off-by: Jonas Zohren <git-pbkyr@jzohren.de>	2021-09-14 12:32:07 +02:00
Timo Kösters	31264e3d1d	Merge branch 'ci-remove-element-web-register-test' into 'next' Remove the "register an account with element" test See merge request famedly/conduit!202	2021-09-14 09:50:24 +00:00
Jonas Zohren	422bd09e32	Remove the "register an account with element" test Broke due to a timeout and Timo does not like broken tests. Less testing means less failing tests. Also, hopefully sytest is less broken now.	2021-09-14 09:44:15 +00:00
Timo Kösters	b64538c6f1	Merge branch 'efficient-sqlite' into 'next' improvement: more efficient sqlite See merge request famedly/conduit!199	2021-09-14 07:23:02 +00:00
Timo Kösters	178451e9b5	Merge branch 'deduplicate-watchers' into 'next' improvement: deduplicate watchers See merge request famedly/conduit!201	2021-09-14 07:10:59 +00:00
Timo Kösters	583a966f6e	Merge branch 'less-alloc' into 'next' Reduce allocations in server_server See merge request famedly/conduit!190	2021-09-14 07:09:55 +00:00
Timo Kösters	cbee7fe111	improvement: deduplicate watchers	2021-09-13 23:20:30 +02:00
Jonas Platte	d68c93b5fa	Clean up (mostly automated with cargo clippy --fix)	2021-09-13 20:58:58 +02:00
Jonas Platte	979ec6b4fa	Upgrade ruma	2021-09-13 20:55:13 +02:00
Jonas Platte	910ad7fed1	Get rid of more unnecessary intermediate collections	2021-09-13 20:55:12 +02:00
Timo Kösters	2c8412fe58	improvement: more efficient sqlite	2021-09-13 20:12:42 +02:00
Timo Kösters	06b0c9267f	Merge branch 'appservice-signal' into 'next' Add mautrix-signal appservice instructions See merge request famedly/conduit!196	2021-09-13 14:07:18 +00:00
Daniel Wiesenberg	00927a7ce3	Add mautrix-signal appservice instructions	2021-09-09 10:02:11 +02:00
Timo Kösters	4aacafc298	Merge branch 'roomidshort-fix' into 'next' fix sync not firing on new events in room See merge request famedly/conduit!194	2021-09-08 13:39:05 +00:00
Jonathan de Jong	23c5ec8099	fix sync not firing on new events in room	2021-09-08 14:50:44 +02:00
Timo Kösters	562a2524d7	Merge branch 'db-errors' into 'next' fix(database): handle errors in config parsin or database creation Closes #121 See merge request famedly/conduit!193	2021-09-08 09:53:25 +00:00
Rasmus Thomsen	51245d34f1	fix(database): handle errors in config parsin or database creation Showing the user a backtrace can be pretty confusing, so just printing a nice error message makes errors easier to understand for end users. fixes #121	2021-09-08 09:08:45 +01:00
Timo Kösters	4a773a1096	Merge branch 'rm-unused-deps' into 'next' Remove unused dependencies See merge request famedly/conduit!192	2021-09-07 14:00:27 +00:00
Jonas Platte	5821b8e705	Remove unused dependencies	2021-09-07 15:41:01 +02:00
Timo Kösters	0737bc021f	Merge branch 'charludo-next-patch-31617' into 'next' apache2 sample-config error See merge request famedly/conduit!191	2021-09-06 18:38:38 +00:00
charludo	487046571c	These lines get generated by certbot. Having them in the file before running certbot results in an apache2 error, and putting them in afterwards is not necessary, since certbot places them there on its own.	2021-09-06 18:35:35 +00:00
Timo Kösters	2d242523fc	Merge branch 'next' into 'next' Fix deprecated/removed Traefik label See merge request famedly/conduit!189	2021-09-03 20:56:21 +00:00
Daniel Wiesenberg	bbe36810ec	Fix deprecated/removed Traefik label	2021-09-03 21:51:17 +02:00
Timo Kösters	a449b1d73f	Merge branch 'fix-old-events' into 'next' fix: make sure old events don't sneek into the timeline See merge request famedly/conduit!188	2021-09-03 09:47:24 +00:00
Timo Kösters	71341ea05a	fix: make sure old events don't sneek into the timeline	2021-09-03 11:26:15 +02:00
Timo Kösters	73b764322f	Merge branch 'adjust-ci-for-next-and-master-development' into 'master' CI: Adjust CI for master and next branch development model (master) See merge request famedly/conduit!187	2021-09-02 18:25:57 +00:00
Jonas Zohren	a1f51440e2	chore(CI): Adjust CI for master and next branch development model - Build release builds for branches "master" and "next" - Push docker images under different tags, depending on why the pipeline started - branch master: push to `latest` - branch next: push to `next` - tag: push to `$TAG_NAME` Signed-off-by: Jonas Zohren <git-pbkyr@jzohren.de>	2021-09-02 17:17:51 +02:00
Timo Kösters	0d6ae2e1e7	Merge branch 'thanks' into 'master' docs: add thanks to See merge request famedly/conduit!185	2021-09-02 11:10:23 +00:00
Timo Kösters	d0baca44f8	docs: add thanks to	2021-09-02 13:05:19 +02:00
Timo Kösters	5536bfdca5	Merge branch 'ci-also-run-for-tags' into 'master' Also run CI on git tags, not only new commits. See merge request famedly/conduit!184	2021-09-02 07:33:25 +00:00
Jonas Zohren	3357bbec1e	chore: Also run CI on git tags, not only new commits. Signed-off-by: Jonas Zohren <git-pbkyr@jzohren.de>	2021-09-02 09:17:25 +02:00
Timo Kösters	102ff574ea	Merge branch 'readme' into 'master' docs: update readme and version See merge request famedly/conduit!180	2021-09-01 21:28:28 +00:00
Timo Kösters	2770ce2cf6	fix: avoid panic	2021-09-01 23:00:24 +02:00
Timo Kösters	7cd7f1923f	chore: update dependencies	2021-09-01 22:31:25 +02:00
Timo Kösters	43d693ca19	revert Arc<EventId> commit	2021-09-01 22:30:11 +02:00
Timo Kösters	875b345a5f	docs: update readme and version	2021-09-01 22:18:19 +02:00
Timo Kösters	17adf096e9	Merge branch 'server-server-opt' into 'master' Remove unneeded intermediary container collection See merge request famedly/conduit!183	2021-09-01 20:02:37 +00:00
Jonas Platte	487601a249	Remove unneeded intermediary container collection	2021-09-01 21:46:05 +02:00
Timo Kösters	27788af022	Merge branch 'batch_keys' into 'master' Get required keys in batch when joining a room See merge request famedly/conduit!182	2021-09-01 19:12:21 +00:00
Timo Kösters	4b39d7cb64	fix: batch key fetching	2021-09-01 20:02:34 +02:00
Kurt Roeckx	c53d79e287	fixup! Get required keys in batch when joining a room	2021-09-01 19:30:32 +02:00
Kurt Roeckx	984ad5ecd6	fixup! Get required keys in batch when joining a room	2021-09-01 19:30:31 +02:00
Kurt Roeckx	b546a5bf15	Let our server signing be valid for 1 week It was valid for 2 minutes, which can result in the server being asked too much for something that never changes.	2021-09-01 19:30:31 +02:00
Kurt Roeckx	a87519fb71	Get required keys in batch when joining a room We now ask the trusted server for all keys in 1 request, instead of asking each server individual for it's own keys.	2021-09-01 19:30:30 +02:00
Timo Kösters	9c3f1a9272	Merge branch 'master' into 'master' Add Traefik setup, incl. step-by-step, to docker README. See merge request famedly/conduit!158	2021-09-01 16:04:41 +00:00
Timo Kösters	8518d59a0b	Merge branch 'clippy' into 'master' Fix a bunch of clippy lints See merge request famedly/conduit!178	2021-09-01 10:47:23 +00:00
Timo Kösters	4e68c22457	Merge branch 'docs' into 'master' docs: documentation for every endpoint See merge request famedly/conduit!177	2021-09-01 10:20:13 +00:00
Timo Kösters	73d876643c	improvement: make pdu cache capacity configurable	2021-09-01 11:03:45 +02:00
Timo Kösters	22779c21d3	Merge branch 'health_script' into 'master' Fix healthcheck.sh permissions and rearange ci dockerfile See merge request famedly/conduit!179	2021-09-01 08:08:22 +00:00
Daniel Wiesenberg	8c584887c9	Fix healthcheck.sh permissions and rearange ci dockerfile	2021-08-31 23:17:32 +02:00
Timo Kösters	33738dbbc2	improvement: stop prev event fetching if too many events fail	2021-08-31 22:04:47 +02:00
Jonas Platte	a6bb9bbe68	Fix a bunch of clippy lints	2021-08-31 21:20:03 +02:00
Timo Kösters	b9eb39a9c6	docs: documentation for every endpoint	2021-08-31 19:20:34 +02:00
Timo Kösters	364820aae4	Merge branch 'registration-default' into 'master' Default registration true See merge request famedly/conduit!176	2021-08-31 17:03:05 +00:00
Timo Kösters	ec38411620	Merge branch 'health_script' into 'master' Move docker healthcheck into dedicated script. See merge request famedly/conduit!173	2021-08-31 16:59:33 +00:00
Jonathan de Jong	9ec8b7f2b3	registration default true	2021-08-31 18:25:35 +02:00
Daniel Wiesenberg	a08ea15695	Use `$CI_COMMIT_SHORT_SHA` for `GIT_REF` Using `$CI_COMMIT_REF_NAME` means we get `master` for every image build, which is not very useful/informative. Using `$CI_COMMIT_SHORT_SHA`, on the other hand, makes it possible to see exactly from which commit an image was built.	2021-08-31 18:03:44 +02:00
Timo Kösters	49dd3567c6	Merge branch 'connection-lifetime' into 'master' thread_local connections See merge request famedly/conduit!175	2021-08-30 19:05:22 +00:00
Jonathan de Jong	69df9a0145	add tls	2021-08-30 20:18:48 +02:00
Timo Kösters	d0d47675d8	Merge branch 'speed' into 'master' improvement: call /state_ids less often by using state res See merge request famedly/conduit!174	2021-08-30 17:32:41 +00:00
Timo Kösters	bd4ea14a29	improvement: call /state_ids less often by using state res	2021-08-30 16:07:59 +02:00
Timo Kösters	9033cc86aa	Merge branch 'speed' into 'master' fix: make appservices more efficient See merge request famedly/conduit!172	2021-08-30 09:14:08 +00:00
Timo Kösters	1b25e78e3a	fix: inviting dendrite users	2021-08-30 10:56:41 +02:00
Timo Kösters	d57c19802d	improvement: don't do state updates if the event was soft failed	2021-08-30 10:46:36 +02:00
Daniel Wiesenberg	9ded40e983	Change healthcheck in ci dockerfile	2021-08-30 09:14:08 +02:00
Daniel Wiesenberg	a469ca0461	Move docker healthcheck into dedicated script.	2021-08-29 20:01:38 +02:00
Timo Kösters	632a1343eb	fix: make appservices more efficient	2021-08-29 20:00:43 +02:00
Timo Kösters	00c9ad12bd	Merge branch 'speed' into 'master' Speed See merge request famedly/conduit!168	2021-08-28 18:28:25 +00:00
Timo Kösters	1601027605	add warning if calculated event id != requested event id	2021-08-28 19:35:42 +02:00
Timo Kösters	afca61fe7c	fix: don't retry soft failed events	2021-08-28 11:39:33 +02:00
Devin Ragotzy	9bff276fa9	Use Arc<EventId> in place of most EventIds	2021-08-27 20:20:14 +02:00
Timo Kösters	19b89ab91f	fix: server resolution	2021-08-27 20:20:13 +02:00
Timo Kösters	0330d3e270	fix: server resolution with well-known files	2021-08-27 20:20:13 +02:00
Timo Kösters	a1e8a99db5	improvement: less IO for auth chains	2021-08-27 20:20:12 +02:00
Timo Kösters	33172a70e6	fix: improve key fetching	2021-08-27 20:20:12 +02:00
Timo Kösters	1b56a7bbfd	Merge branch 'speed' into 'master' fix: e2ee over federation See merge request famedly/conduit!165	2021-08-26 12:55:47 +00:00
Timo Kösters	d485eb5a24	chore: bump dependencies	2021-08-26 14:38:25 +02:00
Timo Kösters	41d07be97b	improvement: persist cached auth chains in db	2021-08-26 14:19:08 +02:00
Timo Kösters	bef4fe50ce	fix: better migration, force e2ee device key updates	2021-08-26 14:19:08 +02:00
Timo Kösters	9152b877a7	fix: wrong soft fail check, too many events in /sync state response	2021-08-26 14:19:07 +02:00
Timo Kösters	9f8c45c763	fix: e2ee over federation to device events were not being sent	2021-08-26 14:19:07 +02:00
Daniel Wiesenberg	5800e9b797	Add Traefik setup, incl. step-by-step, to docker README.	2021-08-26 12:26:32 +02:00
Timo Kösters	5aa56b92ee	Merge branch 'speed' into 'master' improvement: more efficient auth chain cache See merge request famedly/conduit!164	2021-08-24 20:12:10 +00:00
Timo Kösters	dd87066546	improvement: more efficient auth chain cache	2021-08-24 21:55:18 +02:00
Timo Kösters	30b309b708	Merge branch 'speed' into 'master' Better E2EE and faster incoming event handling See merge request famedly/conduit!163	2021-08-24 18:16:56 +00:00
Timo Kösters	d5695fb9fe	Merge branch 'only-set-max-optimzations-in-ci' into 'master' Only apply max. optimizations in CI builds. See merge request famedly/conduit!162	2021-08-24 17:54:17 +00:00
Timo Kösters	81e056417c	improvement: better e2ee over fed, faster incoming event handling	2021-08-24 19:52:05 +02:00
Timo Kösters	72dd95f500	Merge branch 'createwarnings' into 'master' fix: correct create event warnings See merge request famedly/conduit!161	2021-08-24 17:51:53 +00:00
Jonas Zohren	b6e755f67e	Only apply max. optimizations in CI builds. The average german man has a life expectancy of 78.7 years, or 689884.2 hours. Assuming that Timo is 20 years old, he has rougly 514564.2 hours left on planet earth. Also assuming that cross release builds took him 25 minutes before, but 2-2.5x of that with the current release compilation config he wasted roughly an hour waiting for it to complete. If he continued to work on Conduit for 20 more years (or 175320 hours), and makes a release compilation about once per day, this means 7305 hours or 304 days wasted waiting for the rust compiler. By cutting that back down to the original settings, he get's 182 days of his life back. That's about 0.63% of his remaining life. 182 joyful days he can spend with family and loved ones.	2021-08-22 21:05:32 +00:00
Timo Kösters	4f8cf3bed4	Merge branch 'send_join_v1' into 'master' Implement federation/v1/send_join See merge request famedly/conduit!141	2021-08-22 12:06:26 +00:00
Timo Kösters	b3ea6adc9c	Merge branch 'get-pdu-json' into 'master' Consistent and escaped response in get_pdu See merge request famedly/conduit!160	2021-08-22 11:08:11 +00:00
Timo Kösters	6d83954c40	fix: room upgrades	2021-08-22 13:06:50 +02:00
Timo Kösters	667ffb4239	fix: correct create event warnings	2021-08-22 13:01:13 +02:00
Kurt Roeckx	48494c9464	Implement federation/v1/send_join	2021-08-22 10:07:38 +02:00
Tom Smeding	2ef23b213a	Consistent and escaped response in get_pdu 1. The fallback text of the get_pdu admin room command response message now contains the same text as the formatted_body content (namely, the json instead of Debug-formatting of a serde type). 2. The formatted_body content of the get_pdu response is now html-escaped.	2021-08-21 15:31:26 +02:00
Timo Kösters	33481ec062	Merge branch 'statediffs' into 'master' Statediffs See merge request famedly/conduit!145	2021-08-21 13:05:38 +00:00
Timo Kösters	3b78e43a18	fmt	2021-08-21 14:24:10 +02:00
Timo Kösters	4956fb9fba	improvement: limit prev event fetching	2021-08-21 14:22:21 +02:00
Timo Kösters	b09499c2df	fix: don't save empty tokens	2021-08-19 14:05:23 +02:00
Timo Kösters	46d8a46e1f	improvement: faster incoming transaction handling	2021-08-19 11:01:43 +02:00
Timo Kösters	bf7e019a68	improvement: better prev event fetching, perf improvements	2021-08-17 16:06:09 +02:00
Timo Kösters	75ba8bb565	fix: faster room joins	2021-08-17 00:22:52 +02:00
Timo Kösters	0823506d05	fix: don't load endless prev events and fix room join bug	2021-08-16 23:24:52 +02:00
Timo Kösters	2c3bee34a0	improvement: better sqlite	2021-08-15 13:27:03 +02:00
Timo Kösters	a4310f840e	improvement: state info cache	2021-08-15 06:48:25 +02:00
Timo Kösters	5bd5b41c70	fix: fetch event multiple times	2021-08-15 06:48:25 +02:00
Timo Kösters	f9a2edc0dd	fix: also fetch prev events that are outliers already	2021-08-15 06:48:25 +02:00
Timo Kösters	ecd1e45a44	fix: fetch more than one prev event	2021-08-15 06:48:24 +02:00
Timo Kösters	1d46569929	fix: don't use recursion for prev events	2021-08-15 06:48:24 +02:00
Timo Kösters	1e3a8ca35d	fmt	2021-08-15 06:48:24 +02:00
Timo Kösters	0cb22996be	remove prev event fetch limit	2021-08-15 06:48:14 +02:00
Timo Kösters	38effda799	fix: delta calculation	2021-08-14 19:11:10 +02:00
Timo Kösters	3cf0145bc5	fix: room exists panic	2021-08-14 08:26:45 +02:00
Timo Kösters	3eabaa2a95	finish implementing better state store	2021-08-13 19:49:16 +02:00
Timo Kösters	31f60ad6fd	improvement: migrations, batch inserts	2021-08-12 17:56:35 +02:00
Timo Kösters	41dd620d74	WIP improvement: much better state storage	2021-08-12 17:56:35 +02:00
Timo Kösters	9410d3ef9c	fix: long prev event fetch times for huge rooms	2021-08-12 17:56:14 +02:00
Timo Kösters	665aee11c0	less warnings	2021-08-12 17:56:14 +02:00
Timo Kösters	5173d0deb5	improvement: cache for short event ids	2021-08-12 17:56:14 +02:00
Timo Kösters	c2c6a8673e	improvement: use u64s in auth chain cache	2021-08-12 17:56:14 +02:00
Timo Kösters	096e0971f1	improvement: smaller cache, better prev event fetching	2021-08-12 17:56:13 +02:00