Remove device_one_time_keys_count from is_empty() sync checks, fixing sync issue as reported by Nekron
Closes#115
See merge request famedly/conduit!215
gcc-8-aarch64-linux-gnu is not available in debian 11 (which the rust image uses), so update to gcc-10
Signed-off-by: Jonas Zohren <git-pbkyr@jzohren.de>
- Build release builds for branches "master" and "next"
- Push docker images under different tags, depending on why the pipeline started
- branch master: push to `latest`
- branch next: push to `next`
- tag: push to `$TAG_NAME`
Signed-off-by: Jonas Zohren <git-pbkyr@jzohren.de>
Using `$CI_COMMIT_REF_NAME` means we get `master` for every image build,
which is not very useful/informative. Using `$CI_COMMIT_SHORT_SHA`,
on the other hand, makes it possible to see exactly from which commit an
image was built.
The average german man has a life expectancy of 78.7 years, or 689884.2 hours.
Assuming that Timo is 20 years old, he has rougly 514564.2 hours left on planet earth.
Also assuming that cross release builds took him 25 minutes before,
but 2-2.5x of that with the current release compilation config he wasted roughly an hour waiting for it to complete.
If he continued to work on Conduit for 20 more years (or 175320 hours),
and makes a release compilation about once per day, this means 7305 hours or 304 days wasted waiting for the rust compiler.
By cutting that back down to the original settings, he get's 182 days of his life back.
That's about 0.63% of his remaining life.
182 joyful days he can spend with family and loved ones.
1. The fallback text of the get_pdu admin room command response message
now contains the same text as the formatted_body content (namely, the
json instead of Debug-formatting of a serde type).
2. The formatted_body content of the get_pdu response is now
html-escaped.
The spec does not require servers to apply a room canonical alias event upon room creation (yet).
Still, synapse does that, since users can set their desired alias in Elements room creation dialog.
With this commit, conduit also sets that alias if it is a valid one.
This closes https://gitlab.com/famedly/conduit/-/issues/123
4 years ago
74 changed files with 6494 additions and 3717 deletions
If you run into any problems while setting up an Appservice, write an email to `timo@koesters.xyz`, ask us in `#conduit:matrix.org` or [open an issue on GitLab](https://gitlab.com/famedly/conduit/-/issues/new).
## Tested appservices
Here are some appservices we tested and that work with Conduit:
- matrix-appservice-discord
- mautrix-hangouts
- mautrix-telegram
## Set up the appservice
## Set up the appservice - general instructions
Follow whatever instructions are given by the appservice. This usually includes
downloading, changing its config (setting domain, homeserver url, port etc.)
@ -46,3 +39,48 @@ Then you are done. Conduit will send messages to the appservices and the
@@ -46,3 +39,48 @@ Then you are done. Conduit will send messages to the appservices and the
appservice can send requests to the homeserver. You don't need to restart
Conduit, but if it doesn't work, restarting while the appservice is running
could help.
## Appservice-specific instructions
### Tested appservices
These appservices have been tested and work with Conduit without any extra steps:
--- levels.users[self.main_intent.mxid] = 9001 if is_initial else 100
+++ levels.users[self.main_intent.mxid] = 100 if is_initial else 100
```
- [Between lines 1041 and 1042](https://github.com/mautrix/signal/blob/4ea831536f154aba6419d13292479eb383ea3308/mautrix_signal/portal.py#L1041-L1042) add a new line:
```diff
"type": str(EventType.ROOM_POWER_LEVELS),
+++ "state_key": "",
"content": power_levels.serialize(),
```
3. **Deploy the patch**. This is different depending on how you have `mautrix-signal` deployed:
- [*If using virtualenv*] Copy your patched `portal.py` to `./lib/python3.7/site-packages/mautrix_signal/portal.py` (the exact version of Python may be different on your system).
- [*If using Docker*] Map the patched `portal.py` into the `mautrix-signal` container:
If you run into any problems while setting up Conduit, write an email to `timo@koesters.xyz`, ask us in `#conduit:matrix.org` or [open an issue on GitLab](https://gitlab.com/famedly/conduit/-/issues/new).
If you run into any problems while setting up Conduit, write an email to `timo@koesters.xyz`, ask us
in `#conduit:matrix.org` or [open an issue on GitLab](https://gitlab.com/famedly/conduit/-/issues/new).
## Installing Conduit
You may simply download the binary that fits your machine. Run `uname -m` to see
what you need. Now copy the right url:
Although you might be able to compile Conduit for Windows, we do recommend running it on a linux server. We therefore
only offer Linux binaries.
| CPU Architecture | GNU (Ubuntu, Debian, ArchLinux, ...) | MUSL (Alpine, ... ) |
Alternatively, you may compile the binary yourself using
```bash
$ cargo build --release
```
Note that this currently requires Rust 1.50.
If you want to cross compile Conduit to another architecture, read the [Cross-Compile Guide](CROSS_COMPILE.md).
## Adding a Conduit user
While Conduit can run as any user it is usually better to use dedicated users for different services.
This also allows you to make sure that the file permissions are correctly set up.
While Conduit can run as any user it is usually better to use dedicated users for different services. This also allows
you to make sure that the file permissions are correctly set up.
In Debian you can use this command to create a Conduit user:
```
```bash
sudo adduser --system conduit --no-create-home
```
## Setting up a systemd service
Now we'll set up a systemd service for Conduit, so it's easy to start/stop
Conduit and set it to autostart when your server reboots. Simply paste the
default systemd service you can find below into
Now we'll set up a systemd service for Conduit, so it's easy to start/stop Conduit and set it to autostart when your
server reboots. Simply paste the default systemd service you can find below into
`/etc/systemd/system/conduit.service`.
```systemd
@ -74,10 +78,10 @@ Finally, run
@@ -74,10 +78,10 @@ Finally, run
$ sudo systemctl daemon-reload
```
## Creating the Conduit configuration file
Now we need to create the Conduit's config file in `/etc/matrix-conduit/conduit.toml`. Paste this in **and take a moment to read it. You need to change at least the server name.**
Now we need to create the Conduit's config file in `/etc/matrix-conduit/conduit.toml`. Paste this in **and take a moment
to read it. You need to change at least the server name.**
```toml
[global]
@ -104,8 +108,8 @@ port = 6167
@@ -104,8 +108,8 @@ port = 6167
# Max size for uploads
max_request_size = 20_000_000 # in bytes
# Disabling registration means no new users will be able to register on this server
allow_registration = false
# Enables registration. If set to false, no users can register on this server.
allow_registration = true
# Disable encryption, so no new encrypted rooms can be created
# Note: existing rooms will continue to work
@ -125,21 +129,20 @@ address = "127.0.0.1" # This makes sure Conduit can only be reached using the re
@@ -125,21 +129,20 @@ address = "127.0.0.1" # This makes sure Conduit can only be reached using the re
## Setting the correct file permissions
As we are using a Conduit specific user we need to allow it to read the config.
To do that you can run this command on Debian:
As we are using a Conduit specific user we need to allow it to read the config. To do that you can run this command on
Debian:
```
```bash
sudo chown -R conduit:nogroup /etc/matrix-conduit
```
If you use the default database path you also need to run this:
This depends on whether you use Apache, Nginx or another web server.
@ -148,7 +151,7 @@ This depends on whether you use Apache, Nginx or another web server.
@@ -148,7 +151,7 @@ This depends on whether you use Apache, Nginx or another web server.
Create `/etc/apache2/sites-enabled/050-conduit.conf` and copy-and-paste this:
If you use Nginx and not Apache, add the following server section inside the
http section of `/etc/nginx/nginx.conf`
If you use Nginx and not Apache, add the following server section inside the http section of `/etc/nginx/nginx.conf`
```
```nginx
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
@ -198,13 +196,13 @@ server {
@@ -198,13 +196,13 @@ server {
include /etc/letsencrypt/options-ssl-nginx.conf;
}
```
**You need to make some edits again.** When you are done, run
```bash
$ sudo systemctl reload nginx
```
## SSL Certificate
The easiest way to get an SSL certificate, if you don't have one already, is to install `certbot` and run this:
@ -213,7 +211,6 @@ The easiest way to get an SSL certificate, if you don't have one already, is to
@@ -213,7 +211,6 @@ The easiest way to get an SSL certificate, if you don't have one already, is to
$ sudo certbot -d your.server.name
```
## You're done!
Now you can start Conduit with:
@ -228,4 +225,15 @@ Set it to start automatically when your system boots with:
@@ -228,4 +225,15 @@ Set it to start automatically when your system boots with:
$ sudo systemctl enable conduit
```
## How do I know it works?
You can open <https://app.element.io>, enter your homeserver and try to register.
You can also use these commands as a quick health check.
A fast Matrix homeserver that's easy to set up and just works. You can install
An efficient Matrix homeserver that's easy to set up and just works. You can install
it on a mini-computer like the Raspberry Pi to host Matrix for your family,
friends or company.
#### Can I try it out?
Yes! Just open a Matrix client (<https://app.element.io> or Element Android for
example) and register on the `https://conduit.koesters.xyz` homeserver.
Yes! You can test our Conduit instance by opening a Matrix client (<https://app.element.io> or Element Android for
example) and registering on the `conduit.rs` homeserver.
#### What is it built on?
- [Ruma](https://www.ruma.io): Useful structures for endpoint requests and
responses that can be (de)serialized
- [Sled](https://github.com/spacejam/sled): A simple (key, value) database with
good performance
- [Rocket](https://rocket.rs): A flexible web framework
It is hosted on a ODROID HC 2 with 2GB RAM and a SAMSUNG Exynos 5422 CPU, which
was used in the Samsung Galaxy S5. It joined many big rooms including Matrix
HQ.
#### What is the current status?
Conduit can already be used chat with other users on Conduit, chat with users
from other Matrix servers and even to chat with users on other platforms using
appservices. When chatting with users on the same Conduit server, everything
should work assuming you use a compatible client.
**You should not join Matrix rooms without asking the admins first.** We do not
know whether Conduit is safe for general use yet, so you should assume there is
some chance that it breaks rooms permanently for all participating users. We
are not aware of such a bug today, but we would like to do more testing.
As of 2021-09-01, Conduit is Beta, meaning you can join and participate in most
Matrix rooms, but not all features are supported and you might run into bugs
from time to time.
There are still a few important features missing:
- Database stability (currently you might have to do manual upgrades or even wipe the db for new versions)
- Edge cases for end-to-end encryption over federation
- Typing and presence over federation
- Lots of testing
- E2EE verification over federation
- Outgoing read receipts, typing, presence over federation
Check out the [Conduit 1.0 Release Milestone](https://gitlab.com/famedly/conduit/-/milestones/3).
#### How can I deploy my own?
##### Deploy
Download or compile a Conduit binary, set up the config and call it from somewhere like a systemd script. [Read
more](DEPLOY.md)
If you want to connect an Appservice to Conduit, take a look at the [Appservice Guide](APPSERVICES.md).
##### Deploy using a Debian package
You need to have the `deb` helper command installed that creates Debian packages from Cargo projects (see [cargo-deb](https://github.com/mmstick/cargo-deb/) for more info):
```shell
$ cargo install cargo-deb
```
Then, you can create and install a Debian package at a whim:
Note that `cargo deb` supports [cross-compilation](https://github.com/mmstick/cargo-deb/#cross-compilation) too!
Official Debian packages will follow once Conduit starts to have stable releases.
##### Deploy using Docker
Pull and run the docker image with
``` bash
docker pull matrixconduit/matrix-conduit:latest
docker run -d -p 8448:8000 -v ~/conduit.toml:/srv/conduit/conduit.toml -v db:/srv/conduit/.local/share/conduit matrixconduit/matrix-conduit:latest
```
> <b>Note:</b> You also need to supply a `conduit.toml` config file, you can find an example [here](./conduit-example.toml).
> Or you can pass in `-e CONDUIT_CONFIG=""` and configure Conduit purely with env vars.
Or build and run it with docker or docker-compose. [Read more](docker/README.md)
If you want to connect an Appservice to Conduit, take a look at [APPSERVICES.md](APPSERVICES.md).
#### How can I contribute?
@ -98,6 +50,17 @@ Or build and run it with docker or docker-compose. [Read more](docker/README.md)
@@ -98,6 +50,17 @@ Or build and run it with docker or docker-compose. [Read more](docker/README.md)
3. Fork the repo and work on the issue. #conduit:nordgedanken.dev is happy to help :)
4. Submit a MR
#### Thanks to
Thanks to Famedly, Prototype Fund (DLR and German BMBF) and all other individuals for financially supporting this project.
Thanks to the contributors to Conduit and all libraries we use, for example:
- Ruma: A clean library for the Matrix Spec in Rust
> **Note:** To run and use Conduit you should probably use it with a Domain or Subdomain behind a reverse proxy (like Nginx, Traefik, Apache, ...) with a Lets Encrypt certificate.
## Docker
### Build & Dockerfile
The Dockerfile provided by Conduit has two stages, each of which creates an image.
1. **Builder:** Builds the binary from local context or by cloning a git revision from the official repository.
2. **Runtime:** Copies the built binary from **Builder** and sets up the runtime environment, like creating a volume to persist the database and applying the correct permissions.
The Dockerfile includes a few build arguments that should be supplied when building it.
``` Dockerfile
ARG LOCAL=false
ARG CREATED
ARG VERSION
ARG GIT_REF=origin/master
```
- **CREATED:** Date and time as string (date-time as defined by RFC 3339). Will be used to create the Open Container Initiative compliant label `org.opencontainers.image.created`. Supply by it like this `$(date -u +'%Y-%m-%dT%H:%M:%SZ')`
- **VERSION:** The SemVer version of Conduit, which is in the image. Will be used to create the Open Container Initiative compliant label `org.opencontainers.image.version`. If you have a `Cargo.toml` in your build context, you can get it with `$(grep -m1 -o '[0-9].[0-9].[0-9]' Cargo.toml)`
- **LOCAL:***(Optional)* A boolean value, specifies if the local build context should be used, or if the official repository will be cloned. If not supplied with the build command, it will default to `false`.
- **GIT_REF:***(Optional)* A git ref, like `HEAD` or a commit ID. The supplied ref will be used to create the Open Container Initiative compliant label `org.opencontainers.image.revision` and will be the ref that is cloned from the repository when not building from the local context. If not supplied with the build command, it will default to `origin/master`.
1. **Builder:** Builds the binary from local context or by cloning a git revision from the official repository.
2. **Runner:** Copies the built binary from **Builder** and sets up the runtime environment, like creating a volume to persist the database and applying the correct permissions.
To build the image you can use the following command
which also will tag the resulting image as `matrixconduit/matrix-conduit:latest`.
**Note:** it ommits the two optional `build-arg`s.
### Run
After building the image you can simply run it with
```bash
docker run -d -p 8448:8000 -v ~/conduit.toml:/srv/conduit/conduit.toml -v db:/srv/conduit/.local/share/conduit matrixconduit/matrix-conduit:latest
```bash
docker run -d -p 8448:6167 -v ~/conduit.toml:/srv/conduit/conduit.toml -v db:/srv/conduit/.local/share/conduit matrixconduit/matrix-conduit:latest
```
For detached mode, you also need to use the `-d` flag. You also need to supply a `conduit.toml` config file, you can find an example [here](../conduit-example.toml).
or you can skip the build step and pull the image from one of the following registries:
The `-d` flag lets the container run in detached mode. You now need to supply a `conduit.toml` config file, an example can be found [here](../conduit-example.toml).
You can pass in different env vars to change config values on the fly. You can even configure Conduit completely by using env vars, but for that you need
too pass `-e CONDUIT_CONFIG=""` into your container. For an overview of possible values, please take a look at the `docker-compose.yml` file.
If you just want to test Conduit for a short time, you can use the `--rm` flag, which will clean up everything related to your container after you stop it.
to pass `-e CONDUIT_CONFIG=""` into your container. For an overview of possible values, please take a look at the `docker-compose.yml` file.
If you just want to test Conduit for a short time, you can use the `--rm` flag, which will clean up everything related to your container after you stop it.
## Docker-compose
If the docker command is not for you or your setup, you can also use one of the provided `docker-compose` files. Depending on your proxy setup, use the [`docker-compose.traefik.yml`](docker-compose.traefik.yml) including [`docker-compose.override.traefik.yml`](docker-compose.override.traefik.yml) or the normal [`docker-compose.yml`](../docker-compose.yml) for every other reverse proxy.
If the docker command is not for you or your setup, you can also use one of the provided `docker-compose` files. Depending on your proxy setup, use the [`docker-compose.traefik.yml`](docker-compose.traefik.yml) and [`docker-compose.override.traefik.yml`](docker-compose.override.traefik.yml) for Traefik (don't forget to remove `.traefik` from the filenames) or the normal [`docker-compose.yml`](../docker-compose.yml) for every other reverse proxy. Additional info about deploying
Conduit can be found [here](../DEPLOY.md).
### Build
To build the Conduit image with docker-compose, you first need to open and modify the `docker-compose.yml` file. There you need to comment the `image:` option and uncomment the `build:` option. Then call docker-compose with:
```bash
CREATED=$(date -u +'%Y-%m-%dT%H:%M:%SZ') VERSION=$(grep -m1 -o '[0-9].[0-9].[0-9]' Cargo.toml) docker-compose up
```bash
docker-compose up
```
This will also start the container right afterwards, so if want it to run in detached mode, you also should use the `-d` flag. For possible `build-args`, please take a look at the above `Build & Dockerfile` section.
This will also start the container right afterwards, so if want it to run in detached mode, you also should use the `-d` flag.
### Run
If you already have built the image, you can just start the container and everything else in the compose file in detached mode with:
If you already have built the image or want to use one from the registries, you can just start the container and everything else in the compose file in detached mode with:
```bash
```bash
docker-compose up -d
```
> **Note:** Don't forget to modify and adjust the compose file to your needs.
### Use Traefik as Proxy
As a container user, you probably know about Traefik. It is a easy to use reverse proxy for making containerized app and services available through the web. With the
two provided files, [`docker-compose.traefik.yml`](docker-compose.traefik.yml) and [`docker-compose.override.traefik.yml`](docker-compose.override.traefik.yml), it is
equally easy to deploy and use Conduit, with a little caveat. If you already took a look at the files, then you should have seen the `well-known` service, and that is
the little caveat. Traefik is simply a proxy and loadbalancer and is not able to serve any kind of content, but for Conduit to federate, we need to either expose ports
`443` and `8448` or serve two endpoints `.well-known/matrix/client` and `.well-known/matrix/server`.
With the service `well-known` we use a single `nginx` container that will serve those two files.
So...step by step:
1. Copy [`docker-compose.traefik.yml`](docker-compose.traefik.yml) and [`docker-compose.override.traefik.yml`](docker-compose.override.traefik.yml) from the repository and remove `.traefik` from the filenames.
2. Open both files and modify/adjust them to your needs. Meaning, change the `CONDUIT_SERVER_NAME` and the volume host mappings according to your needs.
3. Create the `conduit.toml` config file, an example can be found [here](../conduit-example.toml), or set `CONDUIT_CONFIG=""` and configure Conduit per env vars.
4. Uncomment the `element-web` service if you want to host your own Element Web Client and create a `element_config.json`.
5. Create the files needed by the `well-known` service.
- `./nginx/matrix.conf` (relative to the compose file, you can change this, but then also need to change the volume mapping)
```nginx
server {
server_name <SUBDOMAIN>.<DOMAIN>;
listen 80 default_server;
location /.well-known/matrix/ {
root /var/www;
default_type application/json;
add_header Access-Control-Allow-Origin *;
}
}
```
- `./nginx/www/.well-known/matrix/client` (relative to the compose file, you can change this, but then also need to change the volume mapping)
```json
{
"m.homeserver": {
"base_url": "https://<SUBDOMAIN>.<DOMAIN>"
}
}
```
- `./nginx/www/.well-known/matrix/server` (relative to the compose file, you can change this, but then also need to change the volume mapping)
```json
{
"m.server": "<SUBDOMAIN>.<DOMAIN>:443"
}
```
6. Run `docker-compose up -d`
7. Connect to your homeserver with your preferred client and create a user. You should do this immediatly after starting Conduit, because the first created user is the admin.
"Thanks for trying out Conduit! This software is still in development, so expect many bugs and missing features. If you have federation enabled, you can join the Conduit chat room by typing `/join #conduit:matrix.org`. **Important: Please don't join any other Matrix rooms over federation without permission from the room's admins.** Some actions might trigger bugs in other server implementations, breaking the chat for everyone else.".to_owned(),
"Thanks for trying out Conduit! This software is still in development, so expect many bugs and missing features. If you have federation enabled, you can join the Conduit chat room by typing <code>/join #conduit:matrix.org</code>. <strong>Important: Please don't join any other Matrix rooms over federation without permission from the room's admins.</strong> Some actions might trigger bugs in other server implementations, breaking the chat for everyone else.".to_owned(),
"## Thank you for trying out Conduit!\n\nConduit is currently in Beta. This means you can join and participate in most Matrix rooms, but not all features are supported and you might run into bugs from time to time.\n\nHelpful links:\n> Website: https://conduit.rs\n> Git and Documentation: https://gitlab.com/famedly/conduit\n> Report issues: https://gitlab.com/famedly/conduit/-/issues\n\nHere are some rooms you can join (by typing the command):\n\nConduit room (Ask questions and get notified on updates):\n`/join #conduit:fachschaften.org`\n\nConduit lounge (Off-topic, only Conduit users are allowed to join)\n`/join #conduit-lounge:conduit.rs`".to_owned(),
"<h2>Thank you for trying out Conduit!</h2>\n<p>Conduit is currently in Beta. This means you can join and participate in most Matrix rooms, but not all features are supported and you might run into bugs from time to time.</p>\n<p>Helpful links:</p>\n<blockquote>\n<p>Website: https://conduit.rs<br>Git and Documentation: https://gitlab.com/famedly/conduit<br>Report issues: https://gitlab.com/famedly/conduit/-/issues</p>\n</blockquote>\n<p>Here are some rooms you can join (by typing the command):</p>\n<p>Conduit room (Ask questions and get notified on updates):<br><code>/join #conduit:fachschaften.org</code></p>\n<p>Conduit lounge (Off-topic, only Conduit users are allowed to join)<br><code>/join #conduit-lounge:conduit.rs</code></p>\n".to_owned(),
.map_err(|_|Error::BadConfig("Database folder doesn't exists and couldn't be created (e.g. due to missing permissions). Please create the database folder yourself."))?;
}
letbuilder=Engine::open(&config)?;
letbuilder=Engine::open(config)?;
ifconfig.max_request_size<1024{
eprintln!("ERROR: Max request size is less than 1KB. Please increase it.");
// Empty prev id forces synapse to resync: https://github.com/matrix-org/synapse/blob/98aec1cc9da2bd6b8e34ffb282c85abf9b8b42ca/synapse/handlers/device.py#L767
// Because synapse resyncs, we can just insert dummy data
Timo Kösters
Jonas Platte
Jonas Zohren
Moritz Bitsch
Jonas Platte
Nyaaori
Andrew Morgan
Luc-pascal Ceccaldi
Greg Sutcliffe
Jonas Zohren
Daniel Wiesenberg
Jonathan de Jong
Rasmus Thomsen
charludo
Kurt Roeckx
Devin Ragotzy
Tom Smeding