From b417f466c8c13632e2fddbfa22dd801c92632168 Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Fri, 31 Jul 2020 02:00:53 +0000 Subject: [PATCH 01/15] Resolve "Stricter TLS requirements for iOS" --- README.md | 8 ++++++++ scripts/install.sh | 4 ++-- scripts/subinstallers/openssl.sh | 10 +++++++--- 3 files changed, 17 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index e210492..6676760 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,9 @@ Tested Operating Systems (in order of thoroughness): - Debian 10 (Buster) - Raspbian Buster +Tested Browsers: +- Firefox + The default tests are run on DigitalOcean Droplets: 2GB RAM, 1 CPU, 50GB hard disk. The Mistborn docker images exist for these architectures: @@ -265,6 +268,8 @@ These are the default credentials to use in the services you choose to use: | Cockpit | cockpit | {{default mistborn password}} | | Nextcloud | mistborn | {{default mistborn password}} | +You can find the credentials sent to the Docker containers in: `/opt/mistborn/.envs/.production/` + # Gateway Setup Mistborn will generate the Wireguard configuration script for the Gateway. From a base Ubuntu/Debian/Raspbian operating system the following packages are recommended to be installed beforehand: @@ -457,6 +462,9 @@ Many features and refinements are in the works at various stages including: - Internal network scan tool and feedback - Anomaly detection in network traffic +# Follow +You can find recent bugfixes, functional additions, some extra documentation and more at the Cyber5K Patreon page: [https://www.patreon.com/cyber5k](https://www.patreon.com/cyber5k) + # Contact Contact me at [steven@cyber5k.com](mailto:steven@cyber5k.com) diff --git a/scripts/install.sh b/scripts/install.sh index d71a6d4..2a5c4ce 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -209,8 +209,8 @@ cp ./compose/production/traefik/traefik.toml.template ./compose/production/traef # setup tls certs source ./scripts/subinstallers/openssl.sh -sudo rm -rf ../mistborn_volumes/base/tls -sudo mv ./tls ../mistborn_volumes/base/ +#sudo rm -rf ../mistborn_volumes/base/tls +#sudo mv ./tls ../mistborn_volumes/base/ # enable and run setup to generate .env sudo systemctl enable Mistborn-setup.service diff --git a/scripts/subinstallers/openssl.sh b/scripts/subinstallers/openssl.sh index c5d58eb..6862cfc 100755 --- a/scripts/subinstallers/openssl.sh +++ b/scripts/subinstallers/openssl.sh @@ -1,6 +1,6 @@ #!/bin/bash -KEY_FOLDER="./tls/" +KEY_FOLDER="/opt/mistborn_volumes/base/tls/" CRT_FILE="cert.crt" KEY_FILE="cert.key" @@ -11,11 +11,15 @@ KEY_PATH="$KEY_FOLDER/$KEY_FILE" sudo -E apt-get install -y openssl # make folder -mkdir -p $KEY_FOLDER +sudo -E mkdir -p $KEY_FOLDER + +# clean old crt and key +sudo -E rm -f ${KEY_FOLDER}/* # generate crt and key -openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:4096 -keyout $KEY_PATH -out $CRT_PATH -subj "/C=US/ST=New York/L=New York/O=cyber5k/OU=mistborn/CN=*.mistborn/emailAddress=mistborn@localhost" +sudo -E openssl req -x509 -sha256 -nodes -days 397 -newkey rsa:4096 -keyout $KEY_PATH -out $CRT_PATH -addext "subjectAltName = DNS:*.mistborn" -addext extendedKeyUsage=serverAuth -subj "/C=US/ST=New York/L=New York/O=cyber5k/OU=mistborn/CN=*.mistborn/emailAddress=mistborn@localhost" # set permissions +sudo -E chown -R mistborn:mistborn ${KEY_FOLDER} chmod 644 $CRT_PATH chmod 600 $KEY_PATH From 2ac26e9a3438b826c18564d3cd422bb861b4f65c Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Tue, 4 Aug 2020 01:20:33 +0000 Subject: [PATCH 02/15] Resolve "Raspberry Pi OS bugs" --- README.md | 16 +++++++++------- scripts/install.sh | 6 +++--- scripts/subinstallers/cockpit.sh | 5 +---- scripts/subinstallers/docker_manual.sh | 8 ++++---- scripts/subinstallers/wireguard.sh | 5 +++-- 5 files changed, 20 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 6676760..50ffc3d 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,9 @@ Tested Operating Systems (in order of thoroughness): - Ubuntu 20.04 LTS - Ubuntu 18.04 LTS - Debian 10 (Buster) -- Raspbian Buster +- Raspberry Pi OS (formerly Raspbian) Buster + +**Note:** Install operating system updates and restart. Raspberry Pi OS particularly needs to be restarted after kernel updates (kernel modules for the currently running kernel may be missing). Tested Browsers: - Firefox @@ -59,8 +61,8 @@ The Mistborn docker images exist for these architectures: | Mistborn Docker Images (hub.docker.com) | Architectures | |------------------------------------------------|---------------------| -| mistborn (django, celery{worker,beat}, flower) | amd64, arm64, armv7 | -| dnscrypt-proxy | amd64, arm64, armv7 | +| mistborn (django, celery{worker,beat}, flower) | amd64, arm64, arm/v7 | +| dnscrypt-proxy | amd64, arm64, arm/v7 | Recommended System Specifications: @@ -132,8 +134,8 @@ Running `install.sh` will do the following: - install Docker - install OpenSSH - install Wireguard -- install Cockpit -- create a `cockpit` system user +- install Cockpit (optional) +- create a `cockpit` system user (if Cockpit is installed) - configure unattended-upgrades - generate a self-signed TLS certificate/key (WebRTC functionality requires TLS) - create and populate traefik.toml @@ -317,7 +319,7 @@ But wait, there's more! You can: | Rocket.Chat | [Rocket.Chat](https://play.google.com/store/apps/details?id=chat.rocket.android) | [Rocket.Chat](https://apps.apple.com/us/app/rocket-chat/id1148741252) | ## TLS Certificate -Some apps require TLS (HTTPS). All traffic to Mistborn domains already occurs over Wireguard but to keep apps running, a TLS certificate exists for Mistborn and can be imported into your device's trusted credentials in the security settings. +Some apps require TLS (HTTPS). All traffic to Mistborn domains already occurs over Wireguard but to keep apps running, a TLS certificate exists for Mistborn and can be imported into your device's trusted credentials in the security settings. This certificate is checked every day and will be re-generated when expiration is less than 30 days away. The TLS certificate can be found here: ``` @@ -446,7 +448,7 @@ These are some notes regarding the technical design and implementations of Mistb ## Additonal Notes - Interface names are not hardcoded anywhere in Mistborn. Two commands that are used in different circumstances to determine the default network interface and the interface that would route a public IP address are: `ip -o -4 route show to default` and `ip -o -4 route get 1.1.1.1`. - The "Update" button will pull updated Docker images for mistborn, postgresql, redis, pihole, and dnscrypt. Those services will then be restarted. -- The generated TLS certificate has an RSA modulus of 4096 bits, is signed with SHA-256, and is good for 10 years. The nanny at Apple has decided to restrict the kinds of certificates iOS users may choose to manually trust and so you may have issues with TLS on an Apple device for now. +- The generated TLS certificate has an RSA modulus of 4096 bits, is signed with SHA-256, and is good for 397 days. The certificate is checked daily and will regenerate when expiration is within 30 days. - Outbound UDP on port 53 is blocked. All DNS requests should be handled by the dnscrypt_proxy service and if any client, service, etc. tries to circumvent that it is blocked. - Unattended upgrades are set to automatically install operating system security updates. diff --git a/scripts/install.sh b/scripts/install.sh index 2a5c4ce..a904d7c 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -69,9 +69,9 @@ fi # Install Cockpit? if [ -z "${MISTBORN_INSTALL_COCKPIT}" ]; then - read -p "Install Cockpit (a somewhat resource-heavy system management graphical user interface)? [Y/n]: " MISTBORN_INSTALL_COCKPIT + read -p "Install Cockpit (a somewhat resource-heavy system management graphical user interface -- NOT RECOMMENDED on Raspberry Pi)? [y/N]: " MISTBORN_INSTALL_COCKPIT echo - MISTBORN_INSTALL_COCKPIT=${MISTBORN_INSTALL_COCKPIT:-Y} + MISTBORN_INSTALL_COCKPIT=${MISTBORN_INSTALL_COCKPIT:-N} fi # SSH keys @@ -132,7 +132,7 @@ sudo -E apt-get install -y dnsutils fail2ban # Install kernel headers if [ "$DISTRO" == "ubuntu" ] || [ "$DISTRO" == "debian" ]; then sudo -E apt install -y linux-headers-$(uname -r) -elif [ "$DISTRO" == "raspbian" ]; then +elif [ "$DISTRO" == "raspbian" ] || [ "$DISTRO" == "raspios" ]; then sudo -E apt install -y raspberrypi-kernel-headers fi diff --git a/scripts/subinstallers/cockpit.sh b/scripts/subinstallers/cockpit.sh index 445ea69..fbdb141 100755 --- a/scripts/subinstallers/cockpit.sh +++ b/scripts/subinstallers/cockpit.sh @@ -10,11 +10,8 @@ elif [ "$DISTRO" == "debian" ]; then && echo "buster-backports already in sources" \ || echo 'deb http://deb.debian.org/debian buster-backports main' | sudo tee -a /etc/apt/sources.list.d/backports.list - -elif [ "$DISTRO" == "raspbian" ]; then - +elif [ "$DISTRO" == "raspbian" ] || [ "$DISTRO" == "raspios" ]; then echo "Raspbian repos contain cockpit" - fi sudo -E apt-get install -y cockpit diff --git a/scripts/subinstallers/docker_manual.sh b/scripts/subinstallers/docker_manual.sh index 5fca1c9..2cbb329 100755 --- a/scripts/subinstallers/docker_manual.sh +++ b/scripts/subinstallers/docker_manual.sh @@ -15,7 +15,7 @@ if [ "$DISTRO" == "ubuntu" ]; then curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - elif [ "$DISTRO" == "debian" ]; then curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - -elif [ "$DISTRO" == "raspbian" ]; then +elif [ "$DISTRO" == "raspbian" ] || [ "$DISTRO" == "raspios" ]; then curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | sudo apt-key add - fi @@ -31,7 +31,7 @@ elif [ "$DISTRO" == "debian" ]; then "deb https://download.docker.com/linux/debian \ $(lsb_release -cs) \ stable" -elif [ "$DISTRO" == "raspbian" ]; then +elif [ "$DISTRO" == "raspbian" ] || [ "$DISTRO" == "raspios" ]; then echo "deb [arch=armhf] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \ $(lsb_release -cs) stable" | \ sudo tee /etc/apt/sources.list.d/docker.list @@ -43,7 +43,7 @@ sudo apt-get update if [ "$DISTRO" == "ubuntu" ] || [ "$DISTRO" == "debian" ]; then sudo -E apt-get install -y docker-ce docker-ce-cli containerd.io -elif [ "$DISTRO" == "raspbian" ]; then +elif [ "$DISTRO" == "raspbian" ] || [ "$DISTRO" == "raspios" ]; then sudo -E apt install -y --no-install-recommends \ docker-ce \ cgroupfs-mount @@ -59,7 +59,7 @@ echo "Installing Docker Compose" # sudo chmod +x /usr/local/bin/docker-compose #elif [ "$DISTRO" == "raspbian" ]; then # Install required packages -sudo -E apt install -y python-backports.ssl-match-hostname +#sudo -E apt install -y python-backports.ssl-match-hostname # Install Docker Compose from pip # This might take a while diff --git a/scripts/subinstallers/wireguard.sh b/scripts/subinstallers/wireguard.sh index a544591..1b7b339 100755 --- a/scripts/subinstallers/wireguard.sh +++ b/scripts/subinstallers/wireguard.sh @@ -9,18 +9,19 @@ if ! $(sudo apt-cache show wireguard > /dev/null 2>&1) ; then echo "Adding Wireguard PPAs" # Wireguard - if [ "$DISTRO" == "raspbian" ]; then + if [ "$DISTRO" == "raspbian" ] || [ "$DISTRO" == "raspios" ]; then echo "Adding Wireguard repo keys" sudo -E apt-get install -y dirmngr sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8B48AD6246925553 sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7638D0442B90D010 sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 04EE7237B7D453EC + sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 648ACFD622F3D138 fi if [ "$DISTRO" == "ubuntu" ]; then # Ubuntu sudo add-apt-repository -y ppa:wireguard/wireguard - elif [ "$DISTRO" == "debian" ] || [ "$DISTRO" == "raspbian" ]; then + elif [ "$DISTRO" == "debian" ] || [ "$DISTRO" == "raspbian" ] || [ "$DISTRO" == "raspios" ]; then # Debian sudo bash -c 'echo "deb http://deb.debian.org/debian/ unstable main" > /etc/apt/sources.list.d/unstable.list' sudo bash -c "printf 'Package: *\nPin: release a=unstable\nPin-Priority: 90\n' > /etc/apt/preferences.d/limit-unstable" From 936822d7779c5b9a13f48401856b19678e70382a Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Wed, 5 Aug 2020 16:41:25 -0400 Subject: [PATCH 03/15] pass environment during installation --- README.md | 6 +++--- scripts/subinstallers/docker.sh | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 50ffc3d..adeb7ce 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ Recommended System Specifications: Starting from base installation ``` git clone https://gitlab.com/cyber5k/mistborn.git -sudo bash ./mistborn/scripts/install.sh +sudo -E bash ./mistborn/scripts/install.sh ``` Get default admin Wireguard profile @@ -123,7 +123,7 @@ Mistborn is regularly tested on Ubuntu 18.04 LTS (DigitalOcean droplet with 2 GB Clone the git repository and run the install script: ``` git clone https://gitlab.com/cyber5k/mistborn.git -sudo bash ./mistborn/scripts/install.sh +sudo -E bash ./mistborn/scripts/install.sh ``` Running `install.sh` will do the following: @@ -157,7 +157,7 @@ See the environment variables needed in `./scripts/noninteractive/.install_bareb This will perform a noninteractive install with the default environment variables set in `.install_barebones`. ``` git clone https://gitlab.com/cyber5k/mistborn.git -sudo bash -c "source ./mistborn/scripts/noninteractive/.install_barebones && ./mistborn/scripts/install.sh" +sudo -E bash -c "source ./mistborn/scripts/noninteractive/.install_barebones && ./mistborn/scripts/install.sh" ``` # Post-Installation diff --git a/scripts/subinstallers/docker.sh b/scripts/subinstallers/docker.sh index a051604..a4206ab 100755 --- a/scripts/subinstallers/docker.sh +++ b/scripts/subinstallers/docker.sh @@ -16,5 +16,5 @@ fi # set docker-compose path used in Mistborn if [ ! -f /usr/local/bin/docker-compose ]; then - sudo ln -s $(which docker-compose) /usr/local/bin/docker-compose + sudo -E ln -s $(which docker-compose) /usr/local/bin/docker-compose fi From 98e90af2d7d04838fcdf94f6fc74cc3c8ffda5fc Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Sat, 8 Aug 2020 23:23:28 +0000 Subject: [PATCH 04/15] Resolve "Migrate to Traefik v2" --- .gitlab-ci.yml | 1 - base.yml | 56 +++- compose/production/traefik/Dockerfile | 4 +- compose/production/traefik/dynamic.toml | 31 +++ .../production/traefik/traefik.toml.template | 257 ------------------ extra/bitwarden.yml | 9 +- extra/homeassistant.yml | 9 +- extra/jellyfin.yml | 9 +- extra/jitsi-meet.yml | 9 +- extra/nextcloud.yml | 9 +- extra/onlyoffice.yml | 9 +- extra/raspap.yml | 9 +- extra/rocketchat.yml | 9 +- extra/syncthing.yml | 9 +- scripts/install.sh | 2 +- scripts/update.sh | 2 +- 16 files changed, 153 insertions(+), 281 deletions(-) create mode 100644 compose/production/traefik/dynamic.toml delete mode 100644 compose/production/traefik/traefik.toml.template diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 03623c1..c38e178 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -19,6 +19,5 @@ docker_build: script: - apk add docker-compose - scripts/subinstallers/gen_prod_env.sh "$MISTBORN_DEFAULT_PASSWORD" - - cp compose/production/traefik/traefik.toml.template compose/production/traefik/traefik.toml - docker-compose -f base.yml build diff --git a/base.yml b/base.yml index 4596b96..f0536ef 100644 --- a/base.yml +++ b/base.yml @@ -14,7 +14,14 @@ services: - redis labels: - "traefik.enable=true" - - "traefik.port=5000" + - "traefik.http.routers.django-http.rule=Host(`home.mistborn`)" + - "traefik.http.routers.django-http.entrypoints=web" + - "traefik.http.routers.django-http.middlewares=mistborn_headers@file" + - "traefik.http.routers.django-https.rule=Host(`home.mistborn`)" + - "traefik.http.routers.django-https.entrypoints=websecure" + - "traefik.http.routers.django-https.middlewares=mistborn_headers@file" + - "traefik.http.routers.django-https.tls.certresolver=basic" + - "traefik.http.services.django-service.loadbalancer.server.port=5000" env_file: - ./.envs/.production/.django - ./.envs/.production/.postgres @@ -38,20 +45,41 @@ services: restart: unless-stopped traefik: - build: - context: . - dockerfile: ./compose/production/traefik/Dockerfile - image: mistborn_production_traefik + #build: + # context: . + # dockerfile: ./compose/production/traefik/Dockerfile + #image: mistborn_production_traefik + image: traefik:v2.2 container_name: mistborn_production_traefik depends_on: - django volumes: - - production_traefik:/etc/traefik/acme + #- production_traefik:/etc/traefik/acme + - ./compose/production/traefik/dynamic.toml:/dynamic.toml:ro - /var/run/docker.sock:/var/run/docker.sock:ro - - ../mistborn_volumes/base/tls:/tls - ports: - - "0.0.0.0:80:80/tcp" - - "0.0.0.0:443:443/tcp" + - ../mistborn_volumes/base/tls:/tls:ro + network_mode: host + #ports: + # - "0.0.0.0:80:80/tcp" + # - "0.0.0.0:443:443/tcp" + # #- "0.0.0.0:8080:8080/tcp" # dashboard + command: + ## API settings + #- --api.insecure=true + #- --api.dashboard=true + #- --api.debug=true + ## Logs + - --log.level=ERROR + ## Providers + - --providers.docker=true + - --providers.docker.exposedbydefault=false + - --providers.file.filename=/dynamic.toml + #- --providers.docker.network=default + ## Entrypoints + - --entrypoints.web.address=:80 + - --entrypoints.websecure.address=:443 + ## Certificates + #- --serversTransport.insecureSkipVerify=true restart: unless-stopped redis: @@ -136,6 +164,14 @@ services: - "${MISTBORN_DNS_BIND_IP}:53:53/udp" labels: - "traefik.enable=true" + - "traefik.http.routers.pihole-http.rule=Host(`pihole.mistborn`)" + - "traefik.http.routers.pihole-http.entrypoints=web" + - "traefik.http.routers.pihole-http.middlewares=mistborn_auth@file" + - "traefik.http.routers.pihole-https.rule=Host(`pihole.mistborn`)" + - "traefik.http.routers.pihole-https.entrypoints=websecure" + - "traefik.http.routers.pihole-https.middlewares=mistborn_auth@file" + - "traefik.http.routers.pihole-https.tls.certresolver=basic" + - "traefik.http.services.pihole-service.loadbalancer.server.port=80" environment: - ServerIP=10.2.0.3 - DNS1='10.2.0.2#5054' # docs say port 5054, was 54; use network_mode: host to see which port is used diff --git a/compose/production/traefik/Dockerfile b/compose/production/traefik/Dockerfile index 7088e6f..04c5f93 100644 --- a/compose/production/traefik/Dockerfile +++ b/compose/production/traefik/Dockerfile @@ -1,5 +1,5 @@ -FROM traefik:alpine +FROM traefik:v2.2 RUN mkdir -p /etc/traefik/acme RUN touch /etc/traefik/acme/acme.json RUN chmod 600 /etc/traefik/acme/acme.json -COPY ./compose/production/traefik/traefik.toml /etc/traefik +#COPY ./compose/production/traefik/dynamic.toml /etc/traefik diff --git a/compose/production/traefik/dynamic.toml b/compose/production/traefik/dynamic.toml new file mode 100644 index 0000000..0eeb776 --- /dev/null +++ b/compose/production/traefik/dynamic.toml @@ -0,0 +1,31 @@ +[tls.stores] + [tls.stores.default] + [tls.stores.default.defaultCertificate] + certFile = "/tls/cert.crt" + keyFile = "/tls/cert.key" + +[tls.options] + [tls.options.default] + minVersion = "VersionTLS12" + +[http.services] + [http.services.cockpit.loadBalancer] + [[http.services.cockpit.loadBalancer.servers]] + url = "http://10.2.3.1:9090" + +[http.routers] + [http.routers.cockpit] + rule = "Host(`cockpit.mistborn`)" + service = "cockpit" + entrypoints = ["web", "websecure"] + #middlewares = + +[http.middlewares] + [http.middlewares.mistborn_auth.forwardAuth] + address = "https://home.mistborn/auth/" + trustForwardHeader = true + [http.middlewares.mistborn_auth.forwardAuth.tls] + insecureSkipVerify = true + + [http.middlewares.mistborn_headers.headers] + hostsProxyHeaders = ['X-CSRFToken'] \ No newline at end of file diff --git a/compose/production/traefik/traefik.toml.template b/compose/production/traefik/traefik.toml.template deleted file mode 100644 index 267d53b..0000000 --- a/compose/production/traefik/traefik.toml.template +++ /dev/null @@ -1,257 +0,0 @@ -#debug = true - -logLevel = "ERROR" #DEBUG, INFO, WARN, ERROR, FATAL, PANIC -InsecureSkipVerify = true - -defaultEntryPoints = ["http", "https"] -#defaultEntryPoints = ["http"] - -# Entrypoints, http and https -[entryPoints] - # http should be redirected to https - [entryPoints.http] - address = ":80" - #[entryPoints.http.redirect] - #entryPoint = "https" - # https is the default - [entryPoints.https] - address = ":443" - [entryPoints.https.tls] - [entryPoints.https.tls.defaultCertificate] - certFile = "/tls/cert.crt" - keyFile = "/tls/cert.key" - -## Enable ACME (Let's Encrypt): automatic SSL -#[acme] -## Email address used for registration -#email = "steven@cyber5k.com" -#storage = "/etc/traefik/acme/acme.json" -#entryPoint = "https" -#onDemand = false -#OnHostRule = true -# # Use a HTTP-01 acme challenge rather than TLS-SNI-01 challenge -# [acme.httpChallenge] -# entryPoint = "http" - -[file] -[backends] - [backends.django] - [backends.django.servers.server1] - url = "http://django:5000" - - [backends.pihole] - [backends.pihole.servers.server1] - url = "http://pihole:80" - - [backends.homeassistant] - [backends.homeassistant.servers.server1] - url = "http://homeassistant:8123" - - [backends.syncthing] - [backends.syncthing.servers.server1] - url = "http://syncthing:8384" - - [backends.rocketchat] - [backends.rocketchat.servers.server1] - url = "http://rocketchat:3000" - - [backends.nextcloud] - [backends.nextcloud.servers.server1] - url = "http://nextcloud:80" - - [backends.onlyoffice] - [backends.onlyoffice.servers.server1] - url = "http://onlyoffice:80" - - [backends.bitwarden] - [backends.bitwarden.servers.server1] - url = "http://bitwarden:80" - - [backends.jellyfin] - [backends.jellyfin.servers.server1] - url = "http://jellyfin:8096" - - [backends.jitsi] - [backends.jitsi.servers.server1] - url = "http://jitsi-web:80" - - [backends.raspap] - [backends.raspap.servers.server1] - url = "http://raspap:80" - - [backends.cockpit] - [backends.cockpit.servers.server1] - url = "http://10.2.3.1:9090" - -[frontends] - [frontends.django] - backend = "django" - passHostHeader = true - [frontends.django.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.django.routes.dr1] - #rule = "Host:mistborn.cyber5k.com" - rule = "Host:home.mistborn" - - [frontends.pihole] - backend = "pihole" - passHostHeader = true - [frontends.pihole.auth] - [frontends.pihole.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.pihole.auth.forward.tls] - insecureSkipVerify = true - [frontends.pihole.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.pihole.routes.dr1] - rule = "Host:pihole.mistborn" - - [frontends.homeassistant] - backend = "homeassistant" - passHostHeader = true - [frontends.homeassistant.auth] - [frontends.homeassistant.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.homeassistant.auth.forward.tls] - insecureSkipVerify = true - [frontends.homeassistant.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.homeassistant.routes.dr1] - rule = "Host:homeassistant.mistborn" - - [frontends.syncthing] - backend = "syncthing" - passHostHeader = true - [frontends.syncthing.auth] - [frontends.syncthing.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.syncthing.auth.forward.tls] - insecureSkipVerify = true - [frontends.syncthing.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.syncthing.routes.dr1] - rule = "Host:syncthing.mistborn" - - [frontends.rocketchat] - backend = "rocketchat" - passHostHeader = true - [frontends.rocketchat.auth] - [frontends.rocketchat.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.rocketchat.auth.forward.tls] - insecureSkipVerify = true - [frontends.rocketchat.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.rocketchat.routes.dr1] - rule = "Host:chat.mistborn" - - [frontends.nextcloud] - backend = "nextcloud" - passHostHeader = true - [frontends.nextcloud.auth] - [frontends.nextcloud.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.nextcloud.auth.forward.tls] - insecureSkipVerify = true - [frontends.nextcloud.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.nextcloud.routes.dr1] - rule = "Host:nextcloud.mistborn" - - [frontends.onlyoffice] - backend = "onlyoffice" - passHostHeader = true - [frontends.onlyoffice.auth] - [frontends.onlyoffice.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.onlyoffice.auth.forward.tls] - insecureSkipVerify = true - [frontends.onlyoffice.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.onlyoffice.routes.dr1] - rule = "Host:onlyoffice.mistborn" - - [frontends.bitwarden] - backend = "bitwarden" - passHostHeader = true - [frontends.bitwarden.auth] - [frontends.bitwarden.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.bitwarden.auth.forward.tls] - insecureSkipVerify = true - [frontends.bitwarden.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.bitwarden.routes.dr1] - rule = "Host:bitwarden.mistborn" - - [frontends.jellyfin] - backend = "jellyfin" - passHostHeader = true - [frontends.jellyfin.auth] - [frontends.jellyfin.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.jellyfin.auth.forward.tls] - insecureSkipVerify = true - [frontends.jellyfin.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.jellyfin.routes.dr1] - rule = "Host:jellyfin.mistborn" - - [frontends.jitsi] - backend = "jitsi" - passHostHeader = true - [frontends.jitsi.auth] - [frontends.jitsi.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.jitsi.auth.forward.tls] - insecureSkipVerify = true - [frontends.jitsi.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.jitsi.routes.dr1] - rule = "Host:jitsi.mistborn" - - [frontends.raspap] - backend = "raspap" - passHostHeader = true - [frontends.raspap.auth] - [frontends.raspap.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.raspap.auth.forward.tls] - insecureSkipVerify = true - [frontends.raspap.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.raspap.routes.dr1] - rule = "Host:raspap.mistborn" - - [frontends.cockpit] - backend = "cockpit" - passHostHeader = true - [frontends.cockpit.auth] - [frontends.cockpit.auth.forward] - address = "https://home.mistborn/auth/" - trustForwardHeader = false - [frontends.cockpit.auth.forward.tls] - insecureSkipVerify = true - [frontends.cockpit.headers] - HostsProxyHeaders = ['X-CSRFToken'] - [frontends.cockpit.routes.dr1] - rule = "Host:cockpit.mistborn" - -## Connection to docker host system (docker.sock) -#[docker] -#endpoint = "unix:///var/run/docker.sock" -#domain = "localhost" -#watch = true -## This will hide all docker containers that don't have explicitly -## set label to "enable" -#exposedbydefault = false diff --git a/extra/bitwarden.yml b/extra/bitwarden.yml index dc6ff56..377b4b8 100644 --- a/extra/bitwarden.yml +++ b/extra/bitwarden.yml @@ -10,7 +10,14 @@ services: - ../../mistborn_volumes/extra/bitwarden:/data labels: - "traefik.enable=true" - - "traefik.port=80" + - "traefik.http.routers.bitwarden-http.rule=Host(`bitwarden.mistborn`)" + - "traefik.http.routers.bitwarden-http.entrypoints=web" + - "traefik.http.routers.bitwarden-http.middlewares=mistborn_auth@file" + - "traefik.http.routers.bitwarden-https.rule=Host(`bitwarden.mistborn`)" + - "traefik.http.routers.bitwarden-https.entrypoints=websecure" + - "traefik.http.routers.bitwarden-https.middlewares=mistborn_auth@file" + - "traefik.http.routers.bitwarden-https.tls.certresolver=basic" + - "traefik.http.services.bitwarden-service.loadbalancer.server.port=80" ports: - 3012:3012/tcp restart: unless-stopped diff --git a/extra/homeassistant.yml b/extra/homeassistant.yml index 2588772..db70d62 100644 --- a/extra/homeassistant.yml +++ b/extra/homeassistant.yml @@ -10,7 +10,14 @@ services: - TZ=America/New_York labels: - "traefik.enable=true" - - "traefik.port=8123" + - "traefik.http.routers.homeassistant-http.rule=Host(`homeassistant.mistborn`)" + - "traefik.http.routers.homeassistant-http.entrypoints=web" + - "traefik.http.routers.homeassistant-http.middlewares=mistborn_auth@file" + - "traefik.http.routers.homeassistant-https.rule=Host(`homeassistant.mistborn`)" + - "traefik.http.routers.homeassistant-https.entrypoints=websecure" + - "traefik.http.routers.homeassistant-https.middlewares=mistborn_auth@file" + - "traefik.http.routers.homeassistant-https.tls.certresolver=basic" + - "traefik.http.services.homeassistant-service.loadbalancer.server.port=8123" restart: unless-stopped networks: diff --git a/extra/jellyfin.yml b/extra/jellyfin.yml index e829e1a..2235322 100644 --- a/extra/jellyfin.yml +++ b/extra/jellyfin.yml @@ -14,7 +14,14 @@ services: - ../../mistborn_volumes/extra/nextcloud:/media:ro labels: - "traefik.enable=true" - - "traefik.port=8096" + - "traefik.http.routers.jellyfin-http.rule=Host(`jellyfin.mistborn`)" + - "traefik.http.routers.jellyfin-http.entrypoints=web" + - "traefik.http.routers.jellyfin-http.middlewares=mistborn_auth@file" + - "traefik.http.routers.jellyfin-https.rule=Host(`jellyfin.mistborn`)" + - "traefik.http.routers.jellyfin-https.entrypoints=websecure" + - "traefik.http.routers.jellyfin-https.middlewares=mistborn_auth@file" + - "traefik.http.routers.jellyfin-https.tls.certresolver=basic" + - "traefik.http.services.jellyfin-service.loadbalancer.server.port=8096" restart: unless-stopped networks: diff --git a/extra/jitsi-meet.yml b/extra/jitsi-meet.yml index cc86c8f..7f4b0ed 100644 --- a/extra/jitsi-meet.yml +++ b/extra/jitsi-meet.yml @@ -9,7 +9,14 @@ services: #- '${HTTPS_PORT}:443' labels: - "traefik.enable=true" - - "traefik.port=${HTTP_PORT}" + - "traefik.http.routers.jitsi-http.rule=Host(`jitsi.mistborn`)" + - "traefik.http.routers.jitsi-http.entrypoints=web" + - "traefik.http.routers.jitsi-http.middlewares=mistborn_auth@file" + - "traefik.http.routers.jitsi-https.rule=Host(`jitsi.mistborn`)" + - "traefik.http.routers.jitsi-https.entrypoints=websecure" + - "traefik.http.routers.jitsi-https.middlewares=mistborn_auth@file" + - "traefik.http.routers.jitsi-https.tls.certresolver=basic" + - "traefik.http.services.jitsi-service.loadbalancer.server.port=${HTTP_PORT}" volumes: - ${CONFIG}/web:/config - ${CONFIG}/web/letsencrypt:/etc/letsencrypt diff --git a/extra/nextcloud.yml b/extra/nextcloud.yml index dc72353..c452356 100644 --- a/extra/nextcloud.yml +++ b/extra/nextcloud.yml @@ -9,7 +9,14 @@ services: - ../.envs/.production/.nextcloud labels: - "traefik.enable=true" - - "traefik.port=80" + - "traefik.http.routers.nextcloud-http.rule=Host(`nextcloud.mistborn`)" + - "traefik.http.routers.nextcloud-http.entrypoints=web" + - "traefik.http.routers.nextcloud-http.middlewares=mistborn_auth@file" + - "traefik.http.routers.nextcloud-https.rule=Host(`nextcloud.mistborn`)" + - "traefik.http.routers.nextcloud-https.entrypoints=websecure" + - "traefik.http.routers.nextcloud-https.middlewares=mistborn_auth@file" + - "traefik.http.routers.nextcloud-https.tls.certresolver=basic" + - "traefik.http.services.nextcloud-service.loadbalancer.server.port=80" volumes: - ../../mistborn_volumes/extra/nextcloud:/var/www/html environment: diff --git a/extra/onlyoffice.yml b/extra/onlyoffice.yml index 63a54a7..043d33e 100644 --- a/extra/onlyoffice.yml +++ b/extra/onlyoffice.yml @@ -11,7 +11,14 @@ services: - ../.envs/.production/.onlyoffice labels: - "traefik.enable=true" - - "traefik.port=80" + - "traefik.http.routers.onlyoffice-http.rule=Host(`onlyoffice.mistborn`)" + - "traefik.http.routers.onlyoffice-http.entrypoints=web" + - "traefik.http.routers.onlyoffice-http.middlewares=mistborn_auth@file" + - "traefik.http.routers.onlyoffice-https.rule=Host(`onlyoffice.mistborn`)" + - "traefik.http.routers.onlyoffice-https.entrypoints=websecure" + - "traefik.http.routers.onlyoffice-https.middlewares=mistborn_auth@file" + - "traefik.http.routers.onlyoffice-https.tls.certresolver=basic" + - "traefik.http.services.onlyoffice-service.loadbalancer.server.port=80" restart: unless-stopped networks: diff --git a/extra/raspap.yml b/extra/raspap.yml index 66575e9..3be7dce 100644 --- a/extra/raspap.yml +++ b/extra/raspap.yml @@ -10,7 +10,14 @@ services: container_name: mistborn_production_raspap labels: - "traefik.enable=true" - - "traefik.port=80" + - "traefik.http.routers.raspap-http.rule=Host(`raspap.mistborn`)" + - "traefik.http.routers.raspap-http.entrypoints=web" + - "traefik.http.routers.raspap-http.middlewares=mistborn_auth@file" + - "traefik.http.routers.raspap-https.rule=Host(`raspap.mistborn`)" + - "traefik.http.routers.raspap-https.entrypoints=websecure" + - "traefik.http.routers.raspap-https.middlewares=mistborn_auth@file" + - "traefik.http.routers.raspap-https.tls.certresolver=basic" + - "traefik.http.services.raspap-service.loadbalancer.server.port=80" env_file: - ../.envs/.production/.pihole command: /start diff --git a/extra/rocketchat.yml b/extra/rocketchat.yml index a568aec..42350c3 100644 --- a/extra/rocketchat.yml +++ b/extra/rocketchat.yml @@ -17,7 +17,14 @@ services: - Accounts_UseDNSDomainCheck=False labels: - "traefik.enable=true" - - "traefik.port=3000" + - "traefik.http.routers.chat-http.rule=Host(`chat.mistborn`)" + - "traefik.http.routers.chat-http.entrypoints=web" + - "traefik.http.routers.chat-http.middlewares=mistborn_auth@file" + - "traefik.http.routers.chat-https.rule=Host(`chat.mistborn`)" + - "traefik.http.routers.chat-https.entrypoints=websecure" + - "traefik.http.routers.chat-https.middlewares=mistborn_auth@file" + - "traefik.http.routers.chat-https.tls.certresolver=basic" + - "traefik.http.services.chat-service.loadbalancer.server.port=3000" depends_on: - mongo #ports: diff --git a/extra/syncthing.yml b/extra/syncthing.yml index 4bd5e89..6083393 100644 --- a/extra/syncthing.yml +++ b/extra/syncthing.yml @@ -19,7 +19,14 @@ services: - 21027:21027/udp # protocol discovery labels: - "traefik.enable=true" - - "traefik.port=8384" + - "traefik.http.routers.syncthing-http.rule=Host(`syncthing.mistborn`)" + - "traefik.http.routers.syncthing-http.entrypoints=web" + - "traefik.http.routers.syncthing-http.middlewares=mistborn_auth@file" + - "traefik.http.routers.syncthing-https.rule=Host(`syncthing.mistborn`)" + - "traefik.http.routers.syncthing-https.entrypoints=websecure" + - "traefik.http.routers.syncthing-https.middlewares=mistborn_auth@file" + - "traefik.http.routers.syncthing-https.tls.certresolver=basic" + - "traefik.http.services.syncthing-service.loadbalancer.server.port=8384" restart: unless-stopped networks: diff --git a/scripts/install.sh b/scripts/install.sh index a904d7c..6caba26 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -205,7 +205,7 @@ sudo mkdir -p ../mistborn_volumes/base/pihole/etc-dnsmasqd sudo mkdir -p ../mistborn_volumes/extra # Traefik final setup (cockpit) -cp ./compose/production/traefik/traefik.toml.template ./compose/production/traefik/traefik.toml +#cp ./compose/production/traefik/traefikv2.toml.template ./compose/production/traefik/traefik.toml # setup tls certs source ./scripts/subinstallers/openssl.sh diff --git a/scripts/update.sh b/scripts/update.sh index 40e1c74..ce5a4e2 100755 --- a/scripts/update.sh +++ b/scripts/update.sh @@ -17,7 +17,7 @@ sudo find /etc/systemd/system/ -type f -name 'Mistborn*' | xargs sudo sed -i "s/ sudo find /etc/systemd/system/ -type f -name 'Mistborn*' | xargs sudo sed -i "s/DIFACE/$iface/" # handle updates to traefik -sudo cp /opt/mistborn/compose/production/traefik/traefik.toml.template /opt/mistborn/compose/production/traefik/traefik.toml +#sudo cp /opt/mistborn/compose/production/traefik/traefik.toml.template /opt/mistborn/compose/production/traefik/traefik.toml sudo systemctl daemon-reload sudo systemctl enable Mistborn-setup.service From 8d1de1695be85ce5073e4dfa493a2619a72195a4 Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Sun, 9 Aug 2020 09:28:15 -0400 Subject: [PATCH 05/15] unsupported OS --- scripts/install.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/scripts/install.sh b/scripts/install.sh index 6caba26..e777b79 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -134,6 +134,9 @@ if [ "$DISTRO" == "ubuntu" ] || [ "$DISTRO" == "debian" ]; then sudo -E apt install -y linux-headers-$(uname -r) elif [ "$DISTRO" == "raspbian" ] || [ "$DISTRO" == "raspios" ]; then sudo -E apt install -y raspberrypi-kernel-headers +else + echo "Unsupported OS: $DISTRO" + exit 1 fi # Wireugard From 9da4cfd43e1dd8ad22ed39c92dec0e161ab80cae Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Tue, 18 Aug 2020 20:15:55 +0000 Subject: [PATCH 06/15] traefik forward auth --- README.md | 2 +- base.yml | 2 ++ compose/production/traefik/dynamic.toml | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index adeb7ce..9115548 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ In Mistborn, Gateways are upstream from the VPN server so connections to third-p The Gateway adds an extra network hop. DNS is still resolved in Mistborn so pihole is still blocking ads. # Installation -Mistborn is regularly tested on Ubuntu 18.04 LTS (DigitalOcean droplet with 2 GB RAM). It has also been successfully used on Debian Buster and Raspbian Buster systems (though not regularly tested). Additionally tested on Ubuntu 20.04 LTS. +Mistborn is regularly tested on Ubuntu 20.04 LTS (DigitalOcean droplet with 2 GB RAM). It has also been successfully used on Debian Buster and Raspbian Buster systems (though not regularly tested). Make sure to install OS updates and restart before installing Mistborn (Wireguard installs differently on recent kernels). Clone the git repository and run the install script: ``` diff --git a/base.yml b/base.yml index f0536ef..04be844 100644 --- a/base.yml +++ b/base.yml @@ -12,6 +12,8 @@ services: depends_on: - postgres - redis + ports: + - "10.2.3.1:5000:5000/tcp" # auth access labels: - "traefik.enable=true" - "traefik.http.routers.django-http.rule=Host(`home.mistborn`)" diff --git a/compose/production/traefik/dynamic.toml b/compose/production/traefik/dynamic.toml index 0eeb776..3d3adbe 100644 --- a/compose/production/traefik/dynamic.toml +++ b/compose/production/traefik/dynamic.toml @@ -22,7 +22,7 @@ [http.middlewares] [http.middlewares.mistborn_auth.forwardAuth] - address = "https://home.mistborn/auth/" + address = "http://10.2.3.1:5000/auth/" trustForwardHeader = true [http.middlewares.mistborn_auth.forwardAuth.tls] insecureSkipVerify = true From c2143d278a633077ebe686f96605188a3c7881e9 Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Mon, 24 Aug 2020 01:54:02 +0000 Subject: [PATCH 07/15] Resolve "Mistborn and wireguard fail after restart and nextcloud" --- scripts/env/setup.sh | 13 +++++++++++++ scripts/install.sh | 19 ++++++------------- scripts/restart.sh | 9 +++++++++ scripts/services/Mistborn-setup.service | 1 + scripts/update.sh | 15 +++++---------- 5 files changed, 34 insertions(+), 23 deletions(-) create mode 100755 scripts/restart.sh diff --git a/scripts/env/setup.sh b/scripts/env/setup.sh index 8234cd1..b7c9643 100755 --- a/scripts/env/setup.sh +++ b/scripts/env/setup.sh @@ -1,5 +1,7 @@ #!/bin/bash +#### ENV file + VAR_FILE=/opt/mistborn/.env source /opt/mistborn/scripts/subinstallers/platform.sh @@ -19,3 +21,14 @@ if [ "$GIT_BRANCH" != "master" ]; then fi echo "MISTBORN_TAG=$MISTBORN_TAG" | sudo tee -a ${VAR_FILE} + +#### install and base services +iface=$(ip -o -4 route show to default | egrep -o 'dev [^ ]*' | awk 'NR==1{print $2}') + +# default interface +sudo cp /opt/mistborn/scripts/services/Mistborn* /etc/systemd/system/ +sudo find /etc/systemd/system/ -type f -name 'Mistborn*' | xargs sudo sed -i "s/User=root/User=$USER/" +#sudo find /etc/systemd/system/ -type f -name 'Mistborn*' | xargs sudo sed -i "s/ root:root / $USER:$USER /" +sudo find /etc/systemd/system/ -type f -name 'Mistborn*' | xargs sudo sed -i "s/DIFACE/$iface/" + +sudo systemctl daemon-reload \ No newline at end of file diff --git a/scripts/install.sh b/scripts/install.sh index e777b79..c87bc2f 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -117,10 +117,10 @@ fi # SSH Server sudo -E apt-get install -y openssh-server -sudo sed -i 's/#PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config -sudo sed -i 's/PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config -sudo sed -i 's/#PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config -sudo sed -i 's/PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config +#sudo sed -i 's/#PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config +#sudo sed -i 's/PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config +#sudo sed -i 's/#PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config +#sudo sed -i 's/PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config sudo sed -i 's/#Port.*/Port 22/' /etc/ssh/sshd_config sudo sed -i 's/Port.*/Port 22/' /etc/ssh/sshd_config sudo systemctl enable ssh @@ -166,8 +166,6 @@ sudo pip3 install -e ./modules/mistborn-cli # Mistborn # final setup vars -iface=$(ip -o -4 route show to default | egrep -o 'dev [^ ]*' | awk 'NR==1{print $2}') -figlet "Mistborn default NIC: $iface" #IPV4_PUBLIC=$(ip -o -4 route show default | egrep -o 'dev [^ ]*' | awk '{print $2}' | xargs ip -4 addr show | grep 'inet ' | awk '{print $2}' | grep -o "^[0-9.]*" | tr -cd '\11\12\15\40-\176' | head -1) # tail -1 to get last IPV4_PUBLIC="10.2.3.1" @@ -188,18 +186,13 @@ sudo systemctl restart unattended-upgrades # setup Mistborn services -# install and start base services -# default interface -sudo cp ./scripts/services/Mistborn* /etc/systemd/system/ -sudo find /etc/systemd/system/ -type f -name 'Mistborn*' | xargs sudo sed -i "s/User=root/User=$USER/" -#sudo find /etc/systemd/system/ -type f -name 'Mistborn*' | xargs sudo sed -i "s/ root:root / $USER:$USER /" -sudo find /etc/systemd/system/ -type f -name 'Mistborn*' | xargs sudo sed -i "s/DIFACE/$iface/" - #if [ "$DISTRO" == "debian" ] || [ "$DISTRO" == "raspbian" ]; then # # remove systemd-resolved lines # sudo sed -i '/.*systemd-resolved/d' /etc/systemd/system/Mistborn-base.service #fi +sudo cp ./scripts/services/Mistborn-setup.service /etc/systemd/system/ + # setup local volumes for pihole sudo mkdir -p ../mistborn_volumes/ sudo chown -R root:root ../mistborn_volumes/ diff --git a/scripts/restart.sh b/scripts/restart.sh new file mode 100755 index 0000000..b3a59a8 --- /dev/null +++ b/scripts/restart.sh @@ -0,0 +1,9 @@ +#!/bin/bash + +set -e + +export DEBIAN_FRONTEND=noninteractive + +sudo systemctl stop Mistborn-base +sudo systemctl restart Mistborn-setup +sudo systemctl restart Mistborn-base \ No newline at end of file diff --git a/scripts/services/Mistborn-setup.service b/scripts/services/Mistborn-setup.service index 9b83c71..706942e 100644 --- a/scripts/services/Mistborn-setup.service +++ b/scripts/services/Mistborn-setup.service @@ -4,6 +4,7 @@ Before=Mistborn-base.service [Service] Type=oneshot +User=mistborn ExecStart=/opt/mistborn/scripts/env/setup.sh [Install] diff --git a/scripts/update.sh b/scripts/update.sh index ce5a4e2..8026316 100755 --- a/scripts/update.sh +++ b/scripts/update.sh @@ -10,19 +10,14 @@ sudo git -C /opt/mistborn submodule update --init --recursive # ensure mistborn-cli is installed sudo pip3 install -e /opt/mistborn/modules/mistborn-cli -# handle updates to Mistborn systemd scripts -iface=$(ip -o -4 route show to default | egrep -o 'dev [^ ]*' | awk 'NR==1{print $2}') -sudo cp /opt/mistborn/scripts/services/Mistborn* /etc/systemd/system/ -sudo find /etc/systemd/system/ -type f -name 'Mistborn*' | xargs sudo sed -i "s/User=root/User=mistborn/" -sudo find /etc/systemd/system/ -type f -name 'Mistborn*' | xargs sudo sed -i "s/DIFACE/$iface/" - # handle updates to traefik #sudo cp /opt/mistborn/compose/production/traefik/traefik.toml.template /opt/mistborn/compose/production/traefik/traefik.toml -sudo systemctl daemon-reload -sudo systemctl enable Mistborn-setup.service -sudo systemctl restart Mistborn-setup.service - sudo mistborn-cli pullbuild +sudo docker container prune -f +sudo docker image prune -f + +sudo systemctl stop Mistborn-base +sudo systemctl restart Mistborn-setup sudo systemctl restart Mistborn-base From 083dcccdf5da412f3999a6d5204a8c4eac0010db Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Tue, 15 Sep 2020 08:26:16 -0400 Subject: [PATCH 08/15] mimic behavior of gen_passwords --- scripts/subinstallers/gen_prod_env.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/scripts/subinstallers/gen_prod_env.sh b/scripts/subinstallers/gen_prod_env.sh index ace958e..f4b9675 100755 --- a/scripts/subinstallers/gen_prod_env.sh +++ b/scripts/subinstallers/gen_prod_env.sh @@ -72,3 +72,9 @@ echo "SIGNUPS_ALLOWED=true" >> $BITWARDEN_PROD_FILE JITSI_PROD_FILE="./.envs/.production/.jitsi" cp ./scripts/conf/jitsi.env $JITSI_PROD_FILE mkdir -p ./.envs/.production/.jitsi-cfg/{web/letsencrypt,transcripts,prosody,jicofo,jvb} +sed -i "s/JICOFO_COMPONENT_SECRET.*/JICOFO_COMPONENT_SECRET=$(python3 -c "import secrets; import string; print(f''.join([secrets.choice(string.ascii_letters+string.digits) for x in range(32)]))")/" "$JITSI_PROD_FILE" +sed -i "s/JICOFO_AUTH_PASSWORD.*/JICOFO_AUTH_PASSWORD=$(python3 -c "import secrets; import string; print(f''.join([secrets.choice(string.ascii_letters+string.digits) for x in range(32)]))")/" "$JITSI_PROD_FILE" +sed -i "s/JVB_AUTH_PASSWORD.*/JVB_AUTH_PASSWORD=$(python3 -c "import secrets; import string; print(f''.join([secrets.choice(string.ascii_letters+string.digits) for x in range(32)]))")/" "$JITSI_PROD_FILE" +sed -i "s/JIGASI_XMPP_PASSWORD.*/JIGASI_XMPP_PASSWORD=$(python3 -c "import secrets; import string; print(f''.join([secrets.choice(string.ascii_letters+string.digits) for x in range(32)]))")/" "$JITSI_PROD_FILE" +sed -i "s/JIBRI_RECORDER_PASSWORD.*/JIBRI_RECORDER_PASSWORD=$(python3 -c "import secrets; import string; print(f''.join([secrets.choice(string.ascii_letters+string.digits) for x in range(32)]))")/" "$JITSI_PROD_FILE" +sed -i "s/JIBRI_XMPP_PASSWORD.*/JIBRI_XMPP_PASSWORD=$(python3 -c "import secrets; import string; print(f''.join([secrets.choice(string.ascii_letters+string.digits) for x in range(32)]))")/" "$JITSI_PROD_FILE" From a29f77b34ed582b078489fd560b47573dc5f6d67 Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Tue, 15 Sep 2020 14:50:22 -0400 Subject: [PATCH 09/15] rocketchat: mounting correct volumes --- extra/rocketchat.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/extra/rocketchat.yml b/extra/rocketchat.yml index 42350c3..66113f1 100644 --- a/extra/rocketchat.yml +++ b/extra/rocketchat.yml @@ -35,8 +35,8 @@ services: container_name: mistborn_production_rocketchat_mongo restart: unless-stopped volumes: - - ../volumes/extra/rocketchat/data/db:/data/db - - ../volumes/extra/rocketchat/data/dump:/dump + - ../../mistborn_volumes/extra/rocketchat/data/db:/data/db + - ../../mistborn_volumes/extra/rocketchat/data/dump:/dump command: mongod --smallfiles --oplogSize 128 --replSet rs0 --storageEngine=mmapv1 # this container's job is just run the command to initialize the replica set. @@ -61,7 +61,7 @@ services: depends_on: - rocketchat volumes: - - ../volumes/extra/rocketchat/hubot/scripts:/home/hubot/scripts + - ../../mistborn_volumes/extra/rocketchat/hubot/scripts:/home/hubot/scripts # this is used to expose the hubot port for notifications on the host on port 3001, e.g. for hubot-jenkins-notifier ports: - 3001:8080/tcp From 221377c01f933ed726aca15628c646415bc2269a Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Wed, 16 Sep 2020 21:27:55 +0000 Subject: [PATCH 10/15] Resolve "Unable to create wireguard config after successful installation on Azure" --- README.md | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 9115548..f51cd7e 100644 --- a/README.md +++ b/README.md @@ -68,10 +68,10 @@ Recommended System Specifications: | Use Case | Description | RAM | Hard Disk | |------------------------|-------------------------------------------------------------------------------|-------|-----------| -| Bare bones | Wireguard, Pihole (no Cockpit, no extra services) | 1 GB | 15 GB | -| Default | Bare bones + Cockpit | 2 GB | 15 GB | -| Low-resource services | Default + Bitwarden, Tor, Syncthing | 3 GB | 20 GB | -| High-resource services | Default + Jitsi, Nextcloud, Jellyfin, Rocket.Chat, Home Assistant, OnlyOffice | 4 GB+ | 25 GB+ | +| Bare bones | Wireguard, Pihole (no Cockpit, no extra services) | 2 GB | 15 GB | +| Default | Bare bones + Cockpit | 2 GB+ | 15 GB | +| Low-resource services | Default + Bitwarden, Tor, Syncthing | 4 GB | 20 GB | +| High-resource services | Default + Jitsi, Nextcloud, Jellyfin, Rocket.Chat, Home Assistant, OnlyOffice | 6 GB+ | 25 GB+ | Starting from base installation ``` @@ -111,7 +111,7 @@ We were getting frustrated at being forced to choose between being connected to *Netflix blocking my connections that it sees coming from a DigitalOcean droplet* -In Mistborn, Gateways are upstream from the VPN server so connections to third-party services (e.g. Netflix, Hulu, etc.) will appear to be coming from the public IP address of the Gateway. I setup a Gateway at home (Mistborn on DigitalOcean) then all Wireguard profiles created with this Gateway will appear to be coming from my house and are not blocked. No port-forwarding required (assuming Mistborn is publicly accessible). +In Mistborn, Gateways are upstream from the VPN server so connections to third-party services (e.g. Netflix, Hulu, etc.) will appear to be coming from the public IP address of the Gateway. I setup a Gateway at home (Raspberry Pi with `wireguard` and `openresolv` installed) and with our Mistborn on DigitalOcean, all Wireguard profiles created with this Gateway will appear to be coming from my house and are not blocked. No port-forwarding required (assuming Mistborn is publicly accessible). ![Mistborn Gateway Diagram](https://gitlab.com/cyber5k/public/-/raw/master/graphics/gateway_network.png) @@ -355,6 +355,22 @@ You can SSH using the Mistborn domain when connected by Wireguard: ssh user@home.mistborn ``` +## How do I change the upstream DNSCrypt servers? +The upstream servers used by dnscrypt-proxy are set in: + +`base.yml`: +``` +services: +... + dnscrypt-proxy: + ... + environment: + ... + - DNSCRYPT_SERVER_NAMES=[...] +``` + +The available options are here: https://download.dnscrypt.info/dnscrypt-resolvers/v2/public-resolvers.md + # Troubleshooting Once you're connected to Wireguard you should see .mistborn domains and the internet should work as expected. Be sure to use http (http://home.mistborn). Wireguard is the encrypted channel so there's usually no need to bother with TLS certs (WebRTC functionality and some mobile apps require TLS so it is available). Here are some things to check if you have issues: From 4b8f82a93f1362d16cf16fbfdb2e18106f2f7811 Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Sat, 19 Sep 2020 11:31:47 -0400 Subject: [PATCH 11/15] syncing with docker image: public IP selection --- README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/README.md b/README.md index f51cd7e..033441f 100644 --- a/README.md +++ b/README.md @@ -375,6 +375,21 @@ The available options are here: https://download.dnscrypt.info/dnscrypt-resolver Once you're connected to Wireguard you should see .mistborn domains and the internet should work as expected. Be sure to use http (http://home.mistborn). Wireguard is the encrypted channel so there's usually no need to bother with TLS certs (WebRTC functionality and some mobile apps require TLS so it is available). Here are some things to check if you have issues: +Check if you can ping an external IP address: +``` +ping 1.1.1.1 +``` + +Check if you can resolve local DNS queries: +``` +dig home.mistborn +``` + +Check if you can resolve external DNS queries: +``` +dig cyber5k.com +``` + See if any docker containers are stopped: ``` sudo docker container ls -a @@ -401,6 +416,9 @@ The `dev/` folder contains a script for completing a hard reset: destroying and sudo ./dev/rebuild.sh ``` +## Troubleshooting Wireguard +Ensure that your public IP address in your client profile (e.g. `Endpoint = :`) is actually publicly available (not in 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16) if you are attempting to access Mistborn across the internet. + ## Troubleshooting Extra Services Each extra service has its own systemd process which can be monitored: ``` From 1fa3a6d9e3ecc90f8aa89e8503f2f11713f097ad Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Wed, 23 Sep 2020 10:23:24 -0400 Subject: [PATCH 12/15] adding project support link --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 033441f..1ad4ed0 100644 --- a/README.md +++ b/README.md @@ -509,4 +509,5 @@ Contact me at [steven@cyber5k.com](mailto:steven@cyber5k.com) Please consider supporting the project via: - [Paypal.me](https://paypal.me/cyber5k) +- [Buy me a drink](https://www.buymeacoffee.com/cyber5k) - [Patreon](https://www.patreon.com/cyber5k) From fe26bcbd16c10910611c9905ec0ebb8ef18696d7 Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Wed, 21 Oct 2020 21:34:02 -0400 Subject: [PATCH 13/15] systemd require netfilter-persistent --- scripts/services/Mistborn-base.service | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/services/Mistborn-base.service b/scripts/services/Mistborn-base.service index 2428a7b..9f82b72 100644 --- a/scripts/services/Mistborn-base.service +++ b/scripts/services/Mistborn-base.service @@ -1,7 +1,9 @@ [Unit] Description=Mistborn Base Requires=docker.service +Requires=netfilter-persistent.service After=docker.service +After=netfilter-persistent.service [Service] Restart=always From 5a306d91eeeb371ea596f83b8a71d100efcf9254 Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Sun, 8 Nov 2020 01:12:57 +0000 Subject: [PATCH 14/15] Resolve "DNS not resolving after reboot" --- scripts/install.sh | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/scripts/install.sh b/scripts/install.sh index c87bc2f..31e0f83 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -216,6 +216,17 @@ sudo systemctl start Mistborn-setup.service sudo docker-compose -f base.yml pull || true sudo docker-compose -f base.yml build +## disable systemd-resolved stub listener (creates symbolic link to /etc/resolv.conf) +if [ -f /etc/systemd/resolved.conf ]; then + sudo sed -i 's/#DNSStubListener.*/DNSStubListener=no/' /etc/systemd/resolved.conf + sudo sed -i 's/DNSStubListener.*/DNSStubListener=no/' /etc/systemd/resolved.conf +fi + +## delete symlink if exists +if [ -L /etc/resolv.conf ]; then + sudo rm /etc/resolv.conf +fi + ## disable other DNS services sudo systemctl stop systemd-resolved 2>/dev/null || true sudo systemctl disable systemd-resolved 2>/dev/null || true From b7c63cb3a7159bbf2bf270ed82c72692aed848c5 Mon Sep 17 00:00:00 2001 From: Steven Foerster Date: Tue, 10 Nov 2020 20:25:44 +0000 Subject: [PATCH 15/15] Resolve "Some Bugs: Root PasswordLogin not disabled as espected, unable to resolve hostnames" --- README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1ad4ed0..fb9cdbd 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # Mistborn A secure platform for easily standing up and managing your own cloud services: including firewall, ad-blocking, and multi-factor Wireguard VPN access +![Mistborn Wireguard](https://gitlab.com/cyber5k/public/-/raw/master/graphics/home.mistborn_wireguard_.png)*Wireguard Management in Mistborn* + # Table of Contents [[_TOC_]] @@ -117,6 +119,11 @@ In Mistborn, Gateways are upstream from the VPN server so connections to third-p The Gateway adds an extra network hop. DNS is still resolved in Mistborn so pihole is still blocking ads. +# Client to client communication +By default direct communication between network clients is blocked. Mistborn clients can all talk to Mistborn and communicate via shared services (Jitsi, Nextcloud, etc). Direct client to client communication can be enabled via the "client-to-client" toggle. + +![System Settings](https://gitlab.com/cyber5k/public/-/raw/master/graphics/system_settings_dropdown.png) + # Installation Mistborn is regularly tested on Ubuntu 20.04 LTS (DigitalOcean droplet with 2 GB RAM). It has also been successfully used on Debian Buster and Raspbian Buster systems (though not regularly tested). Make sure to install OS updates and restart before installing Mistborn (Wireguard installs differently on recent kernels). @@ -348,7 +355,7 @@ If Mistborn is installed via SSH then an iptables rule is added allowing externa SSH is permitted from any device connected to Mistborn by Wireguard. -Password authentication in enabled. Mistborn disables password authentication for root. Fail2ban blocks IPs with excessive failed login attempts. +Password authentication in enabled. Fail2ban blocks IPs with excessive failed login attempts. You can SSH using the Mistborn domain when connected by Wireguard: ``` @@ -475,7 +482,7 @@ These are some notes regarding the technical design and implementations of Mistb ## Firewall - **IPtables**: Iptables rules and chains are manipulated directly. If UFW is present it is disabled. IPtables-persistent is used to save a simple set of secure default rules (most importantly setting the INPUT and FORWARD policies to DROP and allowing ESTABLISHED and RELATED traffic) that will be effective immediately upon system startup. Additional rules and chains are created by Docker on startup. Mistborn also creates some iptables chains during installation that are saved in the persistent rules. Mistborn iptables chains and rules are designed to work with Docker's with logic that is easy to follow. A power cycle will always result in a working state. - **PostUp/PostDown**: Wireguard configuration files on Mistborn include PostUp and PostDown directives that set routes and iptables rules for each Wireguard client individually. -- **Wireguard**: There is a one-to-one mapping between each Wireguard client and server instance listening on Mistborn. By design Wireguard clients cannot talk directly to each other but can use shared services and resources on Mistborn (e.g. Syncthing, Nextcloud, Jitisi, etc.) +- **Wireguard**: There is a one-to-one mapping between each Wireguard client and server instance listening on Mistborn. By default Wireguard clients cannot talk directly to each other but can use shared services and resources on Mistborn (e.g. Syncthing, Nextcloud, Jitisi, etc). Toggling the "client-to-client" option will enable direct client-to-client communication. - **Metrics**: In addition to the iptables INPUT policy set to DROP, an iptables chain exists that logs the packet meta data before dropping it. Mistborn redirects packets that will be dropped to this chain instead. A summary of the data about these dropped packets (unsolicited network traffic) can be found on the Metrics page. - **Coppercloud**: Coppercloud works by populating ipsets with the ipset module in iptables to DROP (blacklist) or ACCEPT (whitelist) a given set of IP addresses. Upon system startup a celery task will compile the IP addresses, create the ipsets, and iptables rules.