middleware

Resolve "Update README"

Closes #181

See merge request cyber5k/mistborn!84

README.md

@@ -15,6 +15,8 @@ The term [Mistborn](http://www.brandonsanderson.com/the-mistborn-saga-the-origin
 
 Mistborn started as a passion project for a husband and father protecting his family. Certain family members insisted on connecting their devices to free public WiFi networks. We needed a way to secure all family devices with a solid VPN (Wireguard). Once we had that we wanted to control DNS to block ads to all devices and block malicious websites across all family devices. Then we wanted chat, file-sharing, and webchat services that we could use for ourselves without entrusting our data to some big tech company. And then... home automation. I know I'll be adding more services so I made that easy to do.
 
+As an [Offensive Security Certified Professional (OSCP)](https://resources.infosecinstitute.com/certification/the-oscp-certification-and-exam/), I designed Mistborn thinking about how it would be attacked by both external and internal threats. In making design trade-off decisions I tend to the paranoid. See [Technical and Security Insights](#technical-and-security-insights).
+
 Ideal for teams who:
 - hate internet ads
 - need to be protected from malicious internet domains
@@ -39,7 +41,10 @@ These tools are not vital to Mistborn itself but are integrated to enhance secur
 - [Pi-hole](https://pi-hole.net): A DNS server for network-wide ad blocking, etc
 - [DNScrypt](https://www.dnscrypt.org): prevents DNS spoofing via cryptographic signatures to verify that responses originate from the chosen DNS resolver and haven't been tampered
 - [Traefik](https://docs.traefik.io): A modern, efficient reverse-proxy
+
+These tools can be turned on from the Mistborn Security Operations Center:
 - [Wazuh](https://wazuh.com/): Wazuh is a free, open source and enterprise-ready security monitoring solution for threat detection, integrity monitoring, incident response and compliance.
+- [Suricata](https://suricata-ids.org/): Suricata is a free and open source, mature, fast and robust network threat detection engine. The Suricata engine is capable of real time intrusion detection (IDS), inline intrusion prevention (IPS), network security monitoring (NSM) and offline pcap processing. Suricata inspects the network traffic using a powerful and extensive rules and signature language, and has powerful Lua scripting support for detection of complex threats.
 
 Within Mistborn is a panel to enable and manage these free extra services (off by default), locally hosted in Docker containers:
 - [Home Assistant](https://www.home-assistant.io): Open source home automation that puts local control and privacy first
@@ -99,7 +104,7 @@ sudo mistborn-cli getconf
 
 Connect via Wireguard then visit `http://home.mistborn`
 
-For more information, see the `Installation` section below.
+For more information, see the [Installation](#installation) section below.
 
 # Network Diagram
 ![Mistborn Network Diagram](https://gitlab.com/cyber5k/public/-/raw/master/graphics/mistborn_network.png)
@@ -117,6 +122,8 @@ See the [Mistborn Network Security](https://gitlab.com/cyber5k/mistborn/-/wikis/
 
 The Mistborn Security Operations Center provides SIEM services with Wazuh. The Wazuh Manager requires an Open Distro for Elasticsearch backend. When the Mistborn host has >8 GB RAM the provided Elasticsearch backend can be used. Just click "Start Wazuh" on the `Security Center` page and enjoy your Enterprise-grade SIEM. Wazuh agents can be installed on just about any OS and all Wazuh agent traffic is communicated over the Wireguard connections. Instructions for adding endpoint agents can be found within Wazuh itself.
 
+Mistborn's Wazuh installs and integrates with Suricata running on Mistborn with logs ingested into Wazuh.
+
 ![Mistborn Security Center: Wazuh Modules](https://gitlab.com/cyber5k/public/-/raw/master/graphics/wazuh_modules.png)
 
 The Wazuh Kibana plugin leverages the power of Elasticsearch:
@@ -552,7 +559,8 @@ Many features and refinements are in the works at various stages including:
 # Featured In
 
 - [Linux Magazine](https://www.linux-magazine.com/Issues/2020/240/Mistborn/(language)/eng-US) November 2020 (featuring Mistborn version from early May 2020)
-- [Awesome Open Source](https://www.youtube.com/watch?v=hekP0_crotw) July 2020 (featuring Mistborn version from early July 2020)
+- [Awesome Open Source](https://www.youtube.com/watch?v=hekP0_crotw) July 2020
+- [DB Tech](https://www.youtube.com/watch?v=UE_OuAOgoZI) May 2021
 
 # Follow
 You can find recent bugfixes, functional additions, some extra documentation and more at the Cyber5K Patreon page: [https://www.patreon.com/cyber5k](https://www.patreon.com/cyber5k)

base.yml

@@ -53,6 +53,13 @@ services:
     #image: mistborn_production_traefik
     image: traefik:v2.2
     container_name: mistborn_production_traefik
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.routers.api.rule=Host(`traefik.mistborn`)"
+      - "traefik.http.routers.api.service=api@internal"
+      - "traefik.http.routers.api.middlewares=mistborn_headers@file"
+      - "traefik.http.routers.api.entrypoints=web"
+      #- "traefik.http.services.internal.loadbalancer.server.port=5000"
     depends_on:
       - django
     volumes:
@@ -67,11 +74,11 @@ services:
     #  #- "0.0.0.0:8080:8080/tcp" # dashboard
     command:
       ## API settings
-      #- --api.insecure=true
+      - --api.insecure=true
-      #- --api.dashboard=true
+      - --api.dashboard=true
-      #- --api.debug=true
+      - --api.debug=true
       ## Logs
-      - --log.level=ERROR
+      - --log.level=INFO
       ## Providers
       - --providers.docker=true
       - --providers.docker.exposedbydefault=false
@@ -168,13 +175,16 @@ services:
       - "traefik.enable=true"
       - "traefik.http.routers.pihole-http.rule=Host(`pihole.mistborn`)"
       - "traefik.http.routers.pihole-http.entrypoints=web"
-      - "traefik.http.routers.pihole-http.middlewares=mistborn_auth@file,add-pihole-admin"
+      - "traefik.http.routers.pihole-http.middlewares=mistborn_auth,add-pihole-admin"
       - "traefik.http.routers.pihole-https.rule=Host(`pihole.mistborn`)"
       - "traefik.http.routers.pihole-https.entrypoints=websecure"
-      - "traefik.http.routers.pihole-https.middlewares=mistborn_auth@file,add-pihole-admin"
+      - "traefik.http.routers.pihole-https.middlewares=mistborn_auth,add-pihole-admin"
       - "traefik.http.routers.pihole-https.tls.certresolver=basic"
       - "traefik.http.middlewares.add-pihole-admin.addPrefix.prefix=/admin"
       - "traefik.http.services.pihole-service.loadbalancer.server.port=80"
+      - "traefik.http.middlewares.mistborn_auth.forwardAuth.address=http://${MISTBORN_BIND_IP}:5000/auth/"
+      - "traefik.http.middlewares.mistborn_auth.forwardAuth.trustForwardHeader=true"
+      - "traefik.http.middlewares.mistborn_auth.forwardAuth.tls.insecureSkipVerify=true"
    environment:
      - ServerIP=10.2.0.3
      - DNS1='10.2.0.2#5054' # docs say port 5054, was 54; use network_mode: host to see which port is used

compose/production/traefik/dynamic.toml

@@ -11,21 +11,15 @@
 [http.services]
   [http.services.cockpit.loadBalancer]
     [[http.services.cockpit.loadBalancer.servers]]
-      url = "http://10.2.3.1:9090"
+      url = "http://127.0.0.1:9090"
 
 [http.routers]
   [http.routers.cockpit]
     rule = "Host(`cockpit.mistborn`)"
     service = "cockpit"
     entrypoints = ["web", "websecure"]
-    middlewares = ["mistborn_auth"]
+    middlewares = ["mistborn_auth@docker"]
 
 [http.middlewares]
-  [http.middlewares.mistborn_auth.forwardAuth]
-    address = "http://10.2.3.1:5000/auth/"
-    trustForwardHeader = true
-    [http.middlewares.mistborn_auth.forwardAuth.tls]
-      insecureSkipVerify = true
-  
   [http.middlewares.mistborn_headers.headers]
     hostsProxyHeaders = ['X-CSRFToken']

scripts/services/Mistborn-wazuh.service

@@ -7,6 +7,7 @@ PartOf=Mistborn-base.service
 [Service]
 Restart=always
 RestartSec=15
+TimeoutStartSec=600
 User=root
 Group=docker
 PermissionsStartOnly=true
@@ -17,8 +18,12 @@ ExecStart=/opt/mistborn/scripts/wrappers/mistborn_docker.sh wazuh docker-compose
 # Agent install
 ExecStartPost=/opt/mistborn/scripts/wrappers/mistborn_docker.sh wazuh /opt/mistborn/scripts/services/wazuh/agent.sh
 ExecStartPost=-/opt/mistborn/scripts/wrappers/mistborn_docker.sh wazuh /opt/mistborn/scripts/services/wazuh/agent_start.sh
+# Suricata
+ExecStartPost=-/opt/mistborn/scripts/wrappers/mistborn_docker.sh wazuh /opt/mistborn/scripts/services/wazuh/suricata/suricata_init.sh
+ExecStartPost=-/opt/mistborn/scripts/wrappers/mistborn_docker.sh wazuh /opt/mistborn/scripts/services/wazuh/suricata/suricata_start.sh
 # Stop container when unit is stopped
 ExecStop=/opt/mistborn/scripts/wrappers/mistborn_docker.sh wazuh docker-compose -f /opt/mistborn/extra/wazuh.yml down
+ExecStopPost=-/opt/mistborn/scripts/wrappers/mistborn_docker.sh wazuh /opt/mistborn/scripts/services/wazuh/suricata/suricata_stop.sh
 ExecStopPost=-/opt/mistborn/scripts/wrappers/mistborn_docker.sh wazuh /opt/mistborn/scripts/services/wazuh/agent_stop.sh
 
 [Install]

scripts/services/wazuh/suricata/suricata_init.sh

@@ -0,0 +1,129 @@
+#!/bin/bash
+
+set -e
+
+# detect if suricata is installed
+if [[ $(dpkg-query -W -f='${Status}' suricata 2>/dev/null | grep -c "ok installed") -eq 1 ]]; then
+    echo "Suricata Installed"
+    exit 0
+fi
+
+source /opt/mistborn/scripts/subinstallers/platform.sh
+
+# minimal dependencies
+sudo -E apt-get -y install libpcre3 libpcre3-dbg libpcre3-dev build-essential libpcap-dev   \
+                libyaml-0-2 libyaml-dev pkg-config zlib1g zlib1g-dev \
+                make libmagic-dev libjansson-dev jq wget
+
+## recommended dependencies
+#sudo -E apt-get -y install libpcre3 libpcre3-dbg libpcre3-dev build-essential libpcap-dev   \
+#                libnet1-dev libyaml-0-2 libyaml-dev pkg-config zlib1g zlib1g-dev \
+#                libcap-ng-dev libcap-ng0 make libmagic-dev         \
+#                libgeoip-dev liblua5.1-dev libhiredis-dev libevent-dev \
+#                python-yaml rustc cargo
+
+# iptables/nftables integration
+sudo -E apt-get -y install libnetfilter-queue-dev libnetfilter-queue1  \
+                libnetfilter-log-dev libnetfilter-log1      \
+                libnfnetlink-dev libnfnetlink0
+
+
+if [ "$DISTRO" == "ubuntu" ]; then
+    echo "Installing Suricata Ubuntu PPA"
+    sudo -E add-apt-repository -y ppa:oisf/suricata-stable
+    sudo -E apt-get update
+    sudo -E apt-get install -y suricata
+elif [ "$DISTRO" == "debian" ]; then
+    # retrieve version codename
+    source /etc/os-release
+    echo "deb http://http.debian.net/debian $VERSION_CODENAME-backports main" | \
+        sudo -E tee /etc/apt/sources.list.d/backports.list
+    sudo -E apt-get update
+    sudo -E apt-get install -y suricata -t ${VERSION_CODENAME}-backports
+else
+    echo "Basic Suricata installation"
+    sudo -E apt-get install -y suricata
+fi
+
+# # iptables
+# sudo iptables -A INPUT -j NFQUEUE
+# sudo iptables -I FORWARD -j NFQUEUE
+# sudo iptables -I OUTPUT -j NFQUEUE
+
+# # rsyslog to create /var/log/suricata.log
+# sudo cp ./scripts/conf/20-suricata.conf /etc/rsyslog.d/
+# sudo chown root:root /etc/rsyslog.d/20-suricata.conf
+# sudo systemctl restart rsyslog
+
+# rules
+pushd .
+cd /tmp
+wget https://rules.emergingthreats.net/open/suricata-4.0/emerging.rules.tar.gz
+tar zxvf emerging.rules.tar.gz
+sudo -E rm /etc/suricata/rules/* -f
+sudo -E mv rules/*.rules /etc/suricata/rules/
+popd
+
+# suricata yaml
+sudo -E rm -f /etc/suricata/suricata.yaml
+sudo -E wget -O /etc/suricata/suricata.yaml http://www.branchnetconsulting.com/wazuh/suricata.yaml
+
+IFACE=$(ip -o -4 route show to default | awk 'NR==1{print $5}')
+sudo sed -i "s/eth0/${IFACE}/g" /etc/suricata/suricata.yaml
+sudo sed -i "s/eth0/${IFACE}/g" /etc/default/suricata
+
+#systemctl restart suricata
+
+# wait for service to be listening
+while ! nc -z 10.2.3.1 55000; do
+    WAIT_TIME=10
+    echo "Waiting ${WAIT_TIME} seconds for Wazuh API..."
+    sleep ${WAIT_TIME}
+done
+
+# set working directory to mistborn for docker-compose
+pushd .
+cd /opt/mistborn
+
+# ensure group exists
+sudo docker-compose --env-file /opt/mistborn/.env -f extra/wazuh.yml exec -T wazuh /var/ossec/bin/agent_groups -a -g suricata -q 2>/dev/null
+
+# add this host to group
+WAZUH_ID=$(sudo docker-compose --env-file /opt/mistborn/.env -f extra/wazuh.yml exec -T wazuh /var/ossec/bin/manage_agents -l | egrep ^\ *ID | grep $(hostname) | awk '{print $2}' | tr -d ',')
+sudo docker-compose --env-file /opt/mistborn/.env -f extra/wazuh.yml exec -T wazuh /var/ossec/bin/agent_groups -a -i ${WAZUH_ID} -g suricata -q
+
+# write agent.conf
+sudo docker-compose --env-file /opt/mistborn/.env -f extra/wazuh.yml exec -T wazuh bash -c "cat > /var/ossec/etc/shared/suricata/agent.conf << EOF
+<agent_config>
+    <localfile>
+        <log_format>json</log_format>
+        <location>/var/log/suricata/eve.json</location>
+    </localfile>
+</agent_config>
+EOF
+"
+
+# restart manager
+sudo docker-compose --env-file /opt/mistborn/.env -f extra/wazuh.yml restart wazuh
+
+popd
+
+# suricata-update
+sudo -E apt install python3-pip
+sudo -E pip3 install pyyaml
+sudo -E pip3 install https://github.com/OISF/suricata-update/archive/master.zip
+
+sudo -E pip3 install --pre --upgrade suricata-update
+
+# sudo -E suricata-update enable-source oisf/trafficid
+# sudo -E suricata-update enable-source etnetera/aggressive
+# sudo -E suricata-update enable-source sslbl/ssl-fp-blacklist
+# sudo -E suricata-update enable-source et/open
+# sudo -E suricata-update enable-source tgreen/hunting
+# sudo -E suricata-update enable-source sslbl/ja3-fingerprints
+# sudo -E suricata-update enable-source ptresearch/attackdetection
+
+sudo -E suricata-update
+
+sudo systemctl daemon-reload
+sudo systemctl restart suricata

scripts/services/wazuh/suricata/suricata_start.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+systemctl start suricata
+systemctl enable suricata
+
+#apt-get install -y python-pyinotify
+#python /opt/mistborn/scripts/services/scirius/suri_reloader -p /etc/suricata/rules &

scripts/services/wazuh/suricata/suricata_stop.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+systemctl stop suricata
+systemctl disable suricata
+
+#kill $(pgrep -f suri_reloader) 2>/dev/null

scripts/subinstallers/docker_manual.sh

@@ -67,3 +67,8 @@ echo "Installing Docker Compose"
 sudo pip3 install cryptography==3.3.2 docker-compose
 #fi
 
+
+# check raspbian fixes
+if [ "$DISTRO" == "raspbian" ] || [ "$DISTRO" == "raspios" ]; then
+    source ./scripts/subinstallers/docker_raspbian.sh 
+fi

scripts/subinstallers/docker_raspbian.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+set +e
+
+compare_version() {
+  local versionOne="${1}"
+  local comparision="${2}"
+  local versionTwo="${3}"
+  local result=
+  local sortOpt=
+  local returncode=1
+ 
+  if [[ "${versionOne}" == "${versionTwo}" ]] ; then
+    return 3
+  fi
+ 
+  case ${comparision} in
+    lower|smaller|older|lt|"<" ) sortOpt= ;;
+    higher|bigger|newer|bt|">" ) sortOpt='r' ;;
+    * ) return 2 ;;
+  esac
+ 
+  result=($(printf "%s\n" "${versionOne}" "${versionTwo}" | sort -${sortOpt}V ))
+  if [[ "${versionOne}" == "${result[0]}" ]] ; then
+    returncode=0
+  fi
+ 
+  return ${returncode}
+} # end of function compare_version
+
+# libseccomp2
+LIBSECCOMP2_VERSION=$(sudo -E apt-cache policy libseccomp2 | egrep ^\ *Inst | awk '{print $2}')
+
+compare_version $LIBSECCOMP2_VERSION '<' '2.5.1-1'
+
+if [ $? -eq 0 ]; then
+    # this is dumb but the raspbian repo managers aren't impressive
+    echo "Installing newer libseccomp2"
+    pushd .
+    cd /tmp
+    wget http://ftp.us.debian.org/debian/pool/main/libs/libseccomp/libseccomp2_2.5.1-1_$(dpkg --print-architecture).deb
+    sudo dpkg -i libseccomp2_2.5.1-1_$(dpkg --print-architecture).deb
+    popd
+fi

scripts/wrappers/mistborn_docker.sh

@@ -37,6 +37,12 @@ for SERVICE in "${SERVICES_ARRAY[@]}"; do
     fi
 done
 
+# check that netcat exists
+if ! [ -x "$(command -v nc)" ]; then
+    echo "Installing netcat"
+    sudo apt-get install -y netcat
+fi
+
 # ensure base is up and listening
 echo "Checking that Mistborn-base has finished starting up..."