CGHMN-Multi-Node

At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.

To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.

CGHMN Network Components Overview

The basic layout looks like this:

• Both the US and the EU Proxmox VE servers have a container called core-router. This is, as the name suggests, the core CGHMN router for each particular node, taking care of routing local core services, locally hosted member servers and the CGHMN Wireguard clients and providing NATed internet access for local guest VMs and containers.
• The Proxmox VE nodes themselves and the core router containers are connected on one flat network through the OVH private network feature called vRack, this serves as the main inter-node routing backbone.
• The Wireguard interface is mirrored on all participating nodes, so that a member can connect to one or the other simply by changing the DNS peer name, without having to change public keys or IP addresses.

In order for a single client to be able to connect to one or the other Wireguard node without reconfiguring their own IP addresses, dynamic routing through OSPF (and more specifically the bird v2 routing daemon) is used. Routes are created over the vRack network when a peer connects and, after a short timeout, removed once the peer disconnects.

Wireguard Peer Monitor and Interface Sync

Since Wireguard is a stateless protocol, we cannot rely on the Wireguard tunnel or kernel module to tell us through hooks when a peer connects or disconnects, so we run a custom Bash script which parses the output of wg show wg-cghmn dump on a short interval and keeps track of the peers' last handshakes and transmitted bytes counters to determine if a peer is dead or alive.

The "aliveness" of a peer is currently determined through the time of its last handshake with the server, if any, and when bytes were last received on the server from the peer, in that order. If a peer transitions from a dead state to being alive, a route is inserted for each allowed IP entry of the peer into the main sytem routing table, which is then picked up by bird and announced as a dynamic route to other Wireguard nodes on the network. If a peer transitions from alive to dead, the opposite happens, all routes on the local Wireguard interface for each allowed IP entry of the peer is removed from the main system routing table and subsequently removed by bird from the announced dynamic routes.

Should this method of detecting peers be insufficient, e.g. when it is deemed to sluggish or fragile to detect peers roaming between nodes, member @DrScriptt (on Discord) has a project called static-route-manager that adds and removes routes based on ICMP ping responses, which could be a solid alternative for detecting connected peers.

In order for the Wireguard interfaces on all nodes to stay in sync, the central Wireguard Controller has been extended to push its Wireguard configuration to the nodes via HTTP, to a small PHP based API running on each node. The node API builds a valid Wireguard configuration from the transmitted JSON payload and applies it to its local Wireguard interface via wg syncconf wg-cghmn. Additionally, the Wireguard API itself is configured to not manage a Wireguard interface itself, so it serves basically just as a message broker and database of sorts.

DNS

Before switching to the multi-node design, peers would usually use 100.64.11.1, 100.64.12.1 or 100.64.12.2, which all more-or-less pointed towards the dnsmasq instance running on the core router. Since we now don't just have one core router, but one for each node, with none of them having a higher priority over the others, the DNS resolver is best moved to its own container on the US OVH Proxmox VE. Caching resolvers can be set up on each Proxmox VE node to resolve against it and cache results for a certain (short) amount of time, to bridge minor outages or high loads on the main dnsmasq instance.

This means in the future, peers can either use 100.64.11.19 to contact the new primary resolver directly, or use the virtual/NATed IP address 100.64.64.64 to contact the caching dnsmasq running on each core router container, which then forwards to 100.64.11.19 internally.

Perhaps an idea for the future: Run a second dnsmasq instance on 100.64.11.19 listening on port 1053 or similar and a matching forwarder on each core router and use an nftable rule on all core routes like ip daddr 100.64.64.65 meta l4proto { tcp, udp } th 53 redirect to 1053 in the destination NAT table to offer a DNS resolver that only resolves non-overridden domains for users with split network setups?