CGHMN-Multi-Node - Revision history

Snep: Update target EU address

2026-10-28T23:54:38Z

Update target EU address

← Older revision		Revision as of 15:54, 28 October 2026
Line 1:		Line 1:
	<s>At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host</s>, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.		<s>At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host</s>, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.

	Edit: It should be possible to connect through the EU node now by changing your peer address from <code>us.wg.cghmn.org:42070</code> to <code>51.75.55.~~235~~:42070</code>, without having to reconfigure anything. The timeout of the peer monitoring script described below is somwhere between two and three minutes, so it'll take some time before your new connection is fully reachable on the network.		Edit: It should be possible to connect through the EU node now by changing your peer address from <code>us.wg.cghmn.org:42070</code> to <code>eu.wg.cghmn.org:42070</code>, without having to reconfigure anything. The timeout of the peer monitoring script described below is somwhere between two and three minutes, so it'll take some time before your new connection is fully reachable on the network.

	To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.		To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.

Snep: Add notes about testing the new connection

2026-10-28T12:08:14Z

Add notes about testing the new connection

← Older revision		Revision as of 04:08, 28 October 2026
Line 1:		Line 1:
	At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.		<s>At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host</s>, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.

			Edit: It should be possible to connect through the EU node now by changing your peer address from <code>us.wg.cghmn.org:42070</code> to <code>51.75.55.235:42070</code>, without having to reconfigure anything. The timeout of the peer monitoring script described below is somwhere between two and three minutes, so it'll take some time before your new connection is fully reachable on the network.

	To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.		To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.
Line 21:		Line 23:

	=== DNS ===		=== DNS ===
	Before switching to the multi-node design, peers would usually use <code>100.64.11.1</code>, <code>100.64.12.1</code> or <code>100.64.12.2</code>, which all more-or-less pointed towards the dnsmasq instance running on the core router. Since we now don't just have one core router, but one for each node, with none of them having a higher priority over the others, the DNS resolver is best moved to its own container on the US OVH Proxmox VE. Caching resolvers ~~can be~~ set up on each Proxmox VE node to resolve against it and cache results for a certain (short) amount of time, to bridge minor outages or high loads on the main dnsmasq instance.		Before switching to the multi-node design, peers would usually use <code>100.64.11.1</code>, <code>100.64.12.1</code> or <code>100.64.12.2</code>, which all more-or-less pointed towards the dnsmasq instance running on the core router. Since we now don't just have one core router, but one for each node, with none of them having a higher priority over the others, the DNS resolver is best moved to its own container on the US OVH Proxmox VE. Caching resolvers are set up on each Proxmox VE node to resolve against it and cache results for a certain (short) amount of time, to bridge minor outages or high loads on the main dnsmasq instance.

			This means, peers can now either use <code>100.64.11.19</code> to contact the new primary resolver directly, or use the virtual/NATed IP address <code>100.64.64.64</code> to contact the caching dnsmasq running on each core router container, which then forwards to <code>100.64.11.19</code> internally.

	~~This means in the future, peers can either use~~ <code>100.64.11.19</code> ~~to contact the new primary resolver directly~~, ~~or use the virtual/NATed IP address~~ <code>100.64.64.64</code> ~~to contact the caching dnsmasq running on each core router container, which then forwards to~~ <code>100.64.11.19</code> ~~internally~~.		The old DNS resolver IP addresses <code>100.64.11.1</code>, <code>100.64.12.1</code> and <code>100.64.12.2</code> are now nftable-redirect aliases to each respective Wireguard node, so they should still work just like before for the time being.

	Perhaps an idea for the future: Run a second dnsmasq instance on <code>100.64.11.19</code> listening on port <code>1053</code> or similar and a matching forwarder on each core router and use an nftable rule on all core routes like <code>ip daddr 100.64.64.65 meta l4proto { tcp, udp } th 53 redirect to 1053</code> in the destination NAT table to offer a DNS resolver that only resolves non-overridden domains for users with split network setups?		Perhaps an idea for the future: Run a second dnsmasq instance on <code>100.64.11.19</code> listening on port <code>1053</code> or similar and a matching forwarder on each core router and use an nftable rule on all core routes like <code>ip daddr 100.64.64.65 meta l4proto { tcp, udp } th 53 redirect to 1053</code> in the destination NAT table to offer a DNS resolver that only resolves non-overridden domains for users with split network setups?

Snep: Add section about DNS

2026-10-28T08:05:45Z

Add section about DNS

← Older revision		Revision as of 00:05, 28 October 2026
Line 11:		Line 11:
	In order for a single client to be able to connect to one or the other Wireguard node without reconfiguring their own IP addresses, dynamic routing through OSPF (and more specifically the bird v2 routing daemon) is used. Routes are created over the vRack network when a peer connects and, after a short timeout, removed once the peer disconnects.		In order for a single client to be able to connect to one or the other Wireguard node without reconfiguring their own IP addresses, dynamic routing through OSPF (and more specifically the bird v2 routing daemon) is used. Routes are created over the vRack network when a peer connects and, after a short timeout, removed once the peer disconnects.

			=== Wireguard Peer Monitor and Interface Sync ===
	Since Wireguard is a stateless protocol, we cannot rely on the Wireguard tunnel or kernel module to tell us through hooks when a peer connects or disconnects, so we run a custom Bash script which parses the output of <code>wg show wg-cghmn dump</code> on a short interval and keeps track of the peers' last handshakes and transmitted bytes counters to determine if a peer is dead or alive.		Since Wireguard is a stateless protocol, we cannot rely on the Wireguard tunnel or kernel module to tell us through hooks when a peer connects or disconnects, so we run a custom Bash script which parses the output of <code>wg show wg-cghmn dump</code> on a short interval and keeps track of the peers' last handshakes and transmitted bytes counters to determine if a peer is dead or alive.

Line 18:		Line 19:

	In order for the Wireguard interfaces on all nodes to stay in sync, the [https://github.com/CGHMN/wireguard-controller central Wireguard Controller] has been extended to push its Wireguard configuration to the nodes via HTTP, to a [https://github.com/CGHMN/wg-endpoint-api small PHP based API] running on each node. The node API builds a valid Wireguard configuration from the transmitted JSON payload and applies it to its local Wireguard interface via <code>wg syncconf wg-cghmn</code>. Additionally, the Wireguard API itself is configured to not manage a Wireguard interface itself, so it serves basically just as a message broker and database of sorts.		In order for the Wireguard interfaces on all nodes to stay in sync, the [https://github.com/CGHMN/wireguard-controller central Wireguard Controller] has been extended to push its Wireguard configuration to the nodes via HTTP, to a [https://github.com/CGHMN/wg-endpoint-api small PHP based API] running on each node. The node API builds a valid Wireguard configuration from the transmitted JSON payload and applies it to its local Wireguard interface via <code>wg syncconf wg-cghmn</code>. Additionally, the Wireguard API itself is configured to not manage a Wireguard interface itself, so it serves basically just as a message broker and database of sorts.

			=== DNS ===
			Before switching to the multi-node design, peers would usually use <code>100.64.11.1</code>, <code>100.64.12.1</code> or <code>100.64.12.2</code>, which all more-or-less pointed towards the dnsmasq instance running on the core router. Since we now don't just have one core router, but one for each node, with none of them having a higher priority over the others, the DNS resolver is best moved to its own container on the US OVH Proxmox VE. Caching resolvers can be set up on each Proxmox VE node to resolve against it and cache results for a certain (short) amount of time, to bridge minor outages or high loads on the main dnsmasq instance.

			This means in the future, peers can either use <code>100.64.11.19</code> to contact the new primary resolver directly, or use the virtual/NATed IP address <code>100.64.64.64</code> to contact the caching dnsmasq running on each core router container, which then forwards to <code>100.64.11.19</code> internally.

			Perhaps an idea for the future: Run a second dnsmasq instance on <code>100.64.11.19</code> listening on port <code>1053</code> or similar and a matching forwarder on each core router and use an nftable rule on all core routes like <code>ip daddr 100.64.64.65 meta l4proto { tcp, udp } th 53 redirect to 1053</code> in the destination NAT table to offer a DNS resolver that only resolves non-overridden domains for users with split network setups?

Snep: WIP: Initial revision

2026-10-28T02:48:55Z

WIP: Initial revision

New page

At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.

To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.
[[File:Multi-Node Overview.png|thumb|CGHMN Network Components Overview]]
The basic layout looks like this:

* Both the US and the EU Proxmox VE servers have a container called core-router. This is, as the name suggests, the core CGHMN router for each particular node, taking care of routing local core services, locally hosted member servers and the CGHMN Wireguard clients and providing NATed internet access for local guest VMs and containers.
* The Proxmox VE nodes themselves and the core router containers are connected on one flat network through the OVH private network feature called vRack, this serves as the main inter-node routing backbone.
* The Wireguard interface is mirrored on all participating nodes, so that a member can connect to one or the other simply by changing the DNS peer name, without having to change public keys or IP addresses.

In order for a single client to be able to connect to one or the other Wireguard node without reconfiguring their own IP addresses, dynamic routing through OSPF (and more specifically the bird v2 routing daemon) is used. Routes are created over the vRack network when a peer connects and, after a short timeout, removed once the peer disconnects.

Since Wireguard is a stateless protocol, we cannot rely on the Wireguard tunnel or kernel module to tell us through hooks when a peer connects or disconnects, so we run a custom Bash script which parses the output of <code>wg show wg-cghmn dump</code> on a short interval and keeps track of the peers' last handshakes and transmitted bytes counters to determine if a peer is dead or alive.

The "aliveness" of a peer is currently determined through the time of its last handshake with the server, if any, and when bytes were last received on the server from the peer, in that order. If a peer transitions from a dead state to being alive, a route is inserted for each allowed IP entry of the peer into the main sytem routing table, which is then picked up by bird and announced as a dynamic route to other Wireguard nodes on the network. If a peer transitions from alive to dead, the opposite happens, all routes on the local Wireguard interface for each allowed IP entry of the peer is removed from the main system routing table and subsequently removed by bird from the announced dynamic routes.

Should this method of detecting peers be insufficient, e.g. when it is deemed to sluggish or fragile to detect peers roaming between nodes, member @DrScriptt (on Discord) has a project called static-route-manager that adds and removes routes based on ICMP ping responses, which could be a solid alternative for detecting connected peers.

In order for the Wireguard interfaces on all nodes to stay in sync, the [https://github.com/CGHMN/wireguard-controller central Wireguard Controller] has been extended to push its Wireguard configuration to the nodes via HTTP, to a [https://github.com/CGHMN/wg-endpoint-api small PHP based API] running on each node. The node API builds a valid Wireguard configuration from the transmitted JSON payload and applies it to its local Wireguard interface via <code>wg syncconf wg-cghmn</code>. Additionally, the Wireguard API itself is configured to not manage a Wireguard interface itself, so it serves basically just as a message broker and database of sorts.