Cursed Silicons Wiki - Recent changes [en]

CGHMN-Multi-Node

Snep — Wed, 28 Oct 2026 23:54:38 GMT

Update target EU address

← Older revision		Revision as of 15:54, 28 October 2026
Line 1:		Line 1:
	<s>At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host</s>, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.		<s>At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host</s>, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.

	Edit: It should be possible to connect through the EU node now by changing your peer address from <code>us.wg.cghmn.org:42070</code> to <code>51.75.55.~~235~~:42070</code>, without having to reconfigure anything. The timeout of the peer monitoring script described below is somwhere between two and three minutes, so it'll take some time before your new connection is fully reachable on the network.		Edit: It should be possible to connect through the EU node now by changing your peer address from <code>us.wg.cghmn.org:42070</code> to <code>eu.wg.cghmn.org:42070</code>, without having to reconfigure anything. The timeout of the peer monitoring script described below is somwhere between two and three minutes, so it'll take some time before your new connection is fully reachable on the network.

	To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.		To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.

CGHMN DNS Information

Bredo — Wed, 28 Oct 2026 17:51:37 GMT

point to 100.64.64.64

← Older revision		Revision as of 09:51, 28 October 2026
Line 7:		Line 7:

	==== Pointing to the right DNS server ====		==== Pointing to the right DNS server ====
	CGHMN has several DNS servers in use for differing purposes. '''The correct default DNS server you should be pointing at while getting started is 100.64.11.~~1 from the core network, or 100.89.128.0 from the Wireguard tunnel~~'''. This is the router, which then forwards the requests to the actual DNS.		CGHMN has several DNS servers in use for differing purposes. '''The correct default DNS server you should be pointing at while getting started is 100.64.64.64'''. This is the router, which then forwards the requests to the actual DNS.

	==== What the different DNS servers are (or, is this thing on?) ====		==== What the different DNS servers are (or, is this thing on?) ====

CGHMN-Multi-Node

Snep — Wed, 28 Oct 2026 12:08:14 GMT

Add notes about testing the new connection

← Older revision		Revision as of 04:08, 28 October 2026
(One intermediate revision by the same user not shown)
Line 1:		Line 1:
	At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.		<s>At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host</s>, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.

			Edit: It should be possible to connect through the EU node now by changing your peer address from <code>us.wg.cghmn.org:42070</code> to <code>51.75.55.235:42070</code>, without having to reconfigure anything. The timeout of the peer monitoring script described below is somwhere between two and three minutes, so it'll take some time before your new connection is fully reachable on the network.

	To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.		To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.
Line 11:		Line 13:
	In order for a single client to be able to connect to one or the other Wireguard node without reconfiguring their own IP addresses, dynamic routing through OSPF (and more specifically the bird v2 routing daemon) is used. Routes are created over the vRack network when a peer connects and, after a short timeout, removed once the peer disconnects.		In order for a single client to be able to connect to one or the other Wireguard node without reconfiguring their own IP addresses, dynamic routing through OSPF (and more specifically the bird v2 routing daemon) is used. Routes are created over the vRack network when a peer connects and, after a short timeout, removed once the peer disconnects.

			=== Wireguard Peer Monitor and Interface Sync ===
	Since Wireguard is a stateless protocol, we cannot rely on the Wireguard tunnel or kernel module to tell us through hooks when a peer connects or disconnects, so we run a custom Bash script which parses the output of <code>wg show wg-cghmn dump</code> on a short interval and keeps track of the peers' last handshakes and transmitted bytes counters to determine if a peer is dead or alive.		Since Wireguard is a stateless protocol, we cannot rely on the Wireguard tunnel or kernel module to tell us through hooks when a peer connects or disconnects, so we run a custom Bash script which parses the output of <code>wg show wg-cghmn dump</code> on a short interval and keeps track of the peers' last handshakes and transmitted bytes counters to determine if a peer is dead or alive.

Line 18:		Line 21:

	In order for the Wireguard interfaces on all nodes to stay in sync, the [https://github.com/CGHMN/wireguard-controller central Wireguard Controller] has been extended to push its Wireguard configuration to the nodes via HTTP, to a [https://github.com/CGHMN/wg-endpoint-api small PHP based API] running on each node. The node API builds a valid Wireguard configuration from the transmitted JSON payload and applies it to its local Wireguard interface via <code>wg syncconf wg-cghmn</code>. Additionally, the Wireguard API itself is configured to not manage a Wireguard interface itself, so it serves basically just as a message broker and database of sorts.		In order for the Wireguard interfaces on all nodes to stay in sync, the [https://github.com/CGHMN/wireguard-controller central Wireguard Controller] has been extended to push its Wireguard configuration to the nodes via HTTP, to a [https://github.com/CGHMN/wg-endpoint-api small PHP based API] running on each node. The node API builds a valid Wireguard configuration from the transmitted JSON payload and applies it to its local Wireguard interface via <code>wg syncconf wg-cghmn</code>. Additionally, the Wireguard API itself is configured to not manage a Wireguard interface itself, so it serves basically just as a message broker and database of sorts.

			=== DNS ===
			Before switching to the multi-node design, peers would usually use <code>100.64.11.1</code>, <code>100.64.12.1</code> or <code>100.64.12.2</code>, which all more-or-less pointed towards the dnsmasq instance running on the core router. Since we now don't just have one core router, but one for each node, with none of them having a higher priority over the others, the DNS resolver is best moved to its own container on the US OVH Proxmox VE. Caching resolvers are set up on each Proxmox VE node to resolve against it and cache results for a certain (short) amount of time, to bridge minor outages or high loads on the main dnsmasq instance.

			This means, peers can now either use <code>100.64.11.19</code> to contact the new primary resolver directly, or use the virtual/NATed IP address <code>100.64.64.64</code> to contact the caching dnsmasq running on each core router container, which then forwards to <code>100.64.11.19</code> internally.

			The old DNS resolver IP addresses <code>100.64.11.1</code>, <code>100.64.12.1</code> and <code>100.64.12.2</code> are now nftable-redirect aliases to each respective Wireguard node, so they should still work just like before for the time being.

			Perhaps an idea for the future: Run a second dnsmasq instance on <code>100.64.11.19</code> listening on port <code>1053</code> or similar and a matching forwarder on each core router and use an nftable rule on all core routes like <code>ip daddr 100.64.64.65 meta l4proto { tcp, udp } th 53 redirect to 1053</code> in the destination NAT table to offer a DNS resolver that only resolves non-overridden domains for users with split network setups?

File:Multi-Node Overview.png

Snep — Wed, 28 Oct 2026 03:09:38 GMT

Snep uploaded a new version of File:Multi-Node Overview.png Adjusted interface and bridge names

New page

Diagram of the basic networking components between the US and EU node making up the CGHMN network

CGHMN-Multi-Node

Snep — Wed, 28 Oct 2026 02:48:55 GMT

WIP: Initial revision

New page

At the moment, the only way to connect up to the CGHMN is through the core VM container on the US PVE host, meaning high latency and slow throughput for members from different regions in the world, especially if both, target and destination are outside of the US.

To work around this issue, we intend on running multiple nodes across the world, interconnected through the private network feature of our hoster, OVH, or through tunnel networks between the nodes and routed with the help of OSPF and a monitoring script to keep track of the otherwise stateless Wireguard connections.
[[File:Multi-Node Overview.png|thumb|CGHMN Network Components Overview]]
The basic layout looks like this:

* Both the US and the EU Proxmox VE servers have a container called core-router. This is, as the name suggests, the core CGHMN router for each particular node, taking care of routing local core services, locally hosted member servers and the CGHMN Wireguard clients and providing NATed internet access for local guest VMs and containers.
* The Proxmox VE nodes themselves and the core router containers are connected on one flat network through the OVH private network feature called vRack, this serves as the main inter-node routing backbone.
* The Wireguard interface is mirrored on all participating nodes, so that a member can connect to one or the other simply by changing the DNS peer name, without having to change public keys or IP addresses.

In order for a single client to be able to connect to one or the other Wireguard node without reconfiguring their own IP addresses, dynamic routing through OSPF (and more specifically the bird v2 routing daemon) is used. Routes are created over the vRack network when a peer connects and, after a short timeout, removed once the peer disconnects.

Since Wireguard is a stateless protocol, we cannot rely on the Wireguard tunnel or kernel module to tell us through hooks when a peer connects or disconnects, so we run a custom Bash script which parses the output of <code>wg show wg-cghmn dump</code> on a short interval and keeps track of the peers' last handshakes and transmitted bytes counters to determine if a peer is dead or alive.

The "aliveness" of a peer is currently determined through the time of its last handshake with the server, if any, and when bytes were last received on the server from the peer, in that order. If a peer transitions from a dead state to being alive, a route is inserted for each allowed IP entry of the peer into the main sytem routing table, which is then picked up by bird and announced as a dynamic route to other Wireguard nodes on the network. If a peer transitions from alive to dead, the opposite happens, all routes on the local Wireguard interface for each allowed IP entry of the peer is removed from the main system routing table and subsequently removed by bird from the announced dynamic routes.

Should this method of detecting peers be insufficient, e.g. when it is deemed to sluggish or fragile to detect peers roaming between nodes, member @DrScriptt (on Discord) has a project called static-route-manager that adds and removes routes based on ICMP ping responses, which could be a solid alternative for detecting connected peers.

In order for the Wireguard interfaces on all nodes to stay in sync, the [https://github.com/CGHMN/wireguard-controller central Wireguard Controller] has been extended to push its Wireguard configuration to the nodes via HTTP, to a [https://github.com/CGHMN/wg-endpoint-api small PHP based API] running on each node. The node API builds a valid Wireguard configuration from the transmitted JSON payload and applies it to its local Wireguard interface via <code>wg syncconf wg-cghmn</code>. Additionally, the Wireguard API itself is configured to not manage a Wireguard interface itself, so it serves basically just as a message broker and database of sorts.

File:Multi-Node Overview.png

Snep — Wed, 28 Oct 2026 02:14:27 GMT

Snep uploaded File:Multi-Node Overview.png

New page

Diagram of the basic networking components between the US and EU node making up the CGHMN network

Services people are running

CursedSilicon — Sun, 18 Oct 2026 04:34:13 GMT

IRC bridge note

← Older revision		Revision as of 20:34, 17 October 2026
Line 26:		Line 26:
	\|no		\|no
	\|#cghmn, #cghmn-events		\|#cghmn, #cghmn-events
	\|~~Bridged onto~~ the ~~regular internet and~~ Discord		\|'''ALL channels are bridged to their Discord equivalents on the Cursed Silicon Discord'''
	\|-		\|-
	\|irc.loganius.retro		\|irc.loganius.retro