Gateway (.1 / door1) optimization & feature-feasibility assessment — 2026-06-14

Scope: audit of 10.0.0.1 (door1 / skyw-gw) + related infra, and a feasibility
verdict for each requested feature on the current skyworks network (the "963
VLAN" framing is dropped — these are assessed against the existing fabric, not a
new segment).

Verdict up front

Almost every requested feature is already running or is a config-level add on the
existing hardware — replacing .1 is not required to obtain them. .1 is a
Dell PowerEdge R730, 2× Xeon E5-2697A v4 (32C/64T), 94 GB RAM, AES-NI, 2×25G=50G
LACP bond; healthy (34 d uptime, 0 kernel errors, 0 failed units, ~30 °C,
conntrack 4403/524288). The bottlenecks are not CPU/RAM — they are (1) a
mis-routed relay IPv6 path and (2) no egress QoS.

Current architecture (measured)

• Split = L3: mosdns split-DNS → nft sets proxy_domain_v4 (~3790) / _v6
  (~2745) → mangle mark 0xdead → ip rule … fwmark 0xdead → table 200 → default dev wg-outbound-sgp. CN goes direct via br-wan (campus public 166.111.17.108).
• Abroad exit = single wg-outbound-sgp WireGuard tunnel to Oracle Cloud SGP
  ("Pulsar", peer RW7T27…).
• DPI L7 engine (TCP TPROXY :12345 + QUIC NFQUEUE/nDPI) is built and nft-wired
  (@cdn_dpi_v4 tcp dport 443 tproxy to :12345, historical 3.5M/3.7M pkt hits) but
  inert: cdn_dpi set empty, daemon not running.
• DoH/DoT cage partially present: nft drops WAN-side :53 and rejects client
  DoH/DoT/DoQ to the known public resolvers (1.1.1.1/8.8.8.8/9.9.9.9/quad9/adguard/
  opendns) from br-lan + wg-server.
• QoS: NONE — wg-outbound-sgp and br-wan are both noqueue. (Global default
  qdisc is fq, CC is bbr.) MSS is already clamped on the SGP path.
• Abroad traffic is legitimate (flow-exporter, cumulative): Cloudflare 5.76 GB,
  Anthropic/Claude 0.83 GB, Fastly, Microsoft, Akamai, Google, Notion; top dst-country
  CA/US (Cloudflare anycast). No BitTorrent / 簧片 saturation signature today.
• skydick RoCEv2 NFS is LIVE: mlx4_0/1 RDMA ACTIVE on a 2×40G=80G bond,
  /proc/fs/nfsd/portlist shows rdma 20049 + tcp 2049.

KEY FINDING — the relay's IPv6 path is the international bottleneck

The relay is in Singapore; CERNET2's IPv6 route to it goes via Los Angeles (HE.net
transit) — a 5.3× penalty. The proxy tunnel currently adds ~140 ms over even the
direct campus path (1.1.1.1/8.8.8.8 ≈ 204 ms direct).

Cost: IPv4 is NOT charged here (confirmed 2026-06-14) — so the 65 ms IPv4 path is a
free win. The only blocker is relay-side ingress: door1's wg set … endpoint 140.245.61.139:27913 handshakes but roams back to IPv6 because the relay never
receives the IPv4 packets. Root cause is NOT the relay host firewall (its nat
redirect iifname eth0 udp dport 20000-40000 redirect to :47390 is in table inet →
dual-stack, accepts IPv4) — it's Oracle Cloud's security list / NSG, which allows
the WG port range over IPv6 but not IPv4 (ICMP works → 65 ms, but UDP 27913 is dropped
before the host).

→ Action to unlock 65 ms (one OCI console step): add an ingress rule on Pulsar's VCN
Security List/NSG: IPv4 source 0.0.0.0/0 (or door1 166.111.17.108/32), UDP dest
20000-40000. Then set .env WG_OUTBOUND_SGP_PEER_ENDPOINT=140.245.61.139:27913
and re-apply — verified handshake works; RTT drops 347 → 65 ms. (Live wg set was
tested + reverted; .env is gitignored/host-local. The IPv6-only ops/sgp-port-lottery.sh
becomes unnecessary on the clean IPv4 path.)

Monitoring added: blackbox-icmp probes for both relay endpoints (v4 + v6) + a
Grafana timeseries panel "Abroad relay RTT (underlay) — IPv4 vs IPv6 path" (Services →
WireGuard) so the 347→65 ms drop is visible and future path regressions get caught.

Feasibility matrix (current network)

Prioritized playbook (highest leverage first)

1. International — relay path (free if done right). Benchmark IPv6 RTT from door1 to
candidate Asian relays (HK/JP/SG providers with good CERNET2 v6 peering); migrate the
tunnel to whichever gives ~<100 ms over free IPv6. Only use IPv4 (65 ms) if billing
allows. Add BBR (v1; BBRv3 is still out-of-tree as of late 2025) + fq on the relay,
verify wg MTU 1420 + MSS clamp (already present). Add a second relay (wg-outbound-2)
with health-checked failover (reuse the existing route-keeper). No MPTCP (TCP-over-TCP
meltdown; not worth it).

2. Exit fairness — make "BT can't take the exit down" structural.

• CAKE on wg-outbound-sgp egress with triple-isolate (per-host + per-flow),
  static rate ≈ 95–99 % of the relay's real bandwidth (need that number). Two tins:
  interactive (default 0xdead) vs bulk (0xbeef, large abroad downloads).
• Behavioral P2P marking with nftables ct count/meters (P2P = connection
  explosion, not port 6881 which is randomized): clients over ~200 live conns →
  mark → forced direct / throttled, never the tunnel.

3. DNS cage hardening (turns L3 split from "leaks" to "enforced"). Single ordered
nft chain: redirect all :53 incl. LAN-to-LAN → block :853 DoT/DoQ (allow your
resolver first) → block DoH by maintained IP+domain list (hagezi / jameshas) →
block hardcoded 8.8.8.8/1.1.1.1/9.9.9.9 on 53/443/853 → whitelist NCSI
(*.msftconnecttest.com, dns.msftncsi.com→131.107.255.255) and Android
connectivitycheck.gstatic.com so probes don't flap. Kill Teredo (UDP 3544) +
6to4 (proto-41). IPv6 decision: simplest correct default = no RA on client
VLANs (everything IPv4 → existing split applies); else fully dual-stack the split
(populate v6 sets + NAT66 + v6 geo backstop). Geo/ASN IP-set backstop so DNS-bypass
traffic to known-abroad prefixes still routes correctly at L3 speed.

4. Selective QUIC suppression (the one good L7 job). For force-through targets, drop
UDP/443 so clients fall back to TCP you can see. Keep DPI off as a router.

Do-NOT list

• Don't make the gateway an RDMA citizen — RoCEv2 is an L2 storage-VLAN property
  (PFC pri-3 + watchdog, ECN/DCQCN, jumbo 9000); keep the gateway out of the RDMA
  path. 963/other-VLAN clients needing RDMA storage get a storage-VLAN leg; others use
  TCP NFS / SMB Multichannel.
• Don't turn the DPI L7 engine on as the primary splitter (can't do 10G of splicing;
  ECH erodes accuracy).
• Don't build a custom BBRv3 kernel on the relay (out-of-tree maintenance ≫ gain).
• Don't deploy MPTCP/MPUDP bonding across relays (control overhead, TCP-over-TCP).

Replace-.1 gate

.1 is healthy and over-spec'd; the case to replace it is operational risk (single
box, single relay, no QoS), all fixable in place. Only cut over to a new box when it
demonstrably delivers all 7 criteria on the bench; until then, harden .1.

Derived from live measurement on door1 (2026-06-14) + a sourced architecture study
(see commit message / chat for citations: WireGuard-10G, CAKE, nft meters, ECH status,
RoCE DCQCN, SMB Direct, DoH/NCSI mitigation, BBRv3 status).