diff --git a/docs/2026-06-14-gateway-optimization-assessment.md b/docs/2026-06-14-gateway-optimization-assessment.md new file mode 100644 index 0000000..d06fe5b --- /dev/null +++ b/docs/2026-06-14-gateway-optimization-assessment.md @@ -0,0 +1,127 @@ +# Gateway (.1 / door1) optimization & feature-feasibility assessment — 2026-06-14 + +Scope: audit of `10.0.0.1` (door1 / skyw-gw) + related infra, and a feasibility +verdict for each requested feature **on the current skyworks network** (the "963 +VLAN" framing is dropped — these are assessed against the existing fabric, not a +new segment). + +## Verdict up front +Almost every requested feature is **already running or is a config-level add on the +existing hardware** — replacing `.1` is **not required** to obtain them. `.1` is a +Dell **PowerEdge R730, 2× Xeon E5-2697A v4 (32C/64T), 94 GB RAM, AES-NI, 2×25G=50G +LACP bond**; healthy (34 d uptime, 0 kernel errors, 0 failed units, ~30 °C, +conntrack 4403/524288). The bottlenecks are **not** CPU/RAM — they are (1) a +**mis-routed relay IPv6 path** and (2) **no egress QoS**. + +## Current architecture (measured) +- **Split = L3:** mosdns split-DNS → nft sets `proxy_domain_v4` (~3790) / `_v6` + (~2745) → mangle mark `0xdead` → `ip rule … fwmark 0xdead → table 200 → default + dev wg-outbound-sgp`. CN goes direct via `br-wan` (campus public 166.111.17.108). +- **Abroad exit = single** `wg-outbound-sgp` WireGuard tunnel to Oracle Cloud SGP + ("Pulsar", peer `RW7T27…`). +- **DPI L7 engine** (TCP TPROXY :12345 + QUIC NFQUEUE/nDPI) is **built and nft-wired** + (`@cdn_dpi_v4 tcp dport 443 tproxy to :12345`, historical 3.5M/3.7M pkt hits) but + **inert**: `cdn_dpi` set empty, daemon not running. +- **DoH/DoT cage partially present:** nft drops WAN-side :53 and **rejects client + DoH/DoT/DoQ** to the known public resolvers (1.1.1.1/8.8.8.8/9.9.9.9/quad9/adguard/ + opendns) from br-lan + wg-server. +- **QoS: NONE** — `wg-outbound-sgp` and `br-wan` are both `noqueue`. (Global default + qdisc is `fq`, CC is **bbr**.) MSS is already clamped on the SGP path. +- **Abroad traffic is legitimate** (flow-exporter, cumulative): Cloudflare 5.76 GB, + Anthropic/Claude 0.83 GB, Fastly, Microsoft, Akamai, Google, Notion; top dst-country + CA/US (Cloudflare anycast). **No BitTorrent / 簧片 saturation signature today.** +- **skydick RoCEv2 NFS is LIVE:** `mlx4_0/1` RDMA ACTIVE on a 2×40G=80G bond, + `/proc/fs/nfsd/portlist` shows **rdma 20049** + tcp 2049. + +## KEY FINDING — the relay's IPv6 path is the international bottleneck +| Endpoint | RTT | Path | +|---|---|---| +| IPv6 (in use) `[2603:c02b:300:b800:0:c539:a6b8:29b9]` | **347 ms** | CERNET2 → **lax2.he.net (Los Angeles)** → SingTel → Oracle | +| IPv4 `140.245.61.139` | **65 ms** | CERNET → Cogent → **Equinix Singapore** → Oracle | + +The relay is in Singapore; CERNET2's **IPv6** route to it goes via Los Angeles (HE.net +transit) — a 5.3× penalty. The proxy tunnel currently adds ~140 ms *over* even the +direct campus path (1.1.1.1/8.8.8.8 ≈ 204 ms direct). + +**Cost: IPv4 is NOT charged here (confirmed 2026-06-14)** — so the 65 ms IPv4 path is a +free win. The only blocker is **relay-side ingress**: door1's `wg set … endpoint +140.245.61.139:27913` handshakes but **roams back to IPv6** because the relay never +receives the IPv4 packets. Root cause is **NOT the relay host firewall** (its nat +redirect `iifname eth0 udp dport 20000-40000 redirect to :47390` is in `table inet` → +dual-stack, accepts IPv4) — it's **Oracle Cloud's security list / NSG**, which allows +the WG port range over IPv6 but not IPv4 (ICMP works → 65 ms, but UDP 27913 is dropped +before the host). + +**→ Action to unlock 65 ms (one OCI console step):** add an ingress rule on Pulsar's VCN +Security List/NSG: **IPv4 source `0.0.0.0/0` (or door1 `166.111.17.108/32`), UDP dest +`20000-40000`**. Then set `.env` `WG_OUTBOUND_SGP_PEER_ENDPOINT=140.245.61.139:27913` +and re-apply — verified handshake works; RTT drops 347 → 65 ms. (Live `wg set` was +tested + reverted; `.env` is gitignored/host-local. The IPv6-only `ops/sgp-port-lottery.sh` +becomes unnecessary on the clean IPv4 path.) + +**Monitoring added:** blackbox-icmp probes for both relay endpoints (v4 + v6) + a +Grafana timeseries panel *"Abroad relay RTT (underlay) — IPv4 vs IPv6 path"* (Services → +WireGuard) so the 347→65 ms drop is visible and future path regressions get caught. + +## Feasibility matrix (current network) +| # | Feature | Verdict | Notes | +|---|---|---|---| +| 1 | Better international access | ⚠️ Feasible, high-impact | relay-IPv6 reroute (free, via better-peered relay) or IPv4 (metered). +BBR/MTU on relay, +2nd relay failover. | +| 2 | Domestic not degraded | ✅ Preserved by design | CN egress is direct via br-wan; untouched by any of these. | +| 3 | 分流 not hijacked by BT/簧片 | ⚠️ Feasible (currently unprotected, not abused) | add CAKE + behavioral P2P marking; BT already mostly direct (peers are bare IPs, no DNS tag). | +| 4 | Not cost-explosive | ✅ Mostly $0 | all but a 2nd relay (1 cheap VPS) and metered-IPv4 are config. | +| 5 | L3-below vs L7-above split | ✅ Hybrid: keep L3 spine | L7 only as a scalpel (QUIC gate / BT id); ECH is depreciating SNI-DPI. | +| 6 | RoCEv2 NFS intranet | ✅ Already live | RoCEv2 **SMB** (SMB Direct) = not readily (Samba/ksmbd gap); SMB Multichannel already saturates the tier. | +| 7 | Windows/Android 神秘网络战 | ✅ Largely handled, improvable | DoH/DoT cage exists; broaden (DoH IP lists, NCSI/gstatic whitelist, Teredo/QUIC, IPv6 decision). | +| 8 | 10G encrypted intranet | ⚠️ Feasible | fabric is 40–80G; **MACsec** for L2 line-rate (RoCE-friendly); WireGuard 10G needs GSO/GRO + multi-flow. | + +## Prioritized playbook (highest leverage first) +**1. International — relay path (free if done right).** Benchmark IPv6 RTT from door1 to +candidate Asian relays (HK/JP/SG providers with good CERNET2 v6 peering); migrate the +tunnel to whichever gives ~<100 ms over **free IPv6**. Only use IPv4 (65 ms) if billing +allows. Add **BBR (v1; BBRv3 is still out-of-tree as of late 2025) + fq** on the relay, +verify wg MTU 1420 + MSS clamp (already present). Add a **second relay** (`wg-outbound-2`) +with health-checked failover (reuse the existing route-keeper). No MPTCP (TCP-over-TCP +meltdown; not worth it). + +**2. Exit fairness — make "BT can't take the exit down" structural.** +- **CAKE** on `wg-outbound-sgp` egress with `triple-isolate` (per-host + per-flow), + static rate ≈ 95–99 % of the relay's real bandwidth (need that number). Two tins: + interactive (default `0xdead`) vs bulk (`0xbeef`, large abroad downloads). +- **Behavioral P2P marking** with nftables `ct count`/meters (P2P = connection + explosion, *not* port 6881 which is randomized): clients over ~200 live conns → + mark → **forced direct / throttled, never the tunnel**. + +**3. DNS cage hardening (turns L3 split from "leaks" to "enforced").** Single ordered +nft chain: redirect **all** :53 incl. **LAN-to-LAN** → block :853 DoT/DoQ (allow your +resolver first) → block DoH by **maintained IP+domain list** (hagezi / jameshas) → +block hardcoded 8.8.8.8/1.1.1.1/9.9.9.9 on 53/443/853 → **whitelist** NCSI +(`*.msftconnecttest.com`, `dns.msftncsi.com`→131.107.255.255) and Android +`connectivitycheck.gstatic.com` so probes don't flap. Kill **Teredo (UDP 3544)** + +6to4 (proto-41). **IPv6 decision:** simplest correct default = **no RA** on client +VLANs (everything IPv4 → existing split applies); else fully dual-stack the split +(populate v6 sets + NAT66 + v6 geo backstop). **Geo/ASN IP-set backstop** so DNS-bypass +traffic to known-abroad prefixes still routes correctly at L3 speed. + +**4. Selective QUIC suppression (the one good L7 job).** For force-through targets, drop +UDP/443 so clients fall back to TCP you can see. Keep DPI **off as a router**. + +## Do-NOT list +- Don't make the gateway an **RDMA citizen** — RoCEv2 is an L2 storage-VLAN property + (PFC pri-3 + watchdog, ECN/DCQCN, jumbo 9000); keep the gateway *out* of the RDMA + path. 963/other-VLAN clients needing RDMA storage get a storage-VLAN leg; others use + TCP NFS / SMB Multichannel. +- Don't turn the **DPI L7 engine** on as the primary splitter (can't do 10G of splicing; + ECH erodes accuracy). +- Don't build a **custom BBRv3 kernel** on the relay (out-of-tree maintenance ≫ gain). +- Don't deploy **MPTCP/MPUDP** bonding across relays (control overhead, TCP-over-TCP). + +## Replace-.1 gate +`.1` is healthy and over-spec'd; the case to replace it is **operational risk** (single +box, single relay, no QoS), all fixable in place. Only cut over to a new box when it +**demonstrably** delivers all 7 criteria on the bench; until then, harden `.1`. + +--- +*Derived from live measurement on door1 (2026-06-14) + a sourced architecture study +(see commit message / chat for citations: WireGuard-10G, CAKE, nft meters, ECH status, +RoCE DCQCN, SMB Direct, DoH/NCSI mitigation, BBRv3 status).*