# Gateway (.1 / door1) optimization & feature-feasibility assessment — 2026-06-14

Scope: audit of `10.0.0.1` (door1 / skyw-gw) + related infra, and a feasibility
verdict for each requested feature **on the current skyworks network** (the "963
VLAN" framing is dropped — these are assessed against the existing fabric, not a
new segment).

## Verdict up front
Almost every requested feature is **already running or is a config-level add on the
existing hardware** — replacing `.1` is **not required** to obtain them. `.1` is a
Dell **PowerEdge R730, 2× Xeon E5-2697A v4 (32C/64T), 94 GB RAM, AES-NI, 2×25G=50G
LACP bond**; healthy (34 d uptime, 0 kernel errors, 0 failed units, ~30 °C,
conntrack 4403/524288). The bottlenecks are **not** CPU/RAM — they are (1) a
**mis-routed relay IPv6 path** and (2) **no egress QoS**.

## Current architecture (measured)
- **Split = L3:** mosdns split-DNS → nft sets `proxy_domain_v4` (~3790) / `_v6`
  (~2745) → mangle mark `0xdead` → `ip rule … fwmark 0xdead → table 200 → default
  dev wg-outbound-sgp`. CN goes direct via `br-wan` (campus public 166.111.17.108).
- **Abroad exit = single** `wg-outbound-sgp` WireGuard tunnel to Oracle Cloud SGP
  ("Pulsar", peer `RW7T27…`).
- **DPI L7 engine** (TCP TPROXY :12345 + QUIC NFQUEUE/nDPI) is **built and nft-wired**
  (`@cdn_dpi_v4 tcp dport 443 tproxy to :12345`, historical 3.5M/3.7M pkt hits) but
  **inert**: `cdn_dpi` set empty, daemon not running.
- **DoH/DoT cage partially present:** nft drops WAN-side :53 and **rejects client
  DoH/DoT/DoQ** to the known public resolvers (1.1.1.1/8.8.8.8/9.9.9.9/quad9/adguard/
  opendns) from br-lan + wg-server.
- **QoS: NONE** — `wg-outbound-sgp` and `br-wan` are both `noqueue`. (Global default
  qdisc is `fq`, CC is **bbr**.) MSS is already clamped on the SGP path.
- **Abroad traffic is legitimate** (flow-exporter, cumulative): Cloudflare 5.76 GB,
  Anthropic/Claude 0.83 GB, Fastly, Microsoft, Akamai, Google, Notion; top dst-country
  CA/US (Cloudflare anycast). **No BitTorrent / 簧片 saturation signature today.**
- **skydick RoCEv2 NFS is LIVE:** `mlx4_0/1` RDMA ACTIVE on a 2×40G=80G bond,
  `/proc/fs/nfsd/portlist` shows **rdma 20049** + tcp 2049.

## KEY FINDING — the relay's IPv6 path is the international bottleneck
| Endpoint | RTT | Path |
|---|---|---|
| IPv6 (in use) `[2603:c02b:300:b800:0:c539:a6b8:29b9]` | **347 ms** | CERNET2 → **lax2.he.net (Los Angeles)** → SingTel → Oracle |
| IPv4 `140.245.61.139` | **65 ms** | CERNET → Cogent → **Equinix Singapore** → Oracle |

The relay is in Singapore; CERNET2's **IPv6** route to it goes via Los Angeles (HE.net
transit) — a 5.3× penalty. The proxy tunnel currently adds ~140 ms *over* even the
direct campus path (1.1.1.1/8.8.8.8 ≈ 204 ms direct).

**Cost: IPv4 is NOT charged here (confirmed 2026-06-14)** — so the 65 ms IPv4 path is a
free win. The only blocker is **relay-side ingress**: door1's `wg set … endpoint
140.245.61.139:27913` handshakes but **roams back to IPv6** because the relay never
receives the IPv4 packets. Root cause is **NOT the relay host firewall** (its nat
redirect `iifname eth0 udp dport 20000-40000 redirect to :47390` is in `table inet` →
dual-stack, accepts IPv4) — it's **Oracle Cloud's security list / NSG**, which allows
the WG port range over IPv6 but not IPv4 (ICMP works → 65 ms, but UDP 27913 is dropped
before the host).

**→ Action to unlock 65 ms (one OCI console step):** add an ingress rule on Pulsar's VCN
Security List/NSG: **IPv4 source `0.0.0.0/0` (or door1 `166.111.17.108/32`), UDP dest
`20000-40000`**. Then set `.env` `WG_OUTBOUND_SGP_PEER_ENDPOINT=140.245.61.139:27913`
and re-apply — verified handshake works; RTT drops 347 → 65 ms. (Live `wg set` was
tested + reverted; `.env` is gitignored/host-local. The IPv6-only `ops/sgp-port-lottery.sh`
becomes unnecessary on the clean IPv4 path.)

**Monitoring added:** blackbox-icmp probes for both relay endpoints (v4 + v6) + a
Grafana timeseries panel *"Abroad relay RTT (underlay) — IPv4 vs IPv6 path"* (Services →
WireGuard) so the 347→65 ms drop is visible and future path regressions get caught.

## Feasibility matrix (current network)
| # | Feature | Verdict | Notes |
|---|---|---|---|
| 1 | Better international access | ⚠️ Feasible, high-impact | relay-IPv6 reroute (free, via better-peered relay) or IPv4 (metered). +BBR/MTU on relay, +2nd relay failover. |
| 2 | Domestic not degraded | ✅ Preserved by design | CN egress is direct via br-wan; untouched by any of these. |
| 3 | 分流 not hijacked by BT/簧片 | ⚠️ Feasible (currently unprotected, not abused) | add CAKE + behavioral P2P marking; BT already mostly direct (peers are bare IPs, no DNS tag). |
| 4 | Not cost-explosive | ✅ Mostly $0 | all but a 2nd relay (1 cheap VPS) and metered-IPv4 are config. |
| 5 | L3-below vs L7-above split | ✅ Hybrid: keep L3 spine | L7 only as a scalpel (QUIC gate / BT id); ECH is depreciating SNI-DPI. |
| 6 | RoCEv2 NFS intranet | ✅ Already live | RoCEv2 **SMB** (SMB Direct) = not readily (Samba/ksmbd gap); SMB Multichannel already saturates the tier. |
| 7 | Windows/Android 神秘网络战 | ✅ Largely handled, improvable | DoH/DoT cage exists; broaden (DoH IP lists, NCSI/gstatic whitelist, Teredo/QUIC, IPv6 decision). |
| 8 | 10G encrypted intranet | ⚠️ Feasible | fabric is 40–80G; **MACsec** for L2 line-rate (RoCE-friendly); WireGuard 10G needs GSO/GRO + multi-flow. |

## Prioritized playbook (highest leverage first)
**1. International — relay path (free if done right).** Benchmark IPv6 RTT from door1 to
candidate Asian relays (HK/JP/SG providers with good CERNET2 v6 peering); migrate the
tunnel to whichever gives ~<100 ms over **free IPv6**. Only use IPv4 (65 ms) if billing
allows. Add **BBR (v1; BBRv3 is still out-of-tree as of late 2025) + fq** on the relay,
verify wg MTU 1420 + MSS clamp (already present). Add a **second relay** (`wg-outbound-2`)
with health-checked failover (reuse the existing route-keeper). No MPTCP (TCP-over-TCP
meltdown; not worth it).

**2. Exit fairness — make "BT can't take the exit down" structural.**
- **CAKE** on `wg-outbound-sgp` egress with `triple-isolate` (per-host + per-flow),
  static rate ≈ 95–99 % of the relay's real bandwidth (need that number). Two tins:
  interactive (default `0xdead`) vs bulk (`0xbeef`, large abroad downloads).
- **Behavioral P2P marking** with nftables `ct count`/meters (P2P = connection
  explosion, *not* port 6881 which is randomized): clients over ~200 live conns →
  mark → **forced direct / throttled, never the tunnel**.

**3. DNS cage hardening (turns L3 split from "leaks" to "enforced").** Single ordered
nft chain: redirect **all** :53 incl. **LAN-to-LAN** → block :853 DoT/DoQ (allow your
resolver first) → block DoH by **maintained IP+domain list** (hagezi / jameshas) →
block hardcoded 8.8.8.8/1.1.1.1/9.9.9.9 on 53/443/853 → **whitelist** NCSI
(`*.msftconnecttest.com`, `dns.msftncsi.com`→131.107.255.255) and Android
`connectivitycheck.gstatic.com` so probes don't flap. Kill **Teredo (UDP 3544)** +
6to4 (proto-41). **IPv6 decision:** simplest correct default = **no RA** on client
VLANs (everything IPv4 → existing split applies); else fully dual-stack the split
(populate v6 sets + NAT66 + v6 geo backstop). **Geo/ASN IP-set backstop** so DNS-bypass
traffic to known-abroad prefixes still routes correctly at L3 speed.

**4. Selective QUIC suppression (the one good L7 job).** For force-through targets, drop
UDP/443 so clients fall back to TCP you can see. Keep DPI **off as a router**.

## Do-NOT list
- Don't make the gateway an **RDMA citizen** — RoCEv2 is an L2 storage-VLAN property
  (PFC pri-3 + watchdog, ECN/DCQCN, jumbo 9000); keep the gateway *out* of the RDMA
  path. 963/other-VLAN clients needing RDMA storage get a storage-VLAN leg; others use
  TCP NFS / SMB Multichannel.
- Don't turn the **DPI L7 engine** on as the primary splitter (can't do 10G of splicing;
  ECH erodes accuracy).
- Don't build a **custom BBRv3 kernel** on the relay (out-of-tree maintenance ≫ gain).
- Don't deploy **MPTCP/MPUDP** bonding across relays (control overhead, TCP-over-TCP).

## Replace-.1 gate
`.1` is healthy and over-spec'd; the case to replace it is **operational risk** (single
box, single relay, no QoS), all fixable in place. Only cut over to a new box when it
**demonstrably** delivers all 7 criteria on the bench; until then, harden `.1`.

---
*Derived from live measurement on door1 (2026-06-14) + a sourced architecture study
(see commit message / chat for citations: WireGuard-10G, CAKE, nft meters, ECH status,
RoCE DCQCN, SMB Direct, DoH/NCSI mitigation, BBRv3 status).*