diff --git a/docs/2026-07-21-network-infrastructure-audit.md b/docs/2026-07-21-network-infrastructure-audit.md new file mode 100644 index 0000000..7f639bf --- /dev/null +++ b/docs/2026-07-21-network-infrastructure-audit.md @@ -0,0 +1,273 @@ +# Network infrastructure audit - 2026-07-21 + +Scope: the `skyworks` NixOS configurations for `xlab-gateway` and `skydick`, +the live main gateway (`door1` / `10.0.0.1`) and its separate +`skynet-server-gateway` repository, and the source policy in `nix-infra`. + +> [!CAUTION] +> **DO NOT rebuild, deploy, or otherwise activate a new NixOS generation on +> `skydick` until the iSCSI persistence problem below is fixed.** The current +> live LIO target can survive in the kernel while an activation replaces its +> saved configuration with `{}`. A later reboot or target restart can then +> remove the target and its client-visible LUN. + +## Evidence labels + +- **LIVE + COMMITTED**: observed on the running host and represented by a + commit in that host's canonical repository. +- **REPO + EVALUATED**: present in this `skyworks` worktree and accepted by + Nix evaluation, but not claimed to be deployed. +- **LIVE FINDING**: observed on a running host during this audit. +- **RECOMMENDATION**: proposed remediation; not yet implemented unless stated. + +`nix flake check --offline --all-systems --no-build` passed on 2026-07-21 for +both NixOS configurations. The xlab changes below were activated with +deploy-rs, committed as `688ac05`, and passed live route, set, lifecycle, and +counter checks. + +## Completed policy work + +### Alibaba, Taobao, and Tmall region-sensitive routing + +The source was `nix-infra` commit +`8692bebcfaf59be065d84d064c367cb01dddc691` (`mesh/proxy: force Alibaba +overseas CDN direct for intl-pinned clients`). Packet capture had shown about +21,000 packets per affected Taobao/Tmall session going to four AS45102 Alibaba +Singapore ranges which are not in APNIC-CN. Sending them through a foreign +exit caused Alibaba geo/risk controls to see a foreign source. DNS tagging +cannot reliably catch this path because the apps use HTTPDNS/GSLB. + +Source references: + +- `nix-infra/hosts/door-pek/networking.nix:630-648` documents the capture and + declares the four ranges; `:813-819` places the direct decision before the + per-client international pin. +- `nix-infra/hosts/door-sha/firewall.nix:218-230` declares the same set; + `:272-276` applies it in a comment that mentions Alibaba/JD. The commit adds + no JD-specific range or packet evidence beyond the same Alibaba set. + +The migrated policy is: + +```text +47.246.0.0/16 +43.109.0.0/16 +163.181.0.0/16 +139.95.0.0/16 +``` + +- **LIVE + COMMITTED, main gateway**: commit `5b3cfc5` adds the four ranges to + `data/infra/nftables.netif`. `nft -c`, live set membership, an unmarked + `br-wan` route, and container health were verified. +- **LIVE + COMMITTED, xlab**: throw routes in routing table 1002 make those + non-APNIC-CN destinations fall through to the local campus WAN + (`hosts/xlab-gateway/networking.nix:393-402`). +- **LIVE + COMMITTED, xlab**: APNIC-delegated CN IPv4 and IPv6 destinations + are loaded into nft interval sets, marked `0x10`, and looked up in the main + table before the existing WireGuard rule + (`hosts/xlab-gateway/networking.nix:206-218`, `:412-427`, `:7-143`, + `:544-600`). The updater strictly validates downloaded fields, retains the + last validated file, rejects undersized data, checks the nft transaction + before applying it, and leaves the old `.1` path intact if no set is + available. +- **LIVE + COMMITTED, xlab**: forwarded packets claiming a source outside the + xlab IPv4 `/24` or IPv6 `/64` are dropped before the trusted-LAN accept + (`hosts/xlab-gateway/networking.nix:170-175`, `:197-203`). + +The live xlab service loaded 8,786 IPv4 and 2,039 IPv6 prefixes. Marked CN +route probes selected `wan99.0`; unmarked foreign probes selected +`wg-to-wgnet`; all four Alibaba ranges selected the campus WAN. The nft CN +counter increased on client traffic, the source guards were present, and the +host reported no failed units after activation. A deliberate nftables reload +repopulated both sets from the validated cache in under one second; during +that transient empty-set interval, packets safely follow the old `.1` path. + +These four `/16`s are the observed Alibaba exception set, not a complete or +future-proof inventory of every JD or Alibaba CDN. Keep packet counters and +capture evidence as the criterion for adding another static exception. + +### PotPlayer resolution + +- **LIVE + COMMITTED, main gateway**: commit `5e1d72f` adds `daumcdn.net` to + `data/infra/mosdns/force_proxy.txt`. EasyPrivacy's `tracker_domain.txt` + blanket match had turned the PotPlayer download domain into `NXDOMAIN`. + After restarting mosdns, A and AAAA resolution and an HTTP/2 200 response + for the exact `PotPlayerSetup64.exe` URL were verified. + +### Main-gateway IPv6 DNS + +- **LIVE + COMMITTED, main gateway**: commit + `2898736f8f42afd7bfd17ce25ef2047c27b6fd58` adds MosDNS UDP and TCP listeners + on `[fd99:23eb:1682::1]:53`. The address and both sockets, IPv6 UDP local and + Taobao resolution, IPv6 TCP DNSSEC with the AD bit, IPv4 regression, and + container health were verified. Only MosDNS was restarted. + +## P0 - fix before routine operations + +### 1. Skydick iSCSI state is erased by the declarative default + +**Verified configuration:** `hosts/skydick/datapool.nix:712-713` only enables +`services.target`; it does not set `services.target.config`. In the pinned +NixOS module, that option defaults to `{}` and manages +`/etc/target/saveconfig.json` as mode `0600`. The documented imperative +`targetcli ... saveconfig` workflow (`hosts/skydick/DATAPOOL.md:516-558`) +therefore writes a file that the next NixOS activation can replace with `{}`. + +**Impact:** an already loaded kernel target can make this look healthy until a +reboot, `iscsi-target` restart, or LIO clear. The restored target will then be +empty. + +**Required remediation:** first back up the live JSON and record `targetcli +ls`; then make restoration declarative or copy a persistent, encrypted runtime +file into place before `iscsi-target` starts. Do not put CHAP secrets directly +in a Nix expression because they would enter the world-readable Nix store. Add +an activation assertion that refuses to replace a non-empty live target with +an empty definition, then test restore and client login before allowing a +rebuild. + +### 2. The NFS pseudo-root can expose unlisted mounted datasets + +**Verified configuration:** `/srv` is exported to all of `10.0.0.0/16` with +`crossmnt` (`hosts/skydick/datapool.nix:479-480`). `crossmnt` implicitly exports +mounted child filesystems with the parent's options. This can expose unlisted +children such as monitoring, Time Machine, private, or future ZFS datasets and +can undermine narrower child exports. + +The surrounding ACLs are also broader than their prose suggests: + +- all LAN hosts can write `/srv/media` as UID 900 (`:482-485`); +- all LAN hosts map to the owner of `ye-lw21` and `zhuyz24` (`:487-500`); +- all LAN hosts receive `no_root_squash` on backup and VM trees (`:502-505`). + +This contradicts "Only you can access your tree" in +`hosts/skydick/DATAPOOL.md:293-304`. Remove `crossmnt` from the `fsid=0` +pseudo-root, export every intended child explicitly, and reduce client matches +to exact IPv4 `/32` and IPv6 `/128` identities. Use SMB or NFS Kerberos +(`sec=krb5p`) where user authentication is required. + +## P1 - high-priority reliability and security + +### Main gateway + +1. **CAKE/IFB is not attached after WireGuard recreation (LIVE FINDING).** The + related units report healthy while the qdisc/filter is absent, so tunnel + fairness can silently disappear. Bind QoS lifecycle to the WireGuard device + recreation path, make setup idempotent, and alert on `tc` state rather than + unit state alone. +2. **Fifteen legacy `df99` peers bypass IPv6 policy; several peers appear + stale (LIVE FINDING).** Migrate active peers to classified source prefixes, + remove expired peers, and default-deny unclassified IPv6 sources. +3. **Public SSH permits passwords and forwarding, and its limiter is global + (LIVE FINDING).** A remote actor can consume the shared allowance and lock + out operators. Prefer VPN/source-restricted key-only access, disable + forwarding unless explicitly required, and use per-source limits. +4. **WAN IPv6 lacks equivalent source anti-spoofing (LIVE FINDING).** Add the + IPv6 reserved, ULA, link-local, multicast, and owned-prefix source checks at + the raw/prerouting boundary. + +### Xlab gateway + +1. **The TSIG DDNS secret is committed in plaintext.** It is embedded at + `hosts/xlab-gateway/dhcp.nix:172-178`. Treat it as compromised: rotate the + key on xlab and the authoritative server, move it to agenix, and render a + root-only runtime Kea fragment. +2. **The new source-spoof guards cover forwarding but not router input.** The + input chain accepts every service from the entire LAN, management VLAN, and + both WireGuard interfaces (`hosts/xlab-gateway/networking.nix:178-194`). Add + the source validation before those accepts and replace full-interface trust + with explicit DHCP, DNS/DDNS, SSH, ICMP, and WireGuard service rules. +3. **CN-direct routing needs monitoring and a controlled failover test.** The + normal-path route and counter checks passed after deployment. Add alerts on + APNIC cache age, set size, and the presence of both policy rules. In a + maintenance window, confirm the intended table-1002 fallback when the local + WAN default is unavailable. Registry allocation is a routing heuristic, + not exact geolocation. + +### Skydick + +1. **Storage services fail open onto placeholder directories.** Tmpfiles + creates the `/srv` tree (`hosts/skydick/datapool.nix:298-338`), while + `dick-zfs-properties` exits successfully when pool `dick` is missing and + only orders itself before NFS/Samba (`:340-368`). Evaluated NFS and Samba + units have no hard mount requirement. Gate NFS, Samba, InfluxDB, and iSCSI + behind a unit that asserts the expected ZFS datasets and mount identities. +2. **LDAP binds and credentials cross the LAN in plaintext.** NSS uses + `ldap://` with TLS disabled (`hosts/skydick/default.nix:326-340`); Samba's + LDAP passdb also disables SSL and strong authentication + (`hosts/skydick/datapool.nix:524-540`). Move both to validated StartTLS or + LDAPS before rotating the bind and admin credentials. +3. **Service exposure is host-wide rather than source/interface scoped.** NFS, + SMB, iSCSI, InfluxDB, and node-exporter ports are globally allowed + (`hosts/skydick/datapool.nix:715-731`, `modules/influxdb.nix:33-62`, + `modules/monitoring.nix:166-173`, `:296-297`). Restrict storage protocols to + exact storage clients/VLANs and monitoring ports to the monitoring host. + Enable TLS for InfluxDB and replace the shared fleet-wide operator token + documented in `hosts/skydick/README.md:20-42` with per-writer tokens and a + read-only Grafana token. +4. **NFS/RDMA is only partially wired.** Live inspection found the mlx4 RDMA + devices active and `/proc/fs/nfsd/portlist` containing `rdma 20049` twice, + but no active client RDMA mount was confirmed. The listener is declared at + `hosts/skydick/datapool.nix:441-469`; port 20049 is absent from the firewall + at `:715-731`, and NFS/RDMA clients require the export's `insecure` option + because they do not use a reserved source port. Scope TCP/UDP 20049 and that + export option to exact clients, reconcile the duplicate writer, and prove + `proto=rdma` from a client mount rather than inferring it from the server + listener. +5. **The pinned NixOS release is out of security support.** `flake.nix:5` + selects `nixos-25.11`; both hosts correctly keep `system.stateVersion = + "25.11"` (`hosts/xlab-gateway/default.nix:107`, + `hosts/skydick/default.nix:401`). NixOS 25.11 reached EOL after 2026-06-30; + [NixOS 26.05](https://nixos.org/blog/announcements/2026/nixos-2605/) is + supported through 2026-12-31. Upgrade the input to 26.05 without changing + `system.stateVersion`, and specifically rebuild/test the exact-source Samba + overlay at `hosts/skydick/datapool.nix:252-279`. + +## P2 - cleanup and optimization + +- **Main gateway:** bind Avahi only to intended LAN interfaces and remove stale + discovery ports; deduplicate policy rules; pin container images by digest; + enable Docker live-restore where compatible; and add a second, health-checked + foreign relay instead of retaining a single exit failure domain. +- **Xlab bonding:** `balance-xor` has no explicit transmit hash policy + (`hosts/xlab-gateway/networking.nix:266-275`). Routed traffic can collapse + onto one member with a layer-2 hash. Validate the switch LAG and use a + layer3+4 hash (or 802.3ad with matching switch configuration), then measure + per-member distribution. +- **Xlab CAPWAP workaround:** `capwap-df-clear` deletes and recreates the whole + ingress qdisc (`hosts/xlab-gateway/default.nix:87-104`). Move to stable + `clsact`/owned filter handles so restarting the unit cannot erase unrelated + ingress policy. +- **Skydick NIC tuning:** the active bond uses `enp130s0` and `enp130s0d1` + (`hosts/skydick/default.nix:79-100`), but ring tuning still targets the + retired `enp4s0f0np0` and `enp4s0f1np1` (`:191-194`). Retarget it and record + before/after drop, latency, and IRQ-affinity metrics. +- **Skydick discovery:** `samba-wsdd` is enabled with `openFirewall = false` + (`hosts/skydick/datapool.nix:707-710`), while UDP 3702/TCP 5357 are absent + from the firewall. Either open them only on the client LAN or disable WSD. +- **Skydick RoCE QoS:** configuration explicitly provides DSCP marking without + end-to-end PFC (`hosts/skydick/datapool.nix:404-410`). Complete and test PFC, + ECN/congestion control, MTU, and watchdog configuration at every hop before + calling the fabric lossless. +- **Privilege boundary:** `ye-lw21` is a Nix `trusted-user` + (`modules/common.nix:7-10`) even though the account is documented as + password-sudo only (`modules/users.nix:23-27`). Nix trusted users are + effectively root-capable; remove the entry unless that is intentional. +- **Secret rotation:** add restart/reload triggers for the Samba LDAP seed and + Telegraf token (`hosts/skydick/datapool.nix:371-389`, + `modules/monitoring.nix:88-99`) so rotating an agenix secret changes the + running credential without manual intervention. + +## Execution order + +1. Back up and make skydick iSCSI state activation-safe. Only then allow a + skydick rebuild. +2. Remove the NFS pseudo-root `crossmnt` exposure and narrow NFS client ACLs. +3. Add a client-path monitor for the repaired main-gateway IPv6 DNS listener, + then repair CAKE/IFB lifecycle and close the `df99` bypasses. +4. Rotate the exposed xlab TSIG key and harden router input/source validation. +5. Monitor the deployed xlab CN-direct policy and run its WAN-failure test in + a maintenance window; retain packet capture as the basis for any future + Alibaba or JD exception ranges. +6. Restrict skydick service exposure, enable LDAP/InfluxDB TLS, and add ZFS + mount guards. +7. Upgrade the NixOS input to 26.05 and run host-specific build and service + regression tests before deployment.