# Network infrastructure audit - 2026-07-21 Scope: the `skyworks` NixOS configurations for `xlab-gateway` and `skydick`, the live main gateway (`door1` / `10.0.0.1`) and its separate `skynet-server-gateway` repository, and the source policy in `nix-infra`. > [!CAUTION] > **DO NOT rebuild, deploy, or otherwise activate a new NixOS generation on > `skydick` until the iSCSI persistence problem below is fixed.** The current > live LIO target can survive in the kernel while an activation replaces its > saved configuration with `{}`. A later reboot or target restart can then > remove the target and its client-visible LUN. ## Evidence labels - **LIVE + COMMITTED**: observed on the running host and represented by a commit in that host's canonical repository. - **REPO + EVALUATED**: present in this `skyworks` worktree and accepted by Nix evaluation, but not claimed to be deployed. - **LIVE FINDING**: observed on a running host during this audit. - **RECOMMENDATION**: proposed remediation; not yet implemented unless stated. `nix flake check --offline --all-systems --no-build` passed on 2026-07-21 for both NixOS configurations. The xlab changes below were activated with deploy-rs, committed as `688ac05`, and passed live route, set, lifecycle, and counter checks. ## Completed policy work ### Alibaba, Taobao, and Tmall region-sensitive routing The source was `nix-infra` commit `8692bebcfaf59be065d84d064c367cb01dddc691` (`mesh/proxy: force Alibaba overseas CDN direct for intl-pinned clients`). Packet capture had shown about 21,000 packets per affected Taobao/Tmall session going to four AS45102 Alibaba Singapore ranges which are not in APNIC-CN. Sending them through a foreign exit caused Alibaba geo/risk controls to see a foreign source. DNS tagging cannot reliably catch this path because the apps use HTTPDNS/GSLB. Source references: - `nix-infra/hosts/door-pek/networking.nix:630-648` documents the capture and declares the four ranges; `:813-819` places the direct decision before the per-client international pin. - `nix-infra/hosts/door-sha/firewall.nix:218-230` declares the same set; `:272-276` applies it in a comment that mentions Alibaba/JD. The commit adds no JD-specific range or packet evidence beyond the same Alibaba set. The migrated policy is: ```text 47.246.0.0/16 43.109.0.0/16 163.181.0.0/16 139.95.0.0/16 ``` - **LIVE + COMMITTED, main gateway**: commit `5b3cfc5` adds the four ranges to `data/infra/nftables.netif`. `nft -c`, live set membership, an unmarked `br-wan` route, and container health were verified. - **LIVE + COMMITTED, xlab**: throw routes in routing table 1002 make those non-APNIC-CN destinations fall through to the local campus WAN (`hosts/xlab-gateway/networking.nix:393-402`). - **LIVE + COMMITTED, xlab**: APNIC-delegated CN IPv4 and IPv6 destinations are loaded into nft interval sets, marked `0x10`, and looked up in the main table before the existing WireGuard rule (`hosts/xlab-gateway/networking.nix:206-218`, `:412-427`, `:7-143`, `:544-600`). The updater strictly validates downloaded fields, retains the last validated file, rejects undersized data, checks the nft transaction before applying it, and leaves the old `.1` path intact if no set is available. - **LIVE + COMMITTED, xlab**: forwarded packets claiming a source outside the xlab IPv4 `/24` or IPv6 `/64` are dropped before the trusted-LAN accept (`hosts/xlab-gateway/networking.nix:170-175`, `:197-203`). The live xlab service loaded 8,786 IPv4 and 2,039 IPv6 prefixes. Marked CN route probes selected `wan99.0`; unmarked foreign probes selected `wg-to-wgnet`; all four Alibaba ranges selected the campus WAN. The nft CN counter increased on client traffic, the source guards were present, and the host reported no failed units after activation. A deliberate nftables reload repopulated both sets from the validated cache in under one second; during that transient empty-set interval, packets safely follow the old `.1` path. These four `/16`s are the observed Alibaba exception set, not a complete or future-proof inventory of every JD or Alibaba CDN. Keep packet counters and capture evidence as the criterion for adding another static exception. ### PotPlayer resolution - **LIVE + COMMITTED, main gateway**: commit `5e1d72f` adds `daumcdn.net` to `data/infra/mosdns/force_proxy.txt`. EasyPrivacy's `tracker_domain.txt` blanket match had turned the PotPlayer download domain into `NXDOMAIN`. After restarting mosdns, A and AAAA resolution and an HTTP/2 200 response for the exact `PotPlayerSetup64.exe` URL were verified. ### Main-gateway IPv6 DNS - **LIVE + COMMITTED, main gateway**: commit `2898736f8f42afd7bfd17ce25ef2047c27b6fd58` adds MosDNS UDP and TCP listeners on `[fd99:23eb:1682::1]:53`. The address and both sockets, IPv6 UDP local and Taobao resolution, IPv6 TCP DNSSEC with the AD bit, IPv4 regression, and container health were verified. Only MosDNS was restarted. ## P0 - fix before routine operations ### 1. Skydick iSCSI state is erased by the declarative default **Verified configuration:** `hosts/skydick/datapool.nix:712-713` only enables `services.target`; it does not set `services.target.config`. In the pinned NixOS module, that option defaults to `{}` and manages `/etc/target/saveconfig.json` as mode `0600`. The documented imperative `targetcli ... saveconfig` workflow (`hosts/skydick/DATAPOOL.md:516-558`) therefore writes a file that the next NixOS activation can replace with `{}`. **Impact:** an already loaded kernel target can make this look healthy until a reboot, `iscsi-target` restart, or LIO clear. The restored target will then be empty. **Required remediation:** first back up the live JSON and record `targetcli ls`; then make restoration declarative or copy a persistent, encrypted runtime file into place before `iscsi-target` starts. Do not put CHAP secrets directly in a Nix expression because they would enter the world-readable Nix store. Add an activation assertion that refuses to replace a non-empty live target with an empty definition, then test restore and client login before allowing a rebuild. ### 2. The NFS pseudo-root can expose unlisted mounted datasets **Verified configuration:** `/srv` is exported to all of `10.0.0.0/16` with `crossmnt` (`hosts/skydick/datapool.nix:479-480`). `crossmnt` implicitly exports mounted child filesystems with the parent's options. This can expose unlisted children such as monitoring, Time Machine, private, or future ZFS datasets and can undermine narrower child exports. The surrounding ACLs are also broader than their prose suggests: - all LAN hosts can write `/srv/media` as UID 900 (`:482-485`); - all LAN hosts map to the owner of `ye-lw21` and `zhuyz24` (`:487-500`); - all LAN hosts receive `no_root_squash` on backup and VM trees (`:502-505`). This contradicts "Only you can access your tree" in `hosts/skydick/DATAPOOL.md:293-304`. Remove `crossmnt` from the `fsid=0` pseudo-root, export every intended child explicitly, and reduce client matches to exact IPv4 `/32` and IPv6 `/128` identities. Use SMB or NFS Kerberos (`sec=krb5p`) where user authentication is required. ## P1 - high-priority reliability and security ### Main gateway 1. **CAKE/IFB is not attached after WireGuard recreation (LIVE FINDING).** The related units report healthy while the qdisc/filter is absent, so tunnel fairness can silently disappear. Bind QoS lifecycle to the WireGuard device recreation path, make setup idempotent, and alert on `tc` state rather than unit state alone. 2. **Fifteen legacy `df99` peers bypass IPv6 policy; several peers appear stale (LIVE FINDING).** Migrate active peers to classified source prefixes, remove expired peers, and default-deny unclassified IPv6 sources. 3. **Public SSH permits passwords and forwarding, and its limiter is global (LIVE FINDING).** A remote actor can consume the shared allowance and lock out operators. Prefer VPN/source-restricted key-only access, disable forwarding unless explicitly required, and use per-source limits. 4. **WAN IPv6 lacks equivalent source anti-spoofing (LIVE FINDING).** Add the IPv6 reserved, ULA, link-local, multicast, and owned-prefix source checks at the raw/prerouting boundary. ### Xlab gateway 1. **The TSIG DDNS secret is committed in plaintext.** It is embedded at `hosts/xlab-gateway/dhcp.nix:172-178`. Treat it as compromised: rotate the key on xlab and the authoritative server, move it to agenix, and render a root-only runtime Kea fragment. 2. **The new source-spoof guards cover forwarding but not router input.** The input chain accepts every service from the entire LAN, management VLAN, and both WireGuard interfaces (`hosts/xlab-gateway/networking.nix:178-194`). Add the source validation before those accepts and replace full-interface trust with explicit DHCP, DNS/DDNS, SSH, ICMP, and WireGuard service rules. 3. **CN-direct routing needs monitoring and a controlled failover test.** The normal-path route and counter checks passed after deployment. Add alerts on APNIC cache age, set size, and the presence of both policy rules. In a maintenance window, confirm the intended table-1002 fallback when the local WAN default is unavailable. Registry allocation is a routing heuristic, not exact geolocation. ### Skydick 1. **Storage services fail open onto placeholder directories.** Tmpfiles creates the `/srv` tree (`hosts/skydick/datapool.nix:298-338`), while `dick-zfs-properties` exits successfully when pool `dick` is missing and only orders itself before NFS/Samba (`:340-368`). Evaluated NFS and Samba units have no hard mount requirement. Gate NFS, Samba, InfluxDB, and iSCSI behind a unit that asserts the expected ZFS datasets and mount identities. 2. **LDAP binds and credentials cross the LAN in plaintext.** NSS uses `ldap://` with TLS disabled (`hosts/skydick/default.nix:326-340`); Samba's LDAP passdb also disables SSL and strong authentication (`hosts/skydick/datapool.nix:524-540`). Move both to validated StartTLS or LDAPS before rotating the bind and admin credentials. 3. **Service exposure is host-wide rather than source/interface scoped.** NFS, SMB, iSCSI, InfluxDB, and node-exporter ports are globally allowed (`hosts/skydick/datapool.nix:715-731`, `modules/influxdb.nix:33-62`, `modules/monitoring.nix:166-173`, `:296-297`). Restrict storage protocols to exact storage clients/VLANs and monitoring ports to the monitoring host. Enable TLS for InfluxDB and replace the shared fleet-wide operator token documented in `hosts/skydick/README.md:20-42` with per-writer tokens and a read-only Grafana token. 4. **NFS/RDMA is only partially wired.** Live inspection found the mlx4 RDMA devices active and `/proc/fs/nfsd/portlist` containing `rdma 20049` twice, but no active client RDMA mount was confirmed. The listener is declared at `hosts/skydick/datapool.nix:441-469`; port 20049 is absent from the firewall at `:715-731`, and NFS/RDMA clients require the export's `insecure` option because they do not use a reserved source port. Scope TCP/UDP 20049 and that export option to exact clients, reconcile the duplicate writer, and prove `proto=rdma` from a client mount rather than inferring it from the server listener. 5. **The pinned NixOS release is out of security support.** `flake.nix:5` selects `nixos-25.11`; both hosts correctly keep `system.stateVersion = "25.11"` (`hosts/xlab-gateway/default.nix:107`, `hosts/skydick/default.nix:401`). NixOS 25.11 reached EOL after 2026-06-30; [NixOS 26.05](https://nixos.org/blog/announcements/2026/nixos-2605/) is supported through 2026-12-31. Upgrade the input to 26.05 without changing `system.stateVersion`, and specifically rebuild/test the exact-source Samba overlay at `hosts/skydick/datapool.nix:252-279`. ## P2 - cleanup and optimization - **Main gateway:** bind Avahi only to intended LAN interfaces and remove stale discovery ports; deduplicate policy rules; pin container images by digest; enable Docker live-restore where compatible; and add a second, health-checked foreign relay instead of retaining a single exit failure domain. - **Xlab bonding:** `balance-xor` has no explicit transmit hash policy (`hosts/xlab-gateway/networking.nix:266-275`). Routed traffic can collapse onto one member with a layer-2 hash. Validate the switch LAG and use a layer3+4 hash (or 802.3ad with matching switch configuration), then measure per-member distribution. - **Xlab CAPWAP workaround:** `capwap-df-clear` deletes and recreates the whole ingress qdisc (`hosts/xlab-gateway/default.nix:87-104`). Move to stable `clsact`/owned filter handles so restarting the unit cannot erase unrelated ingress policy. - **Skydick NIC tuning:** the active bond uses `enp130s0` and `enp130s0d1` (`hosts/skydick/default.nix:79-100`), but ring tuning still targets the retired `enp4s0f0np0` and `enp4s0f1np1` (`:191-194`). Retarget it and record before/after drop, latency, and IRQ-affinity metrics. - **Skydick discovery:** `samba-wsdd` is enabled with `openFirewall = false` (`hosts/skydick/datapool.nix:707-710`), while UDP 3702/TCP 5357 are absent from the firewall. Either open them only on the client LAN or disable WSD. - **Skydick RoCE QoS:** configuration explicitly provides DSCP marking without end-to-end PFC (`hosts/skydick/datapool.nix:404-410`). Complete and test PFC, ECN/congestion control, MTU, and watchdog configuration at every hop before calling the fabric lossless. - **Privilege boundary:** `ye-lw21` is a Nix `trusted-user` (`modules/common.nix:7-10`) even though the account is documented as password-sudo only (`modules/users.nix:23-27`). Nix trusted users are effectively root-capable; remove the entry unless that is intentional. - **Secret rotation:** add restart/reload triggers for the Samba LDAP seed and Telegraf token (`hosts/skydick/datapool.nix:371-389`, `modules/monitoring.nix:88-99`) so rotating an agenix secret changes the running credential without manual intervention. ## Execution order 1. Back up and make skydick iSCSI state activation-safe. Only then allow a skydick rebuild. 2. Remove the NFS pseudo-root `crossmnt` exposure and narrow NFS client ACLs. 3. Add a client-path monitor for the repaired main-gateway IPv6 DNS listener, then repair CAKE/IFB lifecycle and close the `df99` bypasses. 4. Rotate the exposed xlab TSIG key and harden router input/source validation. 5. Monitor the deployed xlab CN-direct policy and run its WAN-failure test in a maintenance window; retain packet capture as the basis for any future Alibaba or JD exception ranges. 6. Restrict skydick service exposure, enable LDAP/InfluxDB TLS, and add ZFS mount guards. 7. Upgrade the NixOS input to 26.05 and run host-specific build and service regression tests before deployment.