# Network infrastructure audit - 2026-07-21/22 Scope: the `skyworks` NixOS configurations for `xlab-gateway` and `skydick`, the live main gateway (`door1`, `10.0.0.1:2222`) and its `skynet-server-gateway` repository, the LDAP deployment in `skynet-server-web`, and the source policy in `nix-infra`. ## Operational cautions > [!CAUTION] > Do not activate the staged SkyDick generation outside a coordinated storage > window. The host currently has 33 established NFS/TCP channels from node1, > node2, and main, including open MySQL data files, plus a logged-in 8 TiB > iSCSI client at `10.0.200.11`. Dry activation would restart ZFS mount/share, > NFS, Samba, InfluxDB, and NSS services. ## Status labels - **LIVE + COMMITTED**: observed on the running host and committed in that host's canonical repository. - **LIVE + PROFILED**: active on the running host and selected for the next boot, but still uncommitted in this worktree. - **STAGED + BUILT**: represented by a complete NixOS closure, but deliberately not activated. - **LIVE FINDING**: observed on a running host and still requires work. ## Current state | System | State | Result | |---|---|---| | Main gateway | **LIVE + COMMITTED** | Region policy, PotPlayer DNS exception, IPv6 DNS, TSIG rotation, SSH/input hardening, and tunnel CAKE lifecycle are deployed. Repository is clean. | | Main LDAP stack | **LIVE + COMMITTED** | LDAP and Luminary credentials use Docker secrets; both containers are healthy. Commit `d507c33`. | | Xlab | **LIVE + COMMITTED, AUTH PENDING** | Both WireGuard paths, automatic recovery, persistent manual disable, CN-direct routing, trusted DNS, DHCP/RA, and restricted public recovery access are live. Polluted AliDNS fallback was removed. Cold-boot campus admission still needs registered-IP status or a dedicated credential. | | C9800 WLAN | **LIVE + SAVED** | `Skyworks` remains WPA3-SAE with PMF required; OKC and band-select are disabled to avoid the observed FlexConnect SAE PMKID reassociation failure. | | SkyDick | **STAGED + BUILT** | NixOS 25.11 remains live and healthy. The final committed 26.05 closure is built, not activated because storage clients are active. | ## Region-sensitive routing The source is `nix-infra` commit `8692bebcfaf59be065d84d064c367cb01dddc691` (`mesh/proxy: force Alibaba overseas CDN direct for intl-pinned clients`). It contains packet-capture evidence for Taobao/Tmall traffic to four Alibaba AS45102 Singapore ranges: ```text 47.246.0.0/16 43.109.0.0/16 163.181.0.0/16 139.95.0.0/16 ``` The commit contains no JD-specific prefix or packet evidence. JD appears only in a comment, so no unsupported JD range was invented during migration. ### Main gateway - **LIVE + COMMITTED**: `5b3cfc5` adds all four ranges to the direct-routing nftables set before foreign-exit selection. - **LIVE + COMMITTED**: `5e1d72f` forces `daumcdn.net` through the proxy path. EasyPrivacy had classified the PotPlayer download CDN as a tracker, causing MosDNS to return `NXDOMAIN`. A/AAAA lookup and the exact installer URL were verified after deployment. - **LIVE + COMMITTED**: `2898736` serves MosDNS over UDP/TCP on the advertised gateway ULA as well as IPv4. - **LIVE + COMMITTED**: `3b957cd` rotates the DDNS TSIG secret, retires the old key, hardens gateway SSH/input exposure, scopes Avahi, and repairs the CAKE/IFB lifecycle. `wg-sgp-qos.service` and `wg-sgp-autorate.service` are currently active, and CAKE is attached to `ifb-sgp` and `wg-outbound-sgp`. ### Xlab client policy The deployed xlab policy uses two nft interval-set layers: - APNIC-delegated CN IPv4 and IPv6 prefixes are downloaded over HTTPS, schema/size/count checked, nft syntax checked, and transactionally loaded. The last validated cache remains usable if refresh fails. The current live sets contain 8,786 IPv4 and 2,039 IPv6 prefixes. - Static regional sets contain the four Alibaba ranges, six Tsinghua/CERNET IPv4 ranges, and three CERNET IPv6 ranges. Matching LAN packets receive mark `0x10`. Rule priority 90 first looks in the main table, so they leave through `wan99.0` while its default route exists. If that default disappears, lookup continues to the existing source rule for table 1002 and exits through `wg-to-wgnet`. The earlier static `throw` design did not have this fallback and has been removed. An isolated network-namespace test loaded the rendered nft rules and proved: ```text regional_direct_v4=10 regional_direct_v6=3 healthy: 47.246.1.1 -> wan99.0 failed WAN: 47.246.1.1 -> wg-to-wgnet table 1002 ``` ## Xlab deployed closure Active system and persistent system profile as of 2026-07-22: ```text /nix/store/1b50bcwxk7dcs5rhgny4p596shc5aabi-nixos-system-xlab-gateway-26.05.20260719.fd14620 ``` Included changes: - NixOS 26.05 while preserving `system.stateVersion = "25.11"`. - APNIC-CN plus static regional direct routing with WAN-loss fallback. - Gateway-local DNS listeners at `10.253.254.1` and `fd99:23eb:1682:1::1`; DHCP and RA advertise those addresses. Their only upstreams are main MosDNS at `10.0.0.1` and `fd99:23eb:1682::1`. `FallbackDNS` is empty so resolved cannot select an unfiltered mainland resolver for foreign domains. Both WireGuard endpoints are numeric, so tunnel bootstrap has no DNS dependency. - Explicit input/source ACLs, fixed WireGuard listen ports, and WAN SSH only from main (`166.111.17.108`) or server3 (`166.111.17.81`). - Explicit `wireguard` module loading and a handshake-aware recovery timer. Networkd configures the netdevs with manual activation; `xlab-wg` waits for the campus route, recreates missing/incomplete interfaces, raises enabled tunnels, and cycles only peers whose latest handshake is stale. - Persistent administrative disable markers let `xlab-wg disable` keep either tunnel down across health checks and reboots. This replaces `wg-quick` for interfaces owned by systemd-networkd. - A numeric server3 endpoint removes the old boot-time DNS dependency. Explicit WAN UDP rules allow either peer to initiate recovery, and WAN SSH is restricted to main and server3 for break-glass access. - Rotated agenix-backed Kea TSIG key, DHCPv4 plus SLAAC-only IPv6, symmetric MSS clamping, safer CAPWAP filters, and layer3+4 bond hashing. - Periodic route/set/cache health checks. Rendered `resolved.conf`, Kea DHCP4, radvd, networkd, and nftables files were inspected. `radvd --configtest`, nftables `--check`, both generated health scripts with `bash -n`, and the complete NixOS build passed. Kea parsed the configuration through socket selection; its only test-host error was the expected absence of xlab's `bond.lan254` on the remote build machine. ### Recovered outage (2026-07-22) The failed boot journal disproved the initial missing-netdev hypothesis. Both WireGuard netdevs and their agenix secrets existed, but networkd raised the tunnels before the WAN route, DNS, and campus admission were usable. The server3 endpoint hostname then failed resolution continuously for roughly 12 hours. The old firewall did not admit inbound UDP 46961/51998, and the old generation had no handshake-aware recovery task, so peer-initiated traffic could not break the bootstrap failure. The campus authentication endpoint reports that the current admission session started at 15:28:44 CST. `wg-to-wgnet` was manually raised at 15:29:37 and handshook immediately. Xlab has no Tunet/GoAuthing unit or stored campus credential, so the missing cold-boot admission was a necessary part of the outage, not merely a WireGuard race. The new readiness service retries and recovers automatically once admission exists, but full cold-boot automation still requires either registered/open IP status or an agenix-backed dedicated campus credential. The preserved evidence remains on xlab at: ```text /var/tmp/xlab-previous-boot-full-20260722.log /var/tmp/xlab-previous-boot-network-20260722.log /var/tmp/xlab-network-boot-20260722.log ``` The new closure was activated without rebooting and tested from both the normal route and restricted public break-glass path. Activation restarted networkd; the readiness service detected stale handshakes and recovered both tunnels in about three seconds each. Further fault injection proved: - disabling `wg-to-skyworks` persists while the health service runs, then an explicit enable restores a current handshake; - deleting the complete `wg-to-skyworks` netdev causes it to be recreated, configured, raised, and handshaken; - forcing `wg-to-wgnet` down from the public recovery session causes automatic recovery and restores `10.253.254.1` reachability. The deployed CN-direct health check passed with 8,786 IPv4 and 2,039 IPv6 APNIC prefixes plus 10/3 static regional prefixes. Marked `223.5.5.5` and `47.246.1.1` route through `wan99.0`; unmarked `1.1.1.1` routes through `wg-to-wgnet`. No systemd units are failed. ### Recovered DNS pollution (2026-07-22) Xlab's resolved configuration listed main MosDNS and AliDNS as peer global servers. systemd-resolved does not treat that list as ordered failover; it had selected `223.5.5.5`, which returned non-Google addresses for `www.google.com` and similarly polluted YouTube names. This explained the reported split symptom: `google.com` resolved and pinged, but its HTTPS redirect to `www.google.com` failed. Path-MTU probes, symmetric MSS clamps, and real client TCP flows were healthy. The live and profiled configuration now uses only the two main MosDNS tunnel addresses. After removing the temporary runtime override and restarting resolved, repeated `www.google.com` queries returned Google A/AAAA records and an interface-bound HTTPS request returned HTTP 200. No systemd unit is failed. ## Skyworks WLAN authentication recovery The controller is a C9800-CL running IOS XE 17.15.5; the xlab AP is an AIR-AP3802I-H in FlexConnect mode. Authoritative post-change `show wlan id 1` output confirms that `Skyworks` is WPA3-SAE-only with CCMP, PMF required, H2E plus HNP, and FT disabled. The authentication mode itself was not changed during this recovery. Always-on WLC trace provided a concrete interoperability failure for a MacBook randomized address. It completed SAE on 5 GHz, received DHCP, moved to 2.4 GHz, then reassociated to 5 GHz and was deleted with: ```text SAE PMKID matching failed in roam case Invalid PMKID IE_VALIDATION_FAILURE type 53 ``` Another failing randomized client associated on 5 GHz but never completed the SAE exchange and timed out in L2AUTH. This matches the FlexConnect/WPA3-SAE/OKC failure family in [Cisco CSCwp20425](https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-15/release-notes/rn-17-15-9800.html); Cisco's documented workaround is to disable OKC. Enabled band-select increased the chance of cross-band reassociation. The controller is already on Cisco's [currently recommended 17.15.5 release](https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html), so there is no justified in-train firmware upgrade to substitute for the verified workaround. The WLC running configuration was backed up to `bootflash:pre-skyworks-auth-fix-20260722.cfg`. OKC and band-select were then disabled on WLAN 1, the WLAN was re-enabled, all three existing Skyworks SAE clients returned to `Run`, and `write memory` completed. The previously failing MacBook then rejoined on 5 GHz using SAE, remained associated well past its former five-second failure point, obtained `10.253.254.103`, and exchanged traffic without decrypt or policy errors. Six Skyworks SAE clients were in `Run` concurrently with no excluded clients. WPA2 transition mode remains an available compatibility fallback, but was not needed and would reduce the current security level. ## SkyDick staged closure Final build: ```text /nix/store/x3b5rnxpp37icqjr0ja91l2mhhm73igz-nixos-system-skydick-26.05.20260719.fd14620 ``` Included changes: - NixOS 26.05 with the exact-source Samba overlay rebuilt; state version stays at 25.11. - A fail-closed ZFS mount-identity gate for NFS, Samba, InfluxDB, and iSCSI. - Persistent root-only LIO state at `/var/lib/target`, with `/etc/target` as a directory symlink so `targetcli` atomic saves remain persistent. - An iSCSI guard requiring block backstores, existing devices, explicit non-wildcard portals, IQN ACLs, CHAP credentials, and LUN mappings. It also prevents `iscsi-target.service` from stopping during a live switch. - Explicit NFS children instead of pseudo-root `crossmnt`; exact media/backup clients where verified; NFSv4 pseudo-root coverage for the declared IPv6 clients. - Source-restricted nftables rules for SSH, NFS, SMB, iSCSI, InfluxDB, and node exporter. NetBIOS and unused WSD discovery are disabled. - Certificate-verified StartTLS for NSS and Samba LDAP, secret restart triggers, corrected 40 GbE ring tuning, and removal of the unnecessary Nix trusted-user grant. - Only main MosDNS at `10.0.0.1` and `fd99:23eb:1682::1` are global resolver candidates. AliDNS and direct public resolvers were removed so the staged closure cannot leak polluted answers or bypass main's split-DNS policy. The live iSCSI configuration was backed up again at `/root/backups/target/20260721T204151Z`. Serializing the live kernel state was byte-identical before and after (`babc4f7a...59650c246`), and the CHAP-aware guard passes it. `targetcli sessions detail` saying `NOT AUTHENTICATED` refers to optional mutual target authentication; initiator CHAP fields are present. Do not activate this closure until node1/node2 MySQL and all other NFS users are stopped or unmounted and the iSCSI initiator is logged out or explicitly accepted as maintenance risk. ## LDAP runtime hardening - Root-only `slapcat` backups are under `/root/backups/ldap/20260721T200920Z` on main. - Commit `d507c33` moves LDAP admin/config/readonly and Luminary bind credentials from container environment variables to Docker secrets. - LDAP and Luminary were recreated and are healthy. Container inspection confirmed no direct password variables and the expected secret mounts. - StartTLS with hostname and the staged SkyDick CA bundle succeeds; NSS group lookup and Samba `pdbedit` both pass. The live SkyDick generation still uses plaintext LDAP until its maintenance deployment. This is a deployment-state issue, not a missing source fix. ## Remaining findings 1. **P0, xlab campus admission:** the active session was established manually after boot. Prefer registering this server IP as `open` with the unit network administrator. Otherwise provision a dedicated Tunet credential through agenix and run a maintained authentication client; do not put the password in the Nix store or service command line. 2. **P0, SkyDick maintenance:** deploy only after coordinated NFS and iSCSI quiescence, then validate mounts, exports, LDAP TLS, Samba, InfluxDB, and target restore before clients reconnect. 3. **P1, NFS identity:** `ye-lw21`, `zhuyz24`, and `/srv/system/vm` retain broad subnet ACLs because the exact client contract is not yet known. Replace them with `/32` and `/128` entries after inventory. Use Kerberos or SMB when per-user authentication, rather than host identity, is required. 4. **P1, NFS/RDMA:** the live port list contains `rdma 20049` twice, but all observed sessions are TCP/2049 and no RDMA client was proven. Port 20049 is intentionally not opened by the staged firewall. Remove the unused listener or validate a real `proto=rdma` client and then scope the firewall/export. 5. **P1, monitoring secrets:** InfluxDB remains HTTP and the documented token model is shared more broadly than necessary. Move to TLS and separate writer/read-only tokens. 6. **P2, LDAP indexes:** live slapd logs report unindexed searches for `cn`, `sambaDomainName`, and `sambaSID`. The database is currently small; add indexes with a backed-up, scheduled slapindex/restart rather than an online ad hoc change. 7. **P2, main peers:** several legacy `df99` IPv6 peers and peers without recent handshakes remain. Inventory ownership before removing or migrating them; do not infer abandonment from one snapshot. 8. **P2, WLC source of truth:** the C9800 WLAN policy is live and saved, but it is not declared in this repository. Export a sanitized controller baseline or add an approved configuration-management path so OKC/band-select cannot silently return after replacement or template changes. ## Remaining execution order 1. Finish xlab cold-boot campus admission through registered-IP status or an agenix-backed dedicated credential. Retain the preserved journal and use `xlab-wg` rather than `wg-quick` for its networkd-owned tunnels. 2. Schedule SkyDick storage downtime, stop/unmount NFS clients, log out iSCSI, activate the staged closure, and perform the storage/LDAP/firewall checks. 3. Narrow the remaining NFS ACLs and address RDMA, InfluxDB TLS/tokens, LDAP indexes, and legacy main-gateway peers as separately reversible changes.