Network infrastructure audit - 2026-07-21/22

Scope: the skyworks NixOS configurations for xlab-gateway and
skydick, the live main gateway (door1, 10.0.0.1:2222) and its
skynet-server-gateway repository, the LDAP deployment in
skynet-server-web, and the source policy in nix-infra.

Operational cautions

[!CAUTION]
Do not activate the staged SkyDick generation outside a coordinated storage
window. The host currently has 33 established NFS/TCP channels from node1,
node2, and main, including open MySQL data files, plus a logged-in 8 TiB
iSCSI client at 10.0.200.11. Dry activation would restart ZFS mount/share,
NFS, Samba, InfluxDB, and NSS services.

Status labels

• LIVE + COMMITTED: observed on the running host and committed in that
  host's canonical repository.
• LIVE + PROFILED: active on the running host and selected for the next
  boot, but still uncommitted in this worktree.
• STAGED + BUILT: represented by a complete NixOS closure, but deliberately
  not activated.
• LIVE FINDING: observed on a running host and still requires work.

Current state

Region-sensitive routing

The source is nix-infra commit
8692bebcfaf59be065d84d064c367cb01dddc691 (mesh/proxy: force Alibaba overseas CDN direct for intl-pinned clients). It contains packet-capture
evidence for Taobao/Tmall traffic to four Alibaba AS45102 Singapore ranges:

47.246.0.0/16
43.109.0.0/16
163.181.0.0/16
139.95.0.0/16

The commit contains no JD-specific prefix or packet evidence. JD appears only
in a comment, so no unsupported JD range was invented during migration.

Main gateway

• LIVE + COMMITTED: 5b3cfc5 adds all four ranges to the direct-routing
  nftables set before foreign-exit selection.
• LIVE + COMMITTED: 5e1d72f forces daumcdn.net through the proxy path.
  EasyPrivacy had classified the PotPlayer download CDN as a tracker, causing
  MosDNS to return NXDOMAIN. A/AAAA lookup and the exact installer URL were
  verified after deployment.
• LIVE + COMMITTED: 2898736 serves MosDNS over UDP/TCP on the advertised
  gateway ULA as well as IPv4.
• LIVE + COMMITTED: 3b957cd rotates the DDNS TSIG secret, retires the old
  key, hardens gateway SSH/input exposure, scopes Avahi, and repairs the
  CAKE/IFB lifecycle. wg-sgp-qos.service and
  wg-sgp-autorate.service are currently active, and CAKE is attached to
  ifb-sgp and wg-outbound-sgp.

Xlab client policy

The deployed xlab policy uses two nft interval-set layers:

• APNIC-delegated CN IPv4 and IPv6 prefixes are downloaded over HTTPS,
  schema/size/count checked, nft syntax checked, and transactionally loaded.
  The last validated cache remains usable if refresh fails. The current live
  sets contain 8,786 IPv4 and 2,039 IPv6 prefixes.
• Static regional sets contain the four Alibaba ranges, six Tsinghua/CERNET
  IPv4 ranges, and three CERNET IPv6 ranges.

Matching LAN packets receive mark 0x10. Rule priority 90 first looks in the
main table, so they leave through wan99.0 while its default route exists. If
that default disappears, lookup continues to the existing source rule for
table 1002 and exits through wg-to-wgnet. The earlier static throw design
did not have this fallback and has been removed.

An isolated network-namespace test loaded the rendered nft rules and proved:

regional_direct_v4=10 regional_direct_v6=3
healthy:    47.246.1.1 -> wan99.0
failed WAN: 47.246.1.1 -> wg-to-wgnet table 1002

Xlab deployed closure

Active system and persistent system profile as of 2026-07-22:

/nix/store/1b50bcwxk7dcs5rhgny4p596shc5aabi-nixos-system-xlab-gateway-26.05.20260719.fd14620

Included changes:

• NixOS 26.05 while preserving system.stateVersion = "25.11".
• APNIC-CN plus static regional direct routing with WAN-loss fallback.
• Gateway-local DNS listeners at 10.253.254.1 and
  fd99:23eb:1682:1::1; DHCP and RA advertise those addresses. Their only
  upstreams are main MosDNS at 10.0.0.1 and fd99:23eb:1682::1.
  FallbackDNS is empty so resolved cannot select an unfiltered mainland
  resolver for foreign domains. Both WireGuard endpoints are numeric, so
  tunnel bootstrap has no DNS dependency.
• Explicit input/source ACLs, fixed WireGuard listen ports, and WAN SSH only
  from main (166.111.17.108) or server3 (166.111.17.81).
• Explicit wireguard module loading and a handshake-aware recovery timer.
  Networkd configures the netdevs with manual activation; xlab-wg waits for
  the campus route, recreates missing/incomplete interfaces, raises enabled
  tunnels, and cycles only peers whose latest handshake is stale.
• Persistent administrative disable markers let xlab-wg disable keep either
  tunnel down across health checks and reboots. This replaces wg-quick for
  interfaces owned by systemd-networkd.
• A numeric server3 endpoint removes the old boot-time DNS dependency. Explicit
  WAN UDP rules allow either peer to initiate recovery, and WAN SSH is restricted
  to main and server3 for break-glass access.
• Rotated agenix-backed Kea TSIG key, DHCPv4 plus SLAAC-only IPv6, symmetric
  MSS clamping, safer CAPWAP filters, and layer3+4 bond hashing.
• Periodic route/set/cache health checks.

Rendered resolved.conf, Kea DHCP4, radvd, networkd, and nftables files were
inspected. radvd --configtest, nftables --check, both generated health
scripts with bash -n, and the complete NixOS build passed. Kea parsed the
configuration through socket selection; its only test-host error was the
expected absence of xlab's bond.lan254 on the remote build machine.

Recovered outage (2026-07-22)

The failed boot journal disproved the initial missing-netdev hypothesis. Both
WireGuard netdevs and their agenix secrets existed, but networkd raised the
tunnels before the WAN route, DNS, and campus admission were usable. The
server3 endpoint hostname then failed resolution continuously for roughly 12
hours. The old firewall did not admit inbound UDP 46961/51998, and the old
generation had no handshake-aware recovery task, so peer-initiated traffic
could not break the bootstrap failure.

The campus authentication endpoint reports that the current admission session
started at 15:28:44 CST. wg-to-wgnet was manually raised at 15:29:37 and
handshook immediately. Xlab has no Tunet/GoAuthing unit or stored campus
credential, so the missing cold-boot admission was a necessary part of the
outage, not merely a WireGuard race. The new readiness service retries and
recovers automatically once admission exists, but full cold-boot automation
still requires either registered/open IP status or an agenix-backed dedicated
campus credential.

The preserved evidence remains on xlab at:

/var/tmp/xlab-previous-boot-full-20260722.log
/var/tmp/xlab-previous-boot-network-20260722.log
/var/tmp/xlab-network-boot-20260722.log

The new closure was activated without rebooting and tested from both the normal
route and restricted public break-glass path. Activation restarted networkd;
the readiness service detected stale handshakes and recovered both tunnels in
about three seconds each. Further fault injection proved:

• disabling wg-to-skyworks persists while the health service runs, then an
  explicit enable restores a current handshake;
• deleting the complete wg-to-skyworks netdev causes it to be recreated,
  configured, raised, and handshaken;
• forcing wg-to-wgnet down from the public recovery session causes automatic
  recovery and restores 10.253.254.1 reachability.

The deployed CN-direct health check passed with 8,786 IPv4 and 2,039 IPv6 APNIC
prefixes plus 10/3 static regional prefixes. Marked 223.5.5.5 and
47.246.1.1 route through wan99.0; unmarked 1.1.1.1 routes through
wg-to-wgnet. No systemd units are failed.

Recovered DNS pollution (2026-07-22)

Xlab's resolved configuration listed main MosDNS and AliDNS as peer global
servers. systemd-resolved does not treat that list as ordered failover; it had
selected 223.5.5.5, which returned non-Google addresses for
www.google.com and similarly polluted YouTube names. This explained the
reported split symptom: google.com resolved and pinged, but its HTTPS redirect
to www.google.com failed. Path-MTU probes, symmetric MSS clamps, and real
client TCP flows were healthy.

The live and profiled configuration now uses only the two main MosDNS tunnel
addresses. After removing the temporary runtime override and restarting
resolved, repeated www.google.com queries returned Google A/AAAA records and
an interface-bound HTTPS request returned HTTP 200. No systemd unit is failed.

Skyworks WLAN authentication recovery

The controller is a C9800-CL running IOS XE 17.15.5; the xlab AP is an
AIR-AP3802I-H in FlexConnect mode. Authoritative post-change show wlan id 1
output confirms that Skyworks is WPA3-SAE-only with CCMP, PMF required,
H2E plus HNP, and FT disabled. The authentication mode itself was not changed
during this recovery.

Always-on WLC trace provided a concrete interoperability failure for a
MacBook randomized address. It completed SAE on 5 GHz, received DHCP, moved to
2.4 GHz, then reassociated to 5 GHz and was deleted with:

SAE PMKID matching failed in roam case
Invalid PMKID
IE_VALIDATION_FAILURE type 53

Another failing randomized client associated on 5 GHz but never completed the
SAE exchange and timed out in L2AUTH. This matches the FlexConnect/WPA3-SAE/OKC
failure family in
Cisco CSCwp20425;
Cisco's documented workaround is to disable OKC. Enabled band-select increased
the chance of cross-band reassociation. The controller is already on Cisco's
currently recommended 17.15.5 release,
so there is no justified in-train firmware upgrade to substitute for the
verified workaround.

The WLC running configuration was backed up to
bootflash:pre-skyworks-auth-fix-20260722.cfg. OKC and band-select were then
disabled on WLAN 1, the WLAN was re-enabled, all three existing Skyworks SAE
clients returned to Run, and write memory completed. The previously
failing MacBook then rejoined on 5 GHz using SAE, remained associated well past
its former five-second failure point, obtained 10.253.254.103, and exchanged
traffic without decrypt or policy errors. Six Skyworks SAE clients were in
Run concurrently with no excluded clients. WPA2 transition mode remains an
available compatibility fallback, but was not needed and would reduce the
current security level.

SkyDick staged closure

Final build:

/nix/store/x3b5rnxpp37icqjr0ja91l2mhhm73igz-nixos-system-skydick-26.05.20260719.fd14620

Included changes:

• NixOS 26.05 with the exact-source Samba overlay rebuilt; state version stays
  at 25.11.
• A fail-closed ZFS mount-identity gate for NFS, Samba, InfluxDB, and iSCSI.
• Persistent root-only LIO state at /var/lib/target, with /etc/target as a
  directory symlink so targetcli atomic saves remain persistent.
• An iSCSI guard requiring block backstores, existing devices, explicit
  non-wildcard portals, IQN ACLs, CHAP credentials, and LUN mappings. It also
  prevents iscsi-target.service from stopping during a live switch.
• Explicit NFS children instead of pseudo-root crossmnt; exact media/backup
  clients where verified; NFSv4 pseudo-root coverage for the declared IPv6
  clients.
• Source-restricted nftables rules for SSH, NFS, SMB, iSCSI, InfluxDB, and
  node exporter. NetBIOS and unused WSD discovery are disabled.
• Certificate-verified StartTLS for NSS and Samba LDAP, secret restart
  triggers, corrected 40 GbE ring tuning, and removal of the unnecessary Nix
  trusted-user grant.
• Only main MosDNS at 10.0.0.1 and fd99:23eb:1682::1 are global resolver
  candidates. AliDNS and direct public resolvers were removed so the staged
  closure cannot leak polluted answers or bypass main's split-DNS policy.

The live iSCSI configuration was backed up again at
/root/backups/target/20260721T204151Z. Serializing the live kernel state was
byte-identical before and after (babc4f7a...59650c246), and the CHAP-aware
guard passes it. targetcli sessions detail saying NOT AUTHENTICATED refers
to optional mutual target authentication; initiator CHAP fields are present.

Do not activate this closure until node1/node2 MySQL and all other NFS users
are stopped or unmounted and the iSCSI initiator is logged out or explicitly
accepted as maintenance risk.

LDAP runtime hardening

• Root-only slapcat backups are under
  /root/backups/ldap/20260721T200920Z on main.
• Commit d507c33 moves LDAP admin/config/readonly and Luminary bind
  credentials from container environment variables to Docker secrets.
• LDAP and Luminary were recreated and are healthy. Container inspection
  confirmed no direct password variables and the expected secret mounts.
• StartTLS with hostname and the staged SkyDick CA bundle succeeds; NSS group
  lookup and Samba pdbedit both pass.

The live SkyDick generation still uses plaintext LDAP until its maintenance
deployment. This is a deployment-state issue, not a missing source fix.

Remaining findings

1. P0, xlab campus admission: the active session was established manually
   after boot. Prefer registering this server IP as open with the unit network
   administrator. Otherwise provision a dedicated Tunet credential through
   agenix and run a maintained authentication client; do not put the password
   in the Nix store or service command line.
2. P0, SkyDick maintenance: deploy only after coordinated NFS and iSCSI
   quiescence, then validate mounts, exports, LDAP TLS, Samba, InfluxDB, and
   target restore before clients reconnect.
3. P1, NFS identity: ye-lw21, zhuyz24, and /srv/system/vm retain
   broad subnet ACLs because the exact client contract is not yet known.
   Replace them with /32 and /128 entries after inventory. Use Kerberos or
   SMB when per-user authentication, rather than host identity, is required.
4. P1, NFS/RDMA: the live port list contains rdma 20049 twice, but all
   observed sessions are TCP/2049 and no RDMA client was proven. Port 20049 is
   intentionally not opened by the staged firewall. Remove the unused listener
   or validate a real proto=rdma client and then scope the firewall/export.
5. P1, monitoring secrets: InfluxDB remains HTTP and the documented token
   model is shared more broadly than necessary. Move to TLS and separate
   writer/read-only tokens.
6. P2, LDAP indexes: live slapd logs report unindexed searches for cn,
   sambaDomainName, and sambaSID. The database is currently small; add
   indexes with a backed-up, scheduled slapindex/restart rather than an online
   ad hoc change.
7. P2, main peers: several legacy df99 IPv6 peers and peers without
   recent handshakes remain. Inventory ownership before removing or migrating
   them; do not infer abandonment from one snapshot.
8. P2, WLC source of truth: the C9800 WLAN policy is live and saved, but it
   is not declared in this repository. Export a sanitized controller baseline
   or add an approved configuration-management path so OKC/band-select cannot
   silently return after replacement or template changes.

Remaining execution order

1. Finish xlab cold-boot campus admission through registered-IP status or an
   agenix-backed dedicated credential. Retain the preserved journal and use
   xlab-wg rather than wg-quick for its networkd-owned tunnels.
2. Schedule SkyDick storage downtime, stop/unmount NFS clients, log out iSCSI,
   activate the staged closure, and perform the storage/LDAP/firewall checks.
3. Narrow the remaining NFS ACLs and address RDMA, InfluxDB TLS/tokens, LDAP
   indexes, and legacy main-gateway peers as separately reversible changes.