Network infrastructure audit - 2026-07-21

Scope: the skyworks NixOS configurations for xlab-gateway and skydick,
the live main gateway (door1 / 10.0.0.1) and its separate
skynet-server-gateway repository, and the source policy in nix-infra.

[!CAUTION]
DO NOT rebuild, deploy, or otherwise activate a new NixOS generation on
skydick until the iSCSI persistence problem below is fixed. The current
live LIO target can survive in the kernel while an activation replaces its
saved configuration with {}. A later reboot or target restart can then
remove the target and its client-visible LUN.

Evidence labels

LIVE + COMMITTED: observed on the running host and represented by a
commit in that host's canonical repository.
REPO + EVALUATED: present in this skyworks worktree and accepted by
Nix evaluation, but not claimed to be deployed.
LIVE FINDING: observed on a running host during this audit.
RECOMMENDATION: proposed remediation; not yet implemented unless stated.

nix flake check --offline --all-systems --no-build passed on 2026-07-21 for
both NixOS configurations. The xlab changes below were activated with
deploy-rs, committed as 688ac05, and passed live route, set, lifecycle, and
counter checks.

Completed policy work

Alibaba, Taobao, and Tmall region-sensitive routing

The source was nix-infra commit
8692bebcfaf59be065d84d064c367cb01dddc691 (mesh/proxy: force Alibaba overseas CDN direct for intl-pinned clients). Packet capture had shown about
21,000 packets per affected Taobao/Tmall session going to four AS45102 Alibaba
Singapore ranges which are not in APNIC-CN. Sending them through a foreign
exit caused Alibaba geo/risk controls to see a foreign source. DNS tagging
cannot reliably catch this path because the apps use HTTPDNS/GSLB.

Source references:

nix-infra/hosts/door-pek/networking.nix:630-648 documents the capture and
declares the four ranges; :813-819 places the direct decision before the
per-client international pin.
nix-infra/hosts/door-sha/firewall.nix:218-230 declares the same set;
:272-276 applies it in a comment that mentions Alibaba/JD. The commit adds
no JD-specific range or packet evidence beyond the same Alibaba set.

The migrated policy is:

47.246.0.0/16
43.109.0.0/16
163.181.0.0/16
139.95.0.0/16

LIVE + COMMITTED, main gateway: commit 5b3cfc5 adds the four ranges to
data/infra/nftables.netif. nft -c, live set membership, an unmarked
br-wan route, and container health were verified.
LIVE + COMMITTED, xlab: throw routes in routing table 1002 make those
non-APNIC-CN destinations fall through to the local campus WAN
(hosts/xlab-gateway/networking.nix:393-402).
LIVE + COMMITTED, xlab: APNIC-delegated CN IPv4 and IPv6 destinations
are loaded into nft interval sets, marked 0x10, and looked up in the main
table before the existing WireGuard rule
(hosts/xlab-gateway/networking.nix:206-218, :412-427, :7-143,
:544-600). The updater strictly validates downloaded fields, retains the
last validated file, rejects undersized data, checks the nft transaction
before applying it, and leaves the old .1 path intact if no set is
available.
LIVE + COMMITTED, xlab: forwarded packets claiming a source outside the
xlab IPv4 /24 or IPv6 /64 are dropped before the trusted-LAN accept
(hosts/xlab-gateway/networking.nix:170-175, :197-203).

The live xlab service loaded 8,786 IPv4 and 2,039 IPv6 prefixes. Marked CN
route probes selected wan99.0; unmarked foreign probes selected
wg-to-wgnet; all four Alibaba ranges selected the campus WAN. The nft CN
counter increased on client traffic, the source guards were present, and the
host reported no failed units after activation. A deliberate nftables reload
repopulated both sets from the validated cache in under one second; during
that transient empty-set interval, packets safely follow the old .1 path.

These four /16s are the observed Alibaba exception set, not a complete or
future-proof inventory of every JD or Alibaba CDN. Keep packet counters and
capture evidence as the criterion for adding another static exception.

PotPlayer resolution

LIVE + COMMITTED, main gateway: commit 5e1d72f adds daumcdn.net to
data/infra/mosdns/force_proxy.txt. EasyPrivacy's tracker_domain.txt
blanket match had turned the PotPlayer download domain into NXDOMAIN.
After restarting mosdns, A and AAAA resolution and an HTTP/2 200 response
for the exact PotPlayerSetup64.exe URL were verified.

Main-gateway IPv6 DNS

LIVE + COMMITTED, main gateway: commit
2898736f8f42afd7bfd17ce25ef2047c27b6fd58 adds MosDNS UDP and TCP listeners
on [fd99:23eb:1682::1]:53. The address and both sockets, IPv6 UDP local and
Taobao resolution, IPv6 TCP DNSSEC with the AD bit, IPv4 regression, and
container health were verified. Only MosDNS was restarted.

P0 - fix before routine operations

1. Skydick iSCSI state is erased by the declarative default

Verified configuration: hosts/skydick/datapool.nix:712-713 only enables
services.target; it does not set services.target.config. In the pinned
NixOS module, that option defaults to {} and manages
/etc/target/saveconfig.json as mode 0600. The documented imperative
targetcli ... saveconfig workflow (hosts/skydick/DATAPOOL.md:516-558)
therefore writes a file that the next NixOS activation can replace with {}.

Impact: an already loaded kernel target can make this look healthy until a
reboot, iscsi-target restart, or LIO clear. The restored target will then be
empty.

Required remediation: first back up the live JSON and record targetcli ls; then make restoration declarative or copy a persistent, encrypted runtime
file into place before iscsi-target starts. Do not put CHAP secrets directly
in a Nix expression because they would enter the world-readable Nix store. Add
an activation assertion that refuses to replace a non-empty live target with
an empty definition, then test restore and client login before allowing a
rebuild.

2. The NFS pseudo-root can expose unlisted mounted datasets

Verified configuration: /srv is exported to all of 10.0.0.0/16 with
crossmnt (hosts/skydick/datapool.nix:479-480). crossmnt implicitly exports
mounted child filesystems with the parent's options. This can expose unlisted
children such as monitoring, Time Machine, private, or future ZFS datasets and
can undermine narrower child exports.

The surrounding ACLs are also broader than their prose suggests:

all LAN hosts can write /srv/media as UID 900 (:482-485);
all LAN hosts map to the owner of ye-lw21 and zhuyz24 (:487-500);
all LAN hosts receive no_root_squash on backup and VM trees (:502-505).

This contradicts "Only you can access your tree" in
hosts/skydick/DATAPOOL.md:293-304. Remove crossmnt from the fsid=0
pseudo-root, export every intended child explicitly, and reduce client matches
to exact IPv4 /32 and IPv6 /128 identities. Use SMB or NFS Kerberos
(sec=krb5p) where user authentication is required.

P1 - high-priority reliability and security

Main gateway

CAKE/IFB is not attached after WireGuard recreation (LIVE FINDING). The
related units report healthy while the qdisc/filter is absent, so tunnel
fairness can silently disappear. Bind QoS lifecycle to the WireGuard device
recreation path, make setup idempotent, and alert on tc state rather than
unit state alone.
Fifteen legacy df99 peers bypass IPv6 policy; several peers appear
stale (LIVE FINDING). Migrate active peers to classified source prefixes,
remove expired peers, and default-deny unclassified IPv6 sources.
Public SSH permits passwords and forwarding, and its limiter is global
(LIVE FINDING). A remote actor can consume the shared allowance and lock
out operators. Prefer VPN/source-restricted key-only access, disable
forwarding unless explicitly required, and use per-source limits.
WAN IPv6 lacks equivalent source anti-spoofing (LIVE FINDING). Add the
IPv6 reserved, ULA, link-local, multicast, and owned-prefix source checks at
the raw/prerouting boundary.

Xlab gateway

The TSIG DDNS secret is committed in plaintext. It is embedded at
hosts/xlab-gateway/dhcp.nix:172-178. Treat it as compromised: rotate the
key on xlab and the authoritative server, move it to agenix, and render a
root-only runtime Kea fragment.
The new source-spoof guards cover forwarding but not router input. The
input chain accepts every service from the entire LAN, management VLAN, and
both WireGuard interfaces (hosts/xlab-gateway/networking.nix:178-194). Add
the source validation before those accepts and replace full-interface trust
with explicit DHCP, DNS/DDNS, SSH, ICMP, and WireGuard service rules.
CN-direct routing needs monitoring and a controlled failover test. The
normal-path route and counter checks passed after deployment. Add alerts on
APNIC cache age, set size, and the presence of both policy rules. In a
maintenance window, confirm the intended table-1002 fallback when the local
WAN default is unavailable. Registry allocation is a routing heuristic,
not exact geolocation.

Skydick

Storage services fail open onto placeholder directories. Tmpfiles
creates the /srv tree (hosts/skydick/datapool.nix:298-338), while
dick-zfs-properties exits successfully when pool dick is missing and
only orders itself before NFS/Samba (:340-368). Evaluated NFS and Samba
units have no hard mount requirement. Gate NFS, Samba, InfluxDB, and iSCSI
behind a unit that asserts the expected ZFS datasets and mount identities.
LDAP binds and credentials cross the LAN in plaintext. NSS uses
ldap:// with TLS disabled (hosts/skydick/default.nix:326-340); Samba's
LDAP passdb also disables SSL and strong authentication
(hosts/skydick/datapool.nix:524-540). Move both to validated StartTLS or
LDAPS before rotating the bind and admin credentials.
Service exposure is host-wide rather than source/interface scoped. NFS,
SMB, iSCSI, InfluxDB, and node-exporter ports are globally allowed
(hosts/skydick/datapool.nix:715-731, modules/influxdb.nix:33-62,
modules/monitoring.nix:166-173, :296-297). Restrict storage protocols to
exact storage clients/VLANs and monitoring ports to the monitoring host.
Enable TLS for InfluxDB and replace the shared fleet-wide operator token
documented in hosts/skydick/README.md:20-42 with per-writer tokens and a
read-only Grafana token.
NFS/RDMA is only partially wired. Live inspection found the mlx4 RDMA
devices active and /proc/fs/nfsd/portlist containing rdma 20049 twice,
but no active client RDMA mount was confirmed. The listener is declared at
hosts/skydick/datapool.nix:441-469; port 20049 is absent from the firewall
at :715-731, and NFS/RDMA clients require the export's insecure option
because they do not use a reserved source port. Scope TCP/UDP 20049 and that
export option to exact clients, reconcile the duplicate writer, and prove
proto=rdma from a client mount rather than inferring it from the server
listener.
The pinned NixOS release is out of security support. flake.nix:5
selects nixos-25.11; both hosts correctly keep system.stateVersion = "25.11" (hosts/xlab-gateway/default.nix:107,
hosts/skydick/default.nix:401). NixOS 25.11 reached EOL after 2026-06-30;
NixOS 26.05 is
supported through 2026-12-31. Upgrade the input to 26.05 without changing
system.stateVersion, and specifically rebuild/test the exact-source Samba
overlay at hosts/skydick/datapool.nix:252-279.

P2 - cleanup and optimization

Main gateway: bind Avahi only to intended LAN interfaces and remove stale
discovery ports; deduplicate policy rules; pin container images by digest;
enable Docker live-restore where compatible; and add a second, health-checked
foreign relay instead of retaining a single exit failure domain.
Xlab bonding: balance-xor has no explicit transmit hash policy
(hosts/xlab-gateway/networking.nix:266-275). Routed traffic can collapse
onto one member with a layer-2 hash. Validate the switch LAG and use a
layer3+4 hash (or 802.3ad with matching switch configuration), then measure
per-member distribution.
Xlab CAPWAP workaround: capwap-df-clear deletes and recreates the whole
ingress qdisc (hosts/xlab-gateway/default.nix:87-104). Move to stable
clsact/owned filter handles so restarting the unit cannot erase unrelated
ingress policy.
Skydick NIC tuning: the active bond uses enp130s0 and enp130s0d1
(hosts/skydick/default.nix:79-100), but ring tuning still targets the
retired enp4s0f0np0 and enp4s0f1np1 (:191-194). Retarget it and record
before/after drop, latency, and IRQ-affinity metrics.
Skydick discovery: samba-wsdd is enabled with openFirewall = false
(hosts/skydick/datapool.nix:707-710), while UDP 3702/TCP 5357 are absent
from the firewall. Either open them only on the client LAN or disable WSD.
Skydick RoCE QoS: configuration explicitly provides DSCP marking without
end-to-end PFC (hosts/skydick/datapool.nix:404-410). Complete and test PFC,
ECN/congestion control, MTU, and watchdog configuration at every hop before
calling the fabric lossless.
Privilege boundary: ye-lw21 is a Nix trusted-user
(modules/common.nix:7-10) even though the account is documented as
password-sudo only (modules/users.nix:23-27). Nix trusted users are
effectively root-capable; remove the entry unless that is intentional.
Secret rotation: add restart/reload triggers for the Samba LDAP seed and
Telegraf token (hosts/skydick/datapool.nix:371-389,
modules/monitoring.nix:88-99) so rotating an agenix secret changes the
running credential without manual intervention.

Execution order

Back up and make skydick iSCSI state activation-safe. Only then allow a
skydick rebuild.
Remove the NFS pseudo-root crossmnt exposure and narrow NFS client ACLs.
Add a client-path monitor for the repaired main-gateway IPv6 DNS listener,
then repair CAKE/IFB lifecycle and close the df99 bypasses.
Rotate the exposed xlab TSIG key and harden router input/source validation.
Monitor the deployed xlab CN-direct policy and run its WAN-failure test in
a maintenance window; retain packet capture as the basis for any future
Alibaba or JD exception ranges.
Restrict skydick service exposure, enable LDAP/InfluxDB TLS, and add ZFS
mount guards.
Upgrade the NixOS input to 26.05 and run host-specific build and service
regression tests before deployment.