diff --git a/README.md b/README.md index 6a3aa64..26401ae 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,17 @@ # skyworks-Nix-infra NixOS configurations for the two NixOS hosts. Flake-based (`flake.nix`) -with `disko` for disk layout, `agenix` for secrets, `deploy-rs` defined -but not consistently used (most ops do `git pull && nixos-rebuild switch` -on the box). +with `disko` for disk layout, `agenix` for secrets, and `deploy-rs` for +push-based deploys from a workstation (the canonical workflow, magic-rollback +on). Hosts can also self-update with `git pull && nixos-rebuild switch` on the +box — handy when the workstation can't cross-build the target closure. ## Hosts | Host | IP (mgmt) | Role | |---|---|---| | `xlab-gateway` | 10.253.254.1 | Lab gateway / WAN router. Bond + VLANs (lan254, wan99, mgmt) + WireGuard tunnels with policy routing, NAT/masquerade. Kea DHCP4/6, radvd, fail2ban. | -| `skydick` | 10.0.1.1 | Storage server. ZFS data pool with hot spares, Samba+LDAP gateway. Heavy use of jumbo frames + bonded 100G. | +| `skydick` | 10.0.1.1 | Storage server. ZFS data pool with hot spares; Samba (LDAP-backed passdb) + NFS + iSCSI. Jumbo frames (MTU 9200) over bonded 2×40G ConnectX-3 (LACP 802.3ad, `bond40g`). | ## Layout @@ -46,10 +47,17 @@ ## Deploy via deploy-rs (from a workstation) ```sh -nix develop # if there's a devShell, otherwise install deploy-rs -deploy .# +# From an aarch64-darwin Mac (can't cross-build the x86_64-linux closure): +nix run github:serokell/deploy-rs -- --remote-build --skip-checks .# ``` +`--remote-build` builds on the target (its store is reachable as +`ssh-ng://ldx@`); `--skip-checks` skips `nix flake check`, whose +x86_64-linux `deployChecks` can't be built on darwin. Magic-rollback is on, so a +bad activation auto-reverts. Note: deploy-rs activates a closure but does **not** +update `/etc/nixos` git on the host — use the on-the-box `git pull` flow if you +want the host's checkout to track `main`. + ## Common gotchas - **DNS**: both hosts route DNS via `10.0.0.1` (mosdns) with a fallback set @@ -57,10 +65,12 @@ `networking.nameservers` — systemd-resolved load-balances and bypasses the analytics filter on 10.0.0.1. -- **IPv6 RA**: `networking.enableIPv6 = false` only stops IPv6 *use*, not - RA *acceptance*. Both `boot.kernel.sysctl."net.ipv6.conf.*.accept_ra"=0` - and `systemd.network.networks."40-bondX".networkConfig.IPv6AcceptRA = false` - are needed if upstream radvd is announcing DNS. +- **IPv6 RA**: `skydick` runs IPv6 *enabled* (SLAAC, stable token `::d1c0` on + `bond40g`). RA-provided DNS is kept out of systemd-resolved with `UseDNS = + false` in `ipv6AcceptRAConfig` — **not** by disabling IPv6. Pin the stable + token address with `IPv6PrivacyExtensions = lib.mkForce false` (NixOS forces + `"kernel"` otherwise) and `tempAddress = "disabled"`. Do not reintroduce + `enableIPv6 = false` / `accept_ra = 0` on skydick. - **deploy-rs ssh user**: defined as `ldx` with sudo escalation to root. Make sure the host key in `secrets.nix` matches reality after a reinstall diff --git a/hosts/skydick/DATAPOOL.md b/hosts/skydick/DATAPOOL.md index 508f33a..53fcac3 100644 --- a/hosts/skydick/DATAPOOL.md +++ b/hosts/skydick/DATAPOOL.md @@ -2,7 +2,7 @@ Server: `skydick` (10.0.1.1), pool: `dick` -Network: 10GbE bonded (bond0), jumbo frames MTU 9000, subnet 10.0.0.0/16. +Network: bonded 2×40G ConnectX-3 (LACP 802.3ad, interface `bond40g`), jumbo frames MTU 9200, subnet 10.0.0.0/16. Pool: 10 × Seagate Exos/Mach2 ST14000NM0001 14TB SAS (4Kn), each presenting 2 LUNs. 8 mirrors (from 8 drives) + 2 hot spares + 2 × Intel DC P4600 Optane 750GB. ~50.9T usable. @@ -15,6 +15,7 @@ |-------|-------------|----------|-----------|--------| | Public files | `/srv/public` | `\\SKYDICK\public` | `/public` | rw, all @storage users | | Media library | `/srv/media/library` | `\\SKYDICK\media` | `/media/library` | ro, all @storage users | +| Torrent payload | `/srv/media/data` | `\\SKYDICK\torrents` | `/media` (via `data/`) | ro, all @storage users — browse & re-seed | | Personal files | `/srv/users//files` | `\\SKYDICK\` | `/users/` | rw, SMB owner-authenticated; NFS network-trusted all_squash to owner | The NFS mount paths in the table above are for Linux/Unix NFSv4 clients using the pseudo-root @@ -117,6 +118,7 @@ ``` \\10.0.1.1\public \\10.0.1.1\media +\\10.0.1.1\torrents \\10.0.1.1\ ``` @@ -132,6 +134,7 @@ ``` smb://10.0.1.1/public smb://10.0.1.1/media +smb://10.0.1.1/torrents smb://10.0.1.1/ ``` @@ -252,9 +255,40 @@ - SMB: read-only for `@storage` - NFS: read-only via `/media/library` export -The full `/srv/media` dataset (including `/srv/media/data` with raw torrent payload) is only -writable by the `qbittorrent` service account (UID 900). Hardlinks between `data/` and -`library/` work because they are directories on the same ZFS dataset. +The qBittorrent + *arr writer stack runs on **door-pek** (not on skydick) and writes +`/srv/media` over NFS as the `qbittorrent` service account (UID 900, via the `/srv/media` +export's `all_squash,anonuid=900`); skydick only serves the dataset. The full `/srv/media` +dataset (including `/srv/media/data` with raw torrent payload) is therefore only *writable* by +that one service account. Hardlinks between `data/` and `library/` work because they are +directories on the same ZFS dataset. + +### Torrents (`/srv/media/data`) + +Read-only view of the raw per-tracker download tree (`BYRBT`, `M-Team - TP`, `TJUPT`, +`IPTorrents`, `qbit`, `tv`, `movies`, `music`, `incomplete`). door-pek's qBittorrent (UID 900) +is the **sole writer**, over NFS; everyone else gets read-only access so they can **re-seed the +same files without re-downloading** (seeding only needs to *read* the payload + the `.torrent` +piece hashes). Most users want the organized `media` share instead — this is the unorganized +payload. + +- SMB: read-only for `@storage` as `\\SKYDICK\torrents` +- NFS: reachable read-only under the `/srv/media` export at `data/` (do **not** mount it + writable from a second client — two writers to the same payload corrupts files / *arr + hardlinks) + +**Re-seeding an existing torrent** (without re-downloading the data): + +1. Mount the payload read-only: `smb://10.0.1.1/torrents` (macOS/Linux GUI), or + `sudo mount -t cifs //10.0.1.1/torrents /mnt/torrents -o ro,username=,vers=3.1.1`. +2. Get the matching `.torrent` (re-grab it from the tracker, or copy it from door-pek's + qBittorrent `BT_backup/`). The `.torrent` is tiny metadata — only the multi-GB payload must + not be re-downloaded. +3. In **your own** client, add the torrent with the save path pointing at the existing bytes + on the read-only mount (e.g. save-path `/mnt/torrents/movies` for a release under + `/srv/media/data/movies/…`). Leave "skip hash check" **off**. +4. Force-recheck → it reads the pieces off the read-only mount, verifies hashes, jumps to + 100% / Seeding with 0 B downloaded. Your client's `.fastresume`/session live in **your own** + local config (or your `/srv/users//bt-state`), never in the read-only data tree. ### Personal files (`/srv/users//files`) @@ -559,7 +593,7 @@ ### Slow NFS transfers -- Ensure MTU 9000 (jumbo frames) is set on both client and server interfaces +- Ensure jumbo frames (MTU 9200 on skydick's `bond40g`) are set on both client and server interfaces - Add `nconnect=16` to mount options for parallel NFS connections - Add `rsize=1048576,wsize=1048576` for 1MB read/write blocks - Check link speed: `ethtool | grep Speed` diff --git a/hosts/skydick/datapool.nix b/hosts/skydick/datapool.nix index efc7eb4..4cd4d9d 100644 --- a/hosts/skydick/datapool.nix +++ b/hosts/skydick/datapool.nix @@ -555,6 +555,25 @@ "vfs objects" = "catia fruit streams_xattr"; }; + # Raw torrent payload — read-only, browseable so every @storage user can + # SEE the per-tracker download tree (BYRBT, M-Team, TJUPT, IPTorrents, + # qbit, tv, movies, music, …) and re-seed the same bytes without + # re-downloading. Seeding needs only READ access, so this share never + # grants write: door-pek's qBittorrent (uid 900), writing /srv/media over + # the rw,all_squash NFS export, stays the SOLE writer. Pointed at + # /srv/media/data directly (not /srv/media whole) to exclude the backups/, + # baidu/, immich/ and *.zfs.zst clutter in the /srv/media root. On-disk + # perms already permit @storage reads (payload 0640, dirs 0750/2775, all + # group-readable/traversable by storage gid 997). `recycle` is omitted — + # meaningless on a read-only share (mirrors the [media] block). + torrents = { + path = "/srv/media/data"; + browseable = "yes"; + "read only" = "yes"; + "valid users" = "@storage"; + "vfs objects" = "catia fruit streams_xattr"; + }; + # Per-user homes — Samba auto-creates \\SKYDICK\ from this template homes = { path = "/srv/users/%S/files";