Adding a PCIe card can renumber the buses behind it, so a hostpci entry
that names a PCI address can pass through the wrong device. These start
the VM only after finding each device by vendor/device/subsystem ID or
by its MAC-pinned interface, repoint its resource mapping if it moved,
and refuse to start on anything ambiguous. Its dependents then wait for
a port on it to answer before starting, so they find their shares up.
/opt/ergosteur is now a checkout rather than a directory kept in step by scp,
so deploying is a pull.
Records why the remote is HTTPS rather than SSH: the repository is public, so
the host needs no credentials and no deploy key, and in exchange it cannot
push, which is the right shape for a deploy target.
Also notes that git carries the executable bit, since a script committed
without it now reaches the host non-executable and breaks any unit using it as
ExecStart. That failure mode did not exist while deploying by scp, which
carries the mode from the working copy regardless of what git records.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It was the only script in the repository tracked as 100644. Every other one is
100755, and it has been executable on the Proxmox host all along, so this never
showed up while deploying by scp, which carries the mode across.
It would have broken the first deployment made by cloning instead:
ipmi-fancontrol.service runs it as ExecStart, and a fresh checkout would have
produced a non-executable file and a permission denied at start, leaving the
fans on whatever duty the BMC last had.
Found while converting /opt/ergosteur to a git checkout, where it showed up as
the one tracked file differing from the host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The usage line listed --docker but nothing said what it did, so the only way to
find out was to read the runcmd generation.
Two things are worth stating beyond "installs Docker". The install runs on
first boot of each clone rather than during the build, because the snippet is a
runcmd and the template is never booted, so every clone fetches Docker itself
and nothing is baked into the image. And the non-Alpine, non-Arch path uses the
convenience script from get.docker.com, so those guests get Docker CE from
Docker's repo rather than the distro package.
Also records that the cloud-init user lands in the docker group, which is
root-equivalent on that guest, and that the flag is positional.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repository had no entry point, and the cloud image pinning in particular
needed explaining: which distributions track "latest" upstream and which bake
an exact version into the filename is the thing that determines whether a pin
rots silently, and it was not evident from reading the script.
Documents the template builder and update-image-pins.sh in full, including the
two properties worth preserving in the updater, deriving URLs from the builder
rather than duplicating them and verifying qcow2 magic rather than a 200, since
both look like needless complexity to anyone simplifying it later.
Indexes the rest of the tooling one line each, and records the deployment step,
because /opt/ergosteur is not a checkout and nothing here is live until copied
across.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shellcheck flagged three things, one of which was worth acting on beyond
silencing it.
echo was emitting a script fragment containing a literal \n meant for the
generated code, not for this shell. Bash's echo leaves it alone by default,
but the handling of backslashes is implementation-defined and xpg_echo would
have expanded it and corrupted the fragment. printf '%s\n' instead.
The discover_* functions were dispatched as "discover_$distro", which reads as
dead code to shellcheck and to anyone grepping for a caller. An explicit case
dispatch replaces it, which is what SC2317 was really complaining about.
The two remaining SC2016 hits are intentional single quotes, one a sed address
matching a literal $DISTRO_CHOICE and one emitting code, and are annotated
rather than fixed.
Also reject an unknown --distro. It previously filtered every distro out and
reported "all pins are already current" with exit 0, so a typo looked like a
clean run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Refreshing the pins by hand meant checking five mirrors and hand-building
URLs to confirm them, which is how Fedora and Alpine drifted two releases
behind without anyone noticing. This does the lookup and the checking.
Prefers each distribution's own metadata over scraping where it exists:
Ubuntu from changelogs.ubuntu.com/meta-release, the same file
do-release-upgrade reads, filtered to supported LTS; Debian from the stable
suite's Release file, so it follows Debian's own definition of stable rather
than guessing from directory names. Fedora, AlmaLinux and Alpine have no such
feed and are read from mirror listings.
Alpine turns out to be automatable after all. latest-stable is a symlink to
the current branch, so the filename there yields the branch, the patch level
and the prefix together, and reading the prefix rather than assuming it is
what would have caught the nocloud_ rename.
Two properties matter more than the lookups:
Candidate URLs are derived by evaluating the pin block and the case statement
out of build-cloudinit-template.sh itself, not by repeating its URL templates
here. A second copy would eventually disagree with the builder and verify
something the builder never downloads.
Nothing is written until the image has been confirmed to serve qcow2 magic,
not merely a 200. Mirrors answer for a deleted image with an HTML error page
under a 200, which would otherwise be pinned as good and fail at build time.
A distro that fails verification keeps its old pin and does not hold up the
others.
Dry run by default. Exit 3 distinguishes "pins are stale" from success, so it
can run as a periodic staleness check, and 2 means a lookup or verification
failed.
The template is read by anchor, so a restructured pin block or case statement
is detected and refused rather than silently producing URLs from unset
variables.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Alpine renamed the prefix from nocloud_ to generic_ in 3.24, which was
hardcoded in the URL and so had to be edited by hand alongside the version
bump. As a pin it sits with the other values a release bump touches, and
update-image-pins.sh can read the real prefix off the mirror rather than
assuming one.
No change to the URL produced: still generic_ for the pinned 3.24.1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fedora was pinned to 41 and Alpine to 3.23.3, both two releases behind and
out of support. Nothing had broken yet, since Fedora 41 is still served from
releases/, but neither gets patches and both embed an exact release number in
the URL, so they 404 with no warning once the mirror rotates them out.
Alpine 3.23.3 to 3.24.1, Ubuntu 24.04 noble to 26.04 resolute, Fedora 41-1.4
to 44-1.7, AlmaLinux 9 to 10. Debian stays on trixie, still current stable
with forky in testing, and Arch is rolling.
Alpine also renamed the cloud image prefix from nocloud_ to generic_ in 3.24.
v3.24 carries only generic_ files, so bumping the pin alone would have 404'd
and the two changes cannot be separated. This does mean ALPINE_MAJOR can no
longer be rolled back to 3.23 or older without also reverting the prefix,
which the comment there records.
AlmaLinux 10 raises the CPU baseline to x86-64-v3, inherited from RHEL 10.
The script builds with --cpu host, so a template built on pre-Haswell
hardware will not boot. Fine on this host's Broadwell Xeons.
Every URL the script generates was checked by evaluating the pin block and
the case statement out of the script itself, rather than by hand: all six
return 200 with qcow2 magic and plausible sizes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Documents the target state for a host that owns a USB UPS and serves it to a
Synology, for hand-configuring a machine where running
setup-nut-synology.sh is not wanted. Verified byte-identical to what that
script generates, so the two cannot drift.
The [admin] password is the only real secret and ships as CHANGEME in both
files that must agree, upsd.users and the MONITOR line of upsmon.conf.
monuser/secret is committed literally. DSM hardcodes both and cannot be
configured otherwise, so they are published constants rather than
credentials, and the upsmon secondary role only permits reading status and
receiving shutdown notifications.
The example UPS serial is a placeholder rather than the real unit's.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Proxmox imports OVA/OVF natively now through a storage with the "import"
content type, or qm importovf. The built-in path also handles per-storage disk
format conversion, which neither script did.
Recorded the known bugs in each header rather than fixing them, since nothing
will run these again: import-ova-pve.py imports every VMDK but only attaches
disk-0, leaving multi-disk appliances partly orphaned, and assumes a
vm-<vmid>-disk-N name that not every storage type uses;
import-ova-to-proxmox.sh passes --format qcow2 unconditionally, which fails on
ZFS and LVM-thin.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sets up NUT from scratch for a USB UPS and serves it to a Synology, which is
otherwise fiddly because DSM's client cannot be configured: it always asks for
a UPS named "ups" and always logs in as monuser with the password "secret".
The script guarantees all three, plus a reachable LISTEN.
Detects the UPS with nut-scanner, writes nut.conf, ups.conf, upsd.conf,
upsd.users and upsmon.conf, then starts and verifies the stack.
Re-runnable: it reuses an existing admin password rather than churning it,
backs up anything it replaces, and prompts before overwriting unless given
--yes. --dry-run prints the files without writing, and --etc-dir writes
elsewhere so the output can be reviewed against a live config.
Warns when another process already holds the USB device, naming QEMU
specifically. That is what broke this host: a passthrough to a guest claimed
the UPS, and the driver crash-looped for months with a message that did not
point at the cause.
Drops nut-scanner's bus, device and busport keys, which change across replug,
and its ###NOTMATCHED-YET### markers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It was meant to restart the NUT driver from a udev RUN+= when the UPS
re-enumerates on USB, but was never wired up: 50-nut-ups.rules only sets MODE
and GROUP.
systemd already covers the case. nut-driver@ups.service is Restart=always
with RestartUSec=15s, so the driver is retried indefinitely and reattaches to
a new device node unaided.
It would also not have worked reliably. systemd-udevd kills RUN+= handlers
still alive when the event completes, so the backgrounded subshell could be
killed mid-sleep, and upsdrvctl would run the driver outside the unit's
supervision, competing with systemd's own restarts.
Kept with a header recording the rule-based alternative if faster recovery is
ever wanted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tailscale's UDP GRO forwarding tuning applies to the machine running
tailscaled. That is VM 100 (ts-subnet-rtr), a KVM guest with its own kernel
and netdevs, so ethtool on the Proxmox host cannot affect its forwarding
path. Host-side tuning would only have been the right lever for a tailscaled
running in an LXC container, which shares the host kernel.
The interfaces were wrong in any case. VM 100 is on vmbr7, backed by
enp4s0f7np7. vmbr0 is backed by enp4s0f0np0 and carries unrelated guests, and
enp4s1f12np12 is the host's own routing NIC and not a bridge port at all.
It never ran: the header says to run it from root's crontab, and root has no
crontab. Both offloads were still off on every interface it named.
Kept with a header recording where the tuning actually belongs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Manual fan mode pins the fans at a fixed duty and never ramps, so a sustained
load had no cooling response until the BMC's own critical-temperature
override stepped in. docker-idrac.sh solved that, but needs Docker, which is
not installed on the host.
ipmi-fancontrol-daemon polls the CPU sensors and switches between the fixed
duty and the BMC's automatic curve around a threshold, which is what that
container did. Measured on the host: 1440 RPM pinned versus roughly 4200 RPM
on the automatic curve at the same 60C.
- Hysteresis, default 5C, stops the fans oscillating at the boundary.
- An unreadable temperature fails safe to automatic rather than staying
pinned.
- SIGTERM restores the automatic curve, with ExecStopPost as the backstop for
an outright kill.
Default to the local "open" interface. It needs no credentials and no
network, which matters for the shutdown path, where lanplus to the iDRAC
could fail once networking is down. ipmi-fancontrol-auto and -manual honour
the same setting and stay available for ad-hoc use.
Enable the third-party PCIe cooling response suppression that docker-idrac.sh
was configured with, so the automatic curve is not held to a raised floor by
the aftermarket NICs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Docker is not installed on pet630, so neither of these can run there:
- docker-idrac.sh is the container-based alternative to the ipmi-fancontrol
scripts, which remain in service.
- webnut-docker/ is the web UI for the NUT daemon. NUT itself is still
running on the host; only this front end is retired.
The gitignored .env moves with the compose file and is still excluded at the
new path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The version line printed unconditionally, so the nightly job would mail a
one-line message on every run even when there was nothing to download. Guard
it like the other status output, leaving interactive runs unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The deployed /etc/cron.d/download-virtio-win never ran. It omitted the user
field that /etc/cron.d entries require between the schedule and the command,
so cron parsed the script path as a username, and its day-of-month field was
0, outside the valid range of 1-31.
Corroborated on the host: the newest ISO is virtio-win-0.1.285 from
2025-09-20, the same day the cron file was created.
Pass --cron so a no-op run emits no progress bar and generates no cron mail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Checked against the live host. None of these are in service:
- sriov-enp4s0.service and set-ixgbe-vfmacs-enp4s0.sh target enp4s0f1np1,
which no longer exists; sriov_numvfs is 0 on every port and the unit was
never installed.
- nic-performance-tweaks.service was never installed either, though its
interfaces do still exist, so it would work if reinstated.
- vm-watchdog.sh watches VMID 2801 "pfsense", which no longer exists.
- lxc-watchdog.sh still carries the upstream example targets: CT 100 and
CT 101 on 192.168.31.0/24, a subnet that is not in use.
Neither watchdog is scheduled; there is no root crontab.
The watchdog config moves to unused/conf/ alongside them, so the scripts
resolve it unchanged from their new location.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both watchdogs hardcoded the notification address. Read it from
conf/watchdog.conf instead, matching the pattern used for the other
credentials, so the repo carries no real contact details.
If the config is absent the watchdogs still run and log; sendAlert notes the
missing address and skips the mail rather than invoking mail(1) with an empty
recipient.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
build-cloudinit-template.sh supersedes make-debian-cloudinit-template.sh,
covering Alpine, Ubuntu, Debian, Fedora, AlmaLinux and Arch, with UEFI,
vendor snippets and an optional --docker runtime.
Carry forward the hardening the Debian script had gained, which the new one
was missing: set -euo pipefail, root/binary/numeric-VMID checks, a VMID
collision check, storage validation, an ERR trap naming the cleanup command,
image caching under /var/tmp, and env-overridable STORAGE, BRIDGE, MEMORY,
CORES, DISK_SIZE, SSH_KEY_FILE and CI_PASSWORD.
Quote the openssl passwd substitution feeding --cipassword. A SHA-512 crypt
hash contains $ and /, so unquoted it was subject to word splitting.
Derive the snippet directory from SNIPPET_STORAGE via pvesm path instead of
hardcoding /var/lib/vz/snippets, which silently disagreed with the storage
whenever SNIPPET_STORAGE was changed. Falls back to parsing storage.cfg.
Resize the imported disk with qm disk resize rather than resizing the image
file. Growing the file in place would mutate the shared cache and defeat the
size comparison wget -N uses to decide whether to re-download.
Check the Snippets content type on the target storage up front. It is off by
default, and without this the run failed much later at qm set --cicustom.
Fall back to timedatectl when /etc/timezone is absent, as it is outside
Debian.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Web UI for the NUT UPS daemon.
UPS credentials come from .env alongside the compose file, which Compose
loads automatically; see .env.example. Required values use ${VAR:?} so a
missing .env fails loudly rather than starting the container with a blank
password.
Drop the obsolete top-level version key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stop-pa-vm.service calls the PAN-OS XML API on host shutdown so the firewall
VM goes down cleanly rather than being killed.
The API key is read from conf/pa-vm.conf, resolved relative to the script.
See conf/pa-vm.conf.example.
Pass the request through curl --config on stdin with the key in an X-PAN-KEY
header, instead of embedding it in the URL. A key in the query string is
visible in ps and would be recorded in any proxy or webserver log along the
path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ipmi-fancontrol.service pins the fans to a fixed speed on start and hands
control back to the BMC's automatic curve on stop, to quiet the chassis.
BMC credentials are read from conf/ipmi-fancontrol.conf, resolved relative
to the scripts themselves so the tree stays self-contained whether it is run
from a checkout or from /opt/ergosteur. See conf/ipmi-fancontrol.conf.example.
Use ipmitool -E rather than -P so the password is passed via the environment
instead of argv, where any local user could read it out of ps.
docker-idrac.sh is the alternative temperature-aware controller.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sriov-enp4s0.service allocates VFs on the ixgbe ports at boot and applies
persistent VF MACs via set-ixgbe-vfmacs-enp4s0.sh, so guests keep stable
addresses across host reboots. The VF MAC assignments are currently all
commented out.
nic-performance-tweaks.service raises ring buffers and txqueuelen on the
uplinks and vmbr0.
unused/ holds the equivalent pair for the retired enp130s0 adapter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two implementations of the same OVA/OVF import flow: a Python version that
parses the OVF with ElementTree and detects UEFI, and a shell version using
xmllint that handles multi-disk appliances. Both check free space on the
target storage before importing.
download-virtio-win-progress.py fetches the latest virtio-win ISO, skipping
the download when the local copy is already current.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
prox-maintenance-mode.sh saves and clears onboot/startup across all VMs and
CTs so a host can be rebooted without guests coming back up, then restores
the saved state. Supports --status and --dry-run.
nut-udev-restart.sh restarts the NUT UPS driver on a USB re-enumeration
event. tailscale-optimize-router.sh applies the recommended UDP GRO
forwarding offloads for subnet routers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cron-driven liveness checks that ping each guest, restart it via qm/pct if
unreachable, and log plus email the outcome. Guest lists are configured in
the associative array at the top of each script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ignore *.conf and .env (keeping the committed *.conf.example and
.env.example templates) so runtime credentials stay out of the repo.
Also ignore bridge-vlan-show.txt, tmp/, and downloaded disk images, which
are host-specific dumps and build artifacts rather than source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ergosteur
