tooling: auto-upgrade gallery-dl + yt-dlp before every run and weekly; document 2026-10-22 empty posts listing

TOOLING.md

@@ -283,6 +283,30 @@ loudly, verified with both a real captured rsync failure and unit tests.
 The other 22 release tags stay on gitea. Pushing them all to GitHub triggers
 one container build per tag, because each tag carries its own workflow file.
 
+## 2026-10-22 — gallery-dl auto-upgrade; posts listing went empty
+
+Scheduled and manual runs were failing quietly from early October: after the session was
+restored, `posts` listings for `heejin`/`kimxxlip` returned `No results` in ~3 s (a caught-up
+profile skips; it does not say that), while stories worked and the posts opened fine in the
+browser. The host was on gallery-dl 1.32.14; 1.32.17 (2026-10-17) lists "instagram: update
+`user` & `posts` query parameters". After upgrading, the profile lookup failed instead
+(`KeyError: 'data'`, `Requested user could not be found`) -- **unresolved when written**, see
+the end of this section.
+
+`scripts/gdl-upgrade.sh` (`pipx upgrade --include-injected gallery-dl`, so yt-dlp too) now runs:
+
+- at the start of every `gdl-cron.sh` run, after the session preflight and before any request;
+  logs `upgrade: up to date` or `upgrade: CHANGED a -> b`, and never fails the run;
+- weekly from `gdl-upgrade.timer` (systemd user timer, installed on `mattellite` with the
+  others -- systemd rather than cron, to match the rest of the host).
+
+It costs no Instagram requests. The trade-off: a bad upstream release is picked up
+automatically; roll back with `pipx install --force gallery-dl==<old>` and re-inject yt-dlp.
+
+**A `--min-interval` trap:** a run that returns `No results` still counts as a fetch, so the
+next run skips the source for 20 h. Re-run after a bad result with `--min-interval 0`
+(not `--force`, which also drops the probe cache).
+
 ## Guards — recreate these after a fresh clone
 
 Neither guard is versioned, so a new clone has **no protection at all**:

scripts/gdl-cron.sh

@@ -84,6 +84,14 @@ if [ -x "$SESSION_CHECK" ]; then
   echo "preflight: $SESSION_MSG" | tee -a "$LOG"
 fi
 
+# Upgrade gallery-dl/yt-dlp first (PyPI only, no Instagram requests; never fatal). Instagram
+# endpoints change and gallery-dl follows within days -- on 1.32.14 the posts listing silently
+# returned "No results" after 1.32.17 had already fixed the query. See gdl-upgrade.sh.
+UPGRADE="$GDL_HOME/gdl-upgrade.sh"
+if [ -x "$UPGRADE" ]; then
+  "$UPGRADE" 2>&1 | tee -a "$LOG" || true
+fi
+
 # Staging is wiped every run ON PURPOSE. What we already hold is decided by the
 # skip-archive (--download-archive), never by which files happen to be sitting
 # in staging, so starting empty is correct -- and it keeps the publish rsync

scripts/gdl-upgrade.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+# Upgrade gallery-dl and its injected yt-dlp (both pipx). Instagram changes its
+# endpoints often and gallery-dl follows within days -- 1.32.17 (2026-10-17)
+# "update user & posts query parameters" landed while we were still on 1.32.14
+# and the posts listing silently returned "No results".
+#
+# Called by gdl-cron.sh before every scrape and weekly by gdl-upgrade.timer.
+# Never fatal: a failed upgrade (offline, PyPI down) must not stop a stories
+# run, which cannot be backfilled -- it just scrapes with what is installed.
+# Costs no Instagram requests (PyPI only).
+set -u
+
+PATH="$HOME/.local/bin:$PATH"; export PATH
+
+VENV_PY="${GDL_VENV_PY:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python}"
+
+versions() {
+  printf 'gallery-dl %s, yt-dlp %s' \
+    "$(gallery-dl --version 2>/dev/null || echo '?')" \
+    "$("$VENV_PY" -c 'import yt_dlp; print(yt_dlp.version.__version__)' 2>/dev/null || echo '?')"
+}
+
+before=$(versions)
+if ! timeout 180 pipx upgrade --quiet --include-injected gallery-dl >/dev/null 2>&1; then
+  echo "upgrade: WARNING pipx upgrade failed; continuing with $before" >&2
+  exit 0
+fi
+after=$(versions)
+
+if [ "$before" = "$after" ]; then
+  echo "upgrade: up to date ($after)"
+else
+  echo "upgrade: CHANGED $before -> $after"
+fi

scripts/systemd/gdl-upgrade.service

@@ -0,0 +1,9 @@
+[Unit]
+Description=Upgrade gallery-dl and yt-dlp (pipx)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=%h/gdl/gdl-upgrade.sh
+Nice=10

scripts/systemd/gdl-upgrade.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Weekly gallery-dl and yt-dlp upgrade
+# gdl-cron.sh also upgrades before every scrape; this keeps the tools current
+# on weeks when no run happens to call it.
+
+[Timer]
+OnCalendar=weekly
+RandomizedDelaySec=1d
+Persistent=true
+
+[Install]
+WantedBy=timers.target