diff --git a/TOOLING.md b/TOOLING.md index e94b1c8..6389e40 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -283,6 +283,30 @@ loudly, verified with both a real captured rsync failure and unit tests. The other 22 release tags stay on gitea. Pushing them all to GitHub triggers one container build per tag, because each tag carries its own workflow file. +## 2026-10-22 — gallery-dl auto-upgrade; posts listing went empty + +Scheduled and manual runs were failing quietly from early October: after the session was +restored, `posts` listings for `heejin`/`kimxxlip` returned `No results` in ~3 s (a caught-up +profile skips; it does not say that), while stories worked and the posts opened fine in the +browser. The host was on gallery-dl 1.32.14; 1.32.17 (2026-10-17) lists "instagram: update +`user` & `posts` query parameters". After upgrading, the profile lookup failed instead +(`KeyError: 'data'`, `Requested user could not be found`) -- **unresolved when written**, see +the end of this section. + +`scripts/gdl-upgrade.sh` (`pipx upgrade --include-injected gallery-dl`, so yt-dlp too) now runs: + +- at the start of every `gdl-cron.sh` run, after the session preflight and before any request; + logs `upgrade: up to date` or `upgrade: CHANGED a -> b`, and never fails the run; +- weekly from `gdl-upgrade.timer` (systemd user timer, installed on `mattellite` with the + others -- systemd rather than cron, to match the rest of the host). + +It costs no Instagram requests. The trade-off: a bad upstream release is picked up +automatically; roll back with `pipx install --force gallery-dl==` and re-inject yt-dlp. + +**A `--min-interval` trap:** a run that returns `No results` still counts as a fetch, so the +next run skips the source for 20 h. Re-run after a bad result with `--min-interval 0` +(not `--force`, which also drops the probe cache). + ## Guards — recreate these after a fresh clone Neither guard is versioned, so a new clone has **no protection at all**: diff --git a/scripts/gdl-cron.sh b/scripts/gdl-cron.sh index f1219c9..a22eb00 100755 --- a/scripts/gdl-cron.sh +++ b/scripts/gdl-cron.sh @@ -84,6 +84,14 @@ if [ -x "$SESSION_CHECK" ]; then echo "preflight: $SESSION_MSG" | tee -a "$LOG" fi +# Upgrade gallery-dl/yt-dlp first (PyPI only, no Instagram requests; never fatal). Instagram +# endpoints change and gallery-dl follows within days -- on 1.32.14 the posts listing silently +# returned "No results" after 1.32.17 had already fixed the query. See gdl-upgrade.sh. +UPGRADE="$GDL_HOME/gdl-upgrade.sh" +if [ -x "$UPGRADE" ]; then + "$UPGRADE" 2>&1 | tee -a "$LOG" || true +fi + # Staging is wiped every run ON PURPOSE. What we already hold is decided by the # skip-archive (--download-archive), never by which files happen to be sitting # in staging, so starting empty is correct -- and it keeps the publish rsync diff --git a/scripts/gdl-upgrade.sh b/scripts/gdl-upgrade.sh new file mode 100755 index 0000000..717d91e --- /dev/null +++ b/scripts/gdl-upgrade.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# Upgrade gallery-dl and its injected yt-dlp (both pipx). Instagram changes its +# endpoints often and gallery-dl follows within days -- 1.32.17 (2026-10-17) +# "update user & posts query parameters" landed while we were still on 1.32.14 +# and the posts listing silently returned "No results". +# +# Called by gdl-cron.sh before every scrape and weekly by gdl-upgrade.timer. +# Never fatal: a failed upgrade (offline, PyPI down) must not stop a stories +# run, which cannot be backfilled -- it just scrapes with what is installed. +# Costs no Instagram requests (PyPI only). +set -u + +PATH="$HOME/.local/bin:$PATH"; export PATH + +VENV_PY="${GDL_VENV_PY:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python}" + +versions() { + printf 'gallery-dl %s, yt-dlp %s' \ + "$(gallery-dl --version 2>/dev/null || echo '?')" \ + "$("$VENV_PY" -c 'import yt_dlp; print(yt_dlp.version.__version__)' 2>/dev/null || echo '?')" +} + +before=$(versions) +if ! timeout 180 pipx upgrade --quiet --include-injected gallery-dl >/dev/null 2>&1; then + echo "upgrade: WARNING pipx upgrade failed; continuing with $before" >&2 + exit 0 +fi +after=$(versions) + +if [ "$before" = "$after" ]; then + echo "upgrade: up to date ($after)" +else + echo "upgrade: CHANGED $before -> $after" +fi diff --git a/scripts/systemd/gdl-upgrade.service b/scripts/systemd/gdl-upgrade.service new file mode 100644 index 0000000..c383ed5 --- /dev/null +++ b/scripts/systemd/gdl-upgrade.service @@ -0,0 +1,9 @@ +[Unit] +Description=Upgrade gallery-dl and yt-dlp (pipx) +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=%h/gdl/gdl-upgrade.sh +Nice=10 diff --git a/scripts/systemd/gdl-upgrade.timer b/scripts/systemd/gdl-upgrade.timer new file mode 100644 index 0000000..2543cc4 --- /dev/null +++ b/scripts/systemd/gdl-upgrade.timer @@ -0,0 +1,12 @@ +[Unit] +Description=Weekly gallery-dl and yt-dlp upgrade +# gdl-cron.sh also upgrades before every scrape; this keeps the tools current +# on weeks when no run happens to call it. + +[Timer] +OnCalendar=weekly +RandomizedDelaySec=1d +Persistent=true + +[Install] +WantedBy=timers.target