From 7c7ce70ff9732f50dd262684120cb917acf2c1a5 Mon Sep 17 00:00:00 2001 From: ergosteur Date: Sat, 22 Aug 2026 11:31:45 -0400 Subject: [PATCH] refactor: rename the modes to say what they actually do `full` was the misleading one: it is the abort-LIMITED run, the one that deliberately stops enumerating a profile as soon as it reaches content already held. Calling it "full" invited exactly the wrong assumption about coverage. And `sweep` gave no hint that it was the exhaustive one. full -> profiles every surface, --abort 50, ~40-60 requests sweep -> full-sweep every surface, no abort, ~420 requests The old names now exit 2 with a pointer to the new one rather than a bare "unknown mode", since muscle memory and any stray crontab will still use them. full-sweep's description now says what it costs. At ~420 requests it is the same order as the run that preceded the 2026-08-21 scraping warning, spent to catch a handful of retroactively edited posts, so the docs suggest running it by hand when you mean to rather than leaving it on a timer. Its cadence was never actually agreed. Units renamed to match and re-verified with systemd-analyze; the old ones are removed from the host. All three timers remain disabled. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF --- TOOLING.md | 37 +++++++++++-------- scripts/gdl-cron.sh | 31 ++++++++++------ scripts/systemd/gdl-sync@full-sweep.timer | 18 +++++++++ ...ync@full.timer => gdl-sync@profiles.timer} | 2 +- scripts/systemd/gdl-sync@sweep.timer | 13 ------- 5 files changed, 60 insertions(+), 41 deletions(-) create mode 100644 scripts/systemd/gdl-sync@full-sweep.timer rename scripts/systemd/{gdl-sync@full.timer => gdl-sync@profiles.timer} (65%) delete mode 100644 scripts/systemd/gdl-sync@sweep.timer diff --git a/TOOLING.md b/TOOLING.md index 61ea2de..881030c 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -63,7 +63,7 @@ every deletion because of it. | path | what it is | |---|---| | `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles | -| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `full` \| `sweep` | +| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` | | `scripts/systemd/` | the timers actually installed on the fetch host | | `scripts/test_gdl_sync.py` | its tests | | `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere | @@ -103,7 +103,7 @@ session-hijack detection looks for. ```sh ssh mattellite -~/gdl/gdl-cron.sh full # or: stories | sweep +~/gdl/gdl-cron.sh profiles # or: stories | full-sweep ``` That is the whole thing: it wipes staging, fetches, and publishes straight to @@ -126,13 +126,17 @@ on cron's PATH and not on a non-login shell's either. ### The three modes +Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run — +and `sweep` did not convey that it is the exhaustive one. The wrapper rejects +the old names with a pointer rather than a bare error. + | mode | cadence | cost | why | |---|---|---|---| | `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped | -| `full` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating once it reaches content already held | -| `sweep` | quarterly | ~420 requests | no abort; the **only** run that notices carousels edited after we archived them (test case 15) | +| `profiles` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating a profile once it reaches content already held. Catches everything **new** | +| `full-sweep` | rarely, by hand | **~420 requests** | no abort; walks every profile to the end. The only run that notices posts **edited** after we archived them (test case 15) | -The skip-archive means an infrequent `full` costs barely more than a frequent +The skip-archive means an infrequent `profiles` run costs barely more than a frequent one — it only fetches what is new. Frequency buys freshness, not completeness, except for stories. @@ -164,8 +168,8 @@ systemctl --user list-timers 'gdl-sync@*' | unit | schedule | next fire (as installed) | |---|---|---| | `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working | -| `gdl-sync@full.timer` | 3rd of each month, 04:00 | 04:37:44 | -| `gdl-sync@sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 | +| `gdl-sync@profiles.timer` | 3rd of each month, 04:00 | 04:37:44 | +| `gdl-sync@full-sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 | Unit files are version-controlled in `scripts/systemd/` and installed to `~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes @@ -186,9 +190,9 @@ Operating them: ```sh export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed over non-interactive ssh systemctl --user start gdl-sync@stories.service # run one now -systemctl --user status gdl-sync@full.timer +systemctl --user status gdl-sync@profiles.timer journalctl --user -u 'gdl-sync@*' -n 50 -systemctl --user disable --now gdl-sync@sweep.timer # stop one +systemctl --user disable --now gdl-sync@full-sweep.timer # stop one ``` `systemctl --user` fails with "Failed to connect to bus" over ssh unless @@ -201,8 +205,8 @@ before installing it. ```cron 17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories -43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full -11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep +43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh profiles +11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full-sweep ``` cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a @@ -234,7 +238,7 @@ highlights endpoint → suspension. If a run logs one, disable the timers and stop for the day: ```sh -systemctl --user disable --now gdl-sync@stories.timer gdl-sync@full.timer gdl-sync@sweep.timer +systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer ``` Then confirm the archive actually grew, from the workstation: @@ -333,7 +337,7 @@ twelve sources across six profiles, `official_artms` included at 1829 posts and 781 reels, finished in minutes for a few dozen requests where the old behaviour would have spent ~400. -Note the gap this leaves: **the scheduled `full` mode still uses the default +Note the gap this leaves: **the scheduled `profiles` mode still uses the default 6-10s pacing**, not the 12-20s used here. Reconcile that before re-enabling the timers, or the automation will be less careful than the hand runs that followed a warning. @@ -399,10 +403,11 @@ broken; these are decisions not yet made and cleanups not yet done. fully-skipped stories run exits 75 with a warning instead of looking like a success. The timers are still **disabled** — enabling them is a separate decision about cadence, not about pacing. -- **`--abort 50` is opt-in.** `gdl-cron.sh full` passes it and the manual runs - used it; `sweep` deliberately does not. It stops noticing **edited +- **`--abort 50` is opt-in.** `gdl-cron.sh profiles` passes it and the manual + runs used it; `full-sweep` deliberately does not. It stops noticing **edited carousels** (test case 15), which only a full enumeration finds — which is - what `sweep` is for. The quarterly cadence was proposed and never agreed. + what `full-sweep` is for. No cadence for it was ever agreed, and at ~420 + requests it is the riskiest thing on the schedule; prefer running it by hand. - **The `seeded` flags in `.state.json` were hand-written**, reconstructed from the 2026-08-17 log rather than derived from the archive DB. They assert "the skip-archive already knows this source". If `artms.db` is ever rebuilt, diff --git a/scripts/gdl-cron.sh b/scripts/gdl-cron.sh index dcf095f..b151772 100755 --- a/scripts/gdl-cron.sh +++ b/scripts/gdl-cron.sh @@ -1,15 +1,19 @@ #!/bin/bash # Unattended wrapper around gdl-sync.py. One argument: the run mode. # -# stories daily ~6 requests; the only surface that cannot be backfilled -# full monthly every surface, --abort 50 (stops at already-held content) -# sweep quarterly every surface, no abort -- the only run that notices -# carousels edited after we archived them +# stories daily ~6 requests; the only surface that cannot be backfilled +# profiles monthly every surface, --abort 50: stops enumerating a profile +# once it reaches content already held, so it costs +# ~40-60 requests and catches everything NEW +# full-sweep rarely every surface, no abort: walks each profile to the end +# for ~420 requests. The only run that notices posts +# EDITED after we archived them, and by far the most +# expensive thing here -- see TOOLING.md before running. # # Exits non-zero if the sync does, so cron mails you. Everything is logged. set -eu -MODE="${1:?usage: gdl-cron.sh stories|full|sweep}" +MODE="${1:?usage: gdl-cron.sh stories|profiles|full-sweep}" GDL_HOME="${GDL_HOME:-$HOME/gdl}" INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}" @@ -38,12 +42,17 @@ case "$MODE" in # skipped all six sources and reported success. A stories fetch is one # request per profile, so the worst case 8h permits is roughly twelve # requests in a day instead of six. - stories) ARGS="--only stories --min-interval 8" ;; - full) ARGS="--only posts,reels,stories,highlights --abort 50" ;; - # No --abort: the whole point of sweep is enumerating to the end, so it is - # the only run that notices carousels edited after we archived them. - sweep) ARGS="--only posts,reels,stories,highlights" ;; - *) echo "unknown mode: $MODE" >&2; exit 2 ;; + stories) ARGS="--only stories --min-interval 8" ;; + profiles) ARGS="--only posts,reels,stories,highlights --abort 50" ;; + # No --abort: the whole point of full-sweep is enumerating to the end, so it + # is the only run that notices carousels edited after we archived them. + full-sweep) ARGS="--only posts,reels,stories,highlights" ;; + # Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and + # "sweep" did not say it was the exhaustive one. Catch the old names rather + # than failing with a bare error, in case something still passes them. + full) echo "mode 'full' was renamed to 'profiles'" >&2; exit 2 ;; + sweep) echo "mode 'sweep' was renamed to 'full-sweep'" >&2; exit 2 ;; + *) echo "unknown mode: $MODE (want stories|profiles|full-sweep)" >&2; exit 2 ;; esac mkdir -p "$GDL_HOME/logs" diff --git a/scripts/systemd/gdl-sync@full-sweep.timer b/scripts/systemd/gdl-sync@full-sweep.timer new file mode 100644 index 0000000..235242c --- /dev/null +++ b/scripts/systemd/gdl-sync@full-sweep.timer @@ -0,0 +1,18 @@ +[Unit] +Description=Full Instagram archive sweep, no abort (~420 requests) +# The only run that enumerates every profile to the end, and so the only one +# that notices carousels edited after we archived them (test case 15). +# +# It is also by far the most expensive thing here: ~420 requests to +# instagram.com, the same order as the run that preceded the 2026-08-21 +# scraping warning, spent to catch a handful of retroactively edited posts. +# Consider running it by hand when you mean to, rather than on a timer. + +[Timer] +# Month names are not valid in OnCalendar's date field -- numeric only. +OnCalendar=*-01,04,07,10-07 04:00:00 +RandomizedDelaySec=45m +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/scripts/systemd/gdl-sync@full.timer b/scripts/systemd/gdl-sync@profiles.timer similarity index 65% rename from scripts/systemd/gdl-sync@full.timer rename to scripts/systemd/gdl-sync@profiles.timer index 9f998a4..131573c 100644 --- a/scripts/systemd/gdl-sync@full.timer +++ b/scripts/systemd/gdl-sync@profiles.timer @@ -1,5 +1,5 @@ [Unit] -Description=Monthly Instagram archive sync (all surfaces, --abort 50) +Description=Monthly Instagram profile sync (all surfaces, --abort 50) [Timer] OnCalendar=*-*-03 04:00:00 diff --git a/scripts/systemd/gdl-sync@sweep.timer b/scripts/systemd/gdl-sync@sweep.timer deleted file mode 100644 index 348a005..0000000 --- a/scripts/systemd/gdl-sync@sweep.timer +++ /dev/null @@ -1,13 +0,0 @@ -[Unit] -Description=Quarterly full Instagram archive sweep (no abort) -# The only run that enumerates every profile to the end, and so the only one -# that notices carousels edited after we archived them (test case 15). - -[Timer] -# Month names are not valid in OnCalendar's date field -- numeric only. -OnCalendar=*-01,04,07,10-07 04:00:00 -RandomizedDelaySec=45m -Persistent=true - -[Install] -WantedBy=timers.target