ergosteur/instaarchive-viewer
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

refactor: rename the modes to say what they actually do

Browse Source 
`full` was the misleading one: it is the abort-LIMITED run, the one that
deliberately stops enumerating a profile as soon as it reaches content already
held. Calling it "full" invited exactly the wrong assumption about coverage.
And `sweep` gave no hint that it was the exhaustive one.

    full   -> profiles     every surface, --abort 50, ~40-60 requests
    sweep  -> full-sweep   every surface, no abort,   ~420 requests

The old names now exit 2 with a pointer to the new one rather than a bare
"unknown mode", since muscle memory and any stray crontab will still use them.

full-sweep's description now says what it costs. At ~420 requests it is the
same order as the run that preceded the 2026-08-21 scraping warning, spent to
catch a handful of retroactively edited posts, so the docs suggest running it
by hand when you mean to rather than leaving it on a timer. Its cadence was
never actually agreed.

Units renamed to match and re-verified with systemd-analyze; the old ones are
removed from the host. All three timers remain disabled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
ergosteur
2026-08-22 11:31:45 -04:00
co-authored by Claude Opus 5
parent 9c574e02eb
commit 7c7ce70ff9
5 changed files with 60 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TOOLING.md
+21 -16
View File
@@ -63,7 +63,7 @@ every deletion because of it.
| path | what it is | | path | what it is |
|---|---| |---|---|
| `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles | | `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `full` \| `sweep` | | `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
| `scripts/systemd/` | the timers actually installed on the fetch host | | `scripts/systemd/` | the timers actually installed on the fetch host |
| `scripts/test_gdl_sync.py` | its tests | | `scripts/test_gdl_sync.py` | its tests |
| `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere | | `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
@@ -103,7 +103,7 @@ session-hijack detection looks for.
```sh ```sh
ssh mattellite ssh mattellite
~/gdl/gdl-cron.sh full # or: stories | sweep ~/gdl/gdl-cron.sh profiles # or: stories | full-sweep
``` ```
That is the whole thing: it wipes staging, fetches, and publishes straight to That is the whole thing: it wipes staging, fetches, and publishes straight to
@@ -126,13 +126,17 @@ on cron's PATH and not on a non-login shell's either.
### The three modes ### The three modes
Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run —
and `sweep` did not convey that it is the exhaustive one. The wrapper rejects
the old names with a pointer rather than a bare error.
| mode | cadence | cost | why | | mode | cadence | cost | why |
|---|---|---|---| |---|---|---|---|
| `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped | | `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped |
| `full` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating once it reaches content already held | | `profiles` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating a profile once it reaches content already held. Catches everything **new** |
| `sweep` | quarterly | ~420 requests | no abort; the **only** run that notices carousels edited after we archived them (test case 15) | | `full-sweep` | rarely, by hand | **~420 requests** | no abort; walks every profile to the end. The only run that notices posts **edited** after we archived them (test case 15) |
The skip-archive means an infrequent `full` costs barely more than a frequent The skip-archive means an infrequent `profiles` run costs barely more than a frequent
one — it only fetches what is new. Frequency buys freshness, not completeness, one — it only fetches what is new. Frequency buys freshness, not completeness,
except for stories. except for stories.
@@ -164,8 +168,8 @@ systemctl --user list-timers 'gdl-sync@*'
| unit | schedule | next fire (as installed) | | unit | schedule | next fire (as installed) |
|---|---|---| |---|---|---|
| `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working | | `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working |
| `gdl-sync@full.timer` | 3rd of each month, 04:00 | 04:37:44 | | `gdl-sync@profiles.timer` | 3rd of each month, 04:00 | 04:37:44 |
| `gdl-sync@sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 | | `gdl-sync@full-sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
Unit files are version-controlled in `scripts/systemd/` and installed to Unit files are version-controlled in `scripts/systemd/` and installed to
`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes `~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
@@ -186,9 +190,9 @@ Operating them:
```sh ```sh
export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed over non-interactive ssh export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed over non-interactive ssh
systemctl --user start gdl-sync@stories.service # run one now systemctl --user start gdl-sync@stories.service # run one now
systemctl --user status gdl-sync@full.timer systemctl --user status gdl-sync@profiles.timer
journalctl --user -u 'gdl-sync@*' -n 50 journalctl --user -u 'gdl-sync@*' -n 50
systemctl --user disable --now gdl-sync@sweep.timer # stop one systemctl --user disable --now gdl-sync@full-sweep.timer # stop one
``` ```
`systemctl --user` fails with "Failed to connect to bus" over ssh unless `systemctl --user` fails with "Failed to connect to bus" over ssh unless
@@ -201,8 +205,8 @@ before installing it.
```cron ```cron
17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories 17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full 43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh profiles
11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep 11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full-sweep
``` ```
cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a
@@ -234,7 +238,7 @@ highlights endpoint → suspension. If a run logs one, disable the timers and
stop for the day: stop for the day:
```sh ```sh
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@full.timer gdl-sync@sweep.timer systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
``` ```
Then confirm the archive actually grew, from the workstation: Then confirm the archive actually grew, from the workstation:
@@ -333,7 +337,7 @@ twelve sources across six profiles, `official_artms` included at 1829 posts
and 781 reels, finished in minutes for a few dozen requests where the old and 781 reels, finished in minutes for a few dozen requests where the old
behaviour would have spent ~400. behaviour would have spent ~400.
Note the gap this leaves: **the scheduled `full` mode still uses the default Note the gap this leaves: **the scheduled `profiles` mode still uses the default
6-10s pacing**, not the 12-20s used here. Reconcile that before re-enabling 6-10s pacing**, not the 12-20s used here. Reconcile that before re-enabling
the timers, or the automation will be less careful than the hand runs that the timers, or the automation will be less careful than the hand runs that
followed a warning. followed a warning.
@@ -399,10 +403,11 @@ broken; these are decisions not yet made and cleanups not yet done.
fully-skipped stories run exits 75 with a warning instead of looking like a fully-skipped stories run exits 75 with a warning instead of looking like a
success. The timers are still **disabled** — enabling them is a separate success. The timers are still **disabled** — enabling them is a separate
decision about cadence, not about pacing. decision about cadence, not about pacing.
- **`--abort 50` is opt-in.** `gdl-cron.sh full` passes it and the manual runs - **`--abort 50` is opt-in.** `gdl-cron.sh profiles` passes it and the manual
used it; `sweep` deliberately does not. It stops noticing **edited runs used it; `full-sweep` deliberately does not. It stops noticing **edited
carousels** (test case 15), which only a full enumeration finds — which is carousels** (test case 15), which only a full enumeration finds — which is
what `sweep` is for. The quarterly cadence was proposed and never agreed. what `full-sweep` is for. No cadence for it was ever agreed, and at ~420
requests it is the riskiest thing on the schedule; prefer running it by hand.
- **The `seeded` flags in `<db>.state.json` were hand-written**, reconstructed - **The `seeded` flags in `<db>.state.json` were hand-written**, reconstructed
from the 2026-08-17 log rather than derived from the archive DB. They assert from the 2026-08-17 log rather than derived from the archive DB. They assert
"the skip-archive already knows this source". If `artms.db` is ever rebuilt, "the skip-archive already knows this source". If `artms.db` is ever rebuilt,
scripts/gdl-cron.sh
+18 -9
View File
@@ -2,14 +2,18 @@
# Unattended wrapper around gdl-sync.py. One argument: the run mode. # Unattended wrapper around gdl-sync.py. One argument: the run mode.
# #
# stories daily ~6 requests; the only surface that cannot be backfilled # stories daily ~6 requests; the only surface that cannot be backfilled
# full monthly every surface, --abort 50 (stops at already-held content) # profiles monthly every surface, --abort 50: stops enumerating a profile
# sweep quarterly every surface, no abort -- the only run that notices # once it reaches content already held, so it costs
# carousels edited after we archived them # ~40-60 requests and catches everything NEW
# full-sweep rarely every surface, no abort: walks each profile to the end
# for ~420 requests. The only run that notices posts
# EDITED after we archived them, and by far the most
# expensive thing here -- see TOOLING.md before running.
# #
# Exits non-zero if the sync does, so cron mails you. Everything is logged. # Exits non-zero if the sync does, so cron mails you. Everything is logged.
set -eu set -eu
MODE="${1:?usage: gdl-cron.sh stories|full|sweep}" MODE="${1:?usage: gdl-cron.sh stories|profiles|full-sweep}"
GDL_HOME="${GDL_HOME:-$HOME/gdl}" GDL_HOME="${GDL_HOME:-$HOME/gdl}"
INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}" INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
@@ -39,11 +43,16 @@ case "$MODE" in
# request per profile, so the worst case 8h permits is roughly twelve # request per profile, so the worst case 8h permits is roughly twelve
# requests in a day instead of six. # requests in a day instead of six.
stories) ARGS="--only stories --min-interval 8" ;; stories) ARGS="--only stories --min-interval 8" ;;
full) ARGS="--only posts,reels,stories,highlights --abort 50" ;; profiles) ARGS="--only posts,reels,stories,highlights --abort 50" ;;
# No --abort: the whole point of sweep is enumerating to the end, so it is # No --abort: the whole point of full-sweep is enumerating to the end, so it
# the only run that notices carousels edited after we archived them. # is the only run that notices carousels edited after we archived them.
sweep) ARGS="--only posts,reels,stories,highlights" ;; full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
*) echo "unknown mode: $MODE" >&2; exit 2 ;; # Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
# "sweep" did not say it was the exhaustive one. Catch the old names rather
# than failing with a bare error, in case something still passes them.
full) echo "mode 'full' was renamed to 'profiles'" >&2; exit 2 ;;
sweep) echo "mode 'sweep' was renamed to 'full-sweep'" >&2; exit 2 ;;
*) echo "unknown mode: $MODE (want stories|profiles|full-sweep)" >&2; exit 2 ;;
esac esac
mkdir -p "$GDL_HOME/logs" mkdir -p "$GDL_HOME/logs"
scripts/systemd/gdl-sync@full-sweep.timer
+18
View File
@@ -0,0 +1,18 @@
[Unit]
Description=Full Instagram archive sweep, no abort (~420 requests)
# The only run that enumerates every profile to the end, and so the only one
# that notices carousels edited after we archived them (test case 15).
#
# It is also by far the most expensive thing here: ~420 requests to
# instagram.com, the same order as the run that preceded the 2026-08-21
# scraping warning, spent to catch a handful of retroactively edited posts.
# Consider running it by hand when you mean to, rather than on a timer.
[Timer]
# Month names are not valid in OnCalendar's date field -- numeric only.
OnCalendar=*-01,04,07,10-07 04:00:00
RandomizedDelaySec=45m
Persistent=true
[Install]
WantedBy=timers.target
scripts/systemd/gdl-sync@full.timer → scripts/systemd/gdl-sync@profiles.timer
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit] [Unit]
Description=Monthly Instagram archive sync (all surfaces, --abort 50) Description=Monthly Instagram profile sync (all surfaces, --abort 50)
[Timer] [Timer]
OnCalendar=*-*-03 04:00:00 OnCalendar=*-*-03 04:00:00
scripts/systemd/gdl-sync@sweep.timer
-13
View File
@@ -1,13 +0,0 @@
[Unit]
Description=Quarterly full Instagram archive sweep (no abort)
# The only run that enumerates every profile to the end, and so the only one
# that notices carousels edited after we archived them (test case 15).
[Timer]
# Month names are not valid in OnCalendar's date field -- numeric only.
OnCalendar=*-01,04,07,10-07 04:00:00
RandomizedDelaySec=45m
Persistent=true
[Install]
WantedBy=timers.target