refactor: rename the modes to say what they actually do
`full` was the misleading one: it is the abort-LIMITED run, the one that
deliberately stops enumerating a profile as soon as it reaches content already
held. Calling it "full" invited exactly the wrong assumption about coverage.
And `sweep` gave no hint that it was the exhaustive one.
full -> profiles every surface, --abort 50, ~40-60 requests
sweep -> full-sweep every surface, no abort, ~420 requests
The old names now exit 2 with a pointer to the new one rather than a bare
"unknown mode", since muscle memory and any stray crontab will still use them.
full-sweep's description now says what it costs. At ~420 requests it is the
same order as the run that preceded the 2026-08-21 scraping warning, spent to
catch a handful of retroactively edited posts, so the docs suggest running it
by hand when you mean to rather than leaving it on a timer. Its cadence was
never actually agreed.
Units renamed to match and re-verified with systemd-analyze; the old ones are
removed from the host. All three timers remain disabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
Claude Opus 5
+21
-16
@@ -63,7 +63,7 @@ every deletion because of it.
|
||||
| path | what it is |
|
||||
|---|---|
|
||||
| `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
|
||||
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `full` \| `sweep` |
|
||||
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
|
||||
| `scripts/systemd/` | the timers actually installed on the fetch host |
|
||||
| `scripts/test_gdl_sync.py` | its tests |
|
||||
| `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
|
||||
@@ -103,7 +103,7 @@ session-hijack detection looks for.
|
||||
|
||||
```sh
|
||||
ssh mattellite
|
||||
~/gdl/gdl-cron.sh full # or: stories | sweep
|
||||
~/gdl/gdl-cron.sh profiles # or: stories | full-sweep
|
||||
```
|
||||
|
||||
That is the whole thing: it wipes staging, fetches, and publishes straight to
|
||||
@@ -126,13 +126,17 @@ on cron's PATH and not on a non-login shell's either.
|
||||
|
||||
### The three modes
|
||||
|
||||
Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run —
|
||||
and `sweep` did not convey that it is the exhaustive one. The wrapper rejects
|
||||
the old names with a pointer rather than a bare error.
|
||||
|
||||
| mode | cadence | cost | why |
|
||||
|---|---|---|---|
|
||||
| `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped |
|
||||
| `full` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating once it reaches content already held |
|
||||
| `sweep` | quarterly | ~420 requests | no abort; the **only** run that notices carousels edited after we archived them (test case 15) |
|
||||
| `profiles` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating a profile once it reaches content already held. Catches everything **new** |
|
||||
| `full-sweep` | rarely, by hand | **~420 requests** | no abort; walks every profile to the end. The only run that notices posts **edited** after we archived them (test case 15) |
|
||||
|
||||
The skip-archive means an infrequent `full` costs barely more than a frequent
|
||||
The skip-archive means an infrequent `profiles` run costs barely more than a frequent
|
||||
one — it only fetches what is new. Frequency buys freshness, not completeness,
|
||||
except for stories.
|
||||
|
||||
@@ -164,8 +168,8 @@ systemctl --user list-timers 'gdl-sync@*'
|
||||
| unit | schedule | next fire (as installed) |
|
||||
|---|---|---|
|
||||
| `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working |
|
||||
| `gdl-sync@full.timer` | 3rd of each month, 04:00 | 04:37:44 |
|
||||
| `gdl-sync@sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
|
||||
| `gdl-sync@profiles.timer` | 3rd of each month, 04:00 | 04:37:44 |
|
||||
| `gdl-sync@full-sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
|
||||
|
||||
Unit files are version-controlled in `scripts/systemd/` and installed to
|
||||
`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
|
||||
@@ -186,9 +190,9 @@ Operating them:
|
||||
```sh
|
||||
export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed over non-interactive ssh
|
||||
systemctl --user start gdl-sync@stories.service # run one now
|
||||
systemctl --user status gdl-sync@full.timer
|
||||
systemctl --user status gdl-sync@profiles.timer
|
||||
journalctl --user -u 'gdl-sync@*' -n 50
|
||||
systemctl --user disable --now gdl-sync@sweep.timer # stop one
|
||||
systemctl --user disable --now gdl-sync@full-sweep.timer # stop one
|
||||
```
|
||||
|
||||
`systemctl --user` fails with "Failed to connect to bus" over ssh unless
|
||||
@@ -201,8 +205,8 @@ before installing it.
|
||||
|
||||
```cron
|
||||
17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
|
||||
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full
|
||||
11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep
|
||||
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh profiles
|
||||
11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full-sweep
|
||||
```
|
||||
|
||||
cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a
|
||||
@@ -234,7 +238,7 @@ highlights endpoint → suspension. If a run logs one, disable the timers and
|
||||
stop for the day:
|
||||
|
||||
```sh
|
||||
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@full.timer gdl-sync@sweep.timer
|
||||
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
|
||||
```
|
||||
|
||||
Then confirm the archive actually grew, from the workstation:
|
||||
@@ -333,7 +337,7 @@ twelve sources across six profiles, `official_artms` included at 1829 posts
|
||||
and 781 reels, finished in minutes for a few dozen requests where the old
|
||||
behaviour would have spent ~400.
|
||||
|
||||
Note the gap this leaves: **the scheduled `full` mode still uses the default
|
||||
Note the gap this leaves: **the scheduled `profiles` mode still uses the default
|
||||
6-10s pacing**, not the 12-20s used here. Reconcile that before re-enabling
|
||||
the timers, or the automation will be less careful than the hand runs that
|
||||
followed a warning.
|
||||
@@ -399,10 +403,11 @@ broken; these are decisions not yet made and cleanups not yet done.
|
||||
fully-skipped stories run exits 75 with a warning instead of looking like a
|
||||
success. The timers are still **disabled** — enabling them is a separate
|
||||
decision about cadence, not about pacing.
|
||||
- **`--abort 50` is opt-in.** `gdl-cron.sh full` passes it and the manual runs
|
||||
used it; `sweep` deliberately does not. It stops noticing **edited
|
||||
- **`--abort 50` is opt-in.** `gdl-cron.sh profiles` passes it and the manual
|
||||
runs used it; `full-sweep` deliberately does not. It stops noticing **edited
|
||||
carousels** (test case 15), which only a full enumeration finds — which is
|
||||
what `sweep` is for. The quarterly cadence was proposed and never agreed.
|
||||
what `full-sweep` is for. No cadence for it was ever agreed, and at ~420
|
||||
requests it is the riskiest thing on the schedule; prefer running it by hand.
|
||||
- **The `seeded` flags in `<db>.state.json` were hand-written**, reconstructed
|
||||
from the 2026-08-17 log rather than derived from the archive DB. They assert
|
||||
"the skip-archive already knows this source". If `artms.db` is ever rebuilt,
|
||||
|
||||
+18
-9
@@ -2,14 +2,18 @@
|
||||
# Unattended wrapper around gdl-sync.py. One argument: the run mode.
|
||||
#
|
||||
# stories daily ~6 requests; the only surface that cannot be backfilled
|
||||
# full monthly every surface, --abort 50 (stops at already-held content)
|
||||
# sweep quarterly every surface, no abort -- the only run that notices
|
||||
# carousels edited after we archived them
|
||||
# profiles monthly every surface, --abort 50: stops enumerating a profile
|
||||
# once it reaches content already held, so it costs
|
||||
# ~40-60 requests and catches everything NEW
|
||||
# full-sweep rarely every surface, no abort: walks each profile to the end
|
||||
# for ~420 requests. The only run that notices posts
|
||||
# EDITED after we archived them, and by far the most
|
||||
# expensive thing here -- see TOOLING.md before running.
|
||||
#
|
||||
# Exits non-zero if the sync does, so cron mails you. Everything is logged.
|
||||
set -eu
|
||||
|
||||
MODE="${1:?usage: gdl-cron.sh stories|full|sweep}"
|
||||
MODE="${1:?usage: gdl-cron.sh stories|profiles|full-sweep}"
|
||||
|
||||
GDL_HOME="${GDL_HOME:-$HOME/gdl}"
|
||||
INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
|
||||
@@ -39,11 +43,16 @@ case "$MODE" in
|
||||
# request per profile, so the worst case 8h permits is roughly twelve
|
||||
# requests in a day instead of six.
|
||||
stories) ARGS="--only stories --min-interval 8" ;;
|
||||
full) ARGS="--only posts,reels,stories,highlights --abort 50" ;;
|
||||
# No --abort: the whole point of sweep is enumerating to the end, so it is
|
||||
# the only run that notices carousels edited after we archived them.
|
||||
sweep) ARGS="--only posts,reels,stories,highlights" ;;
|
||||
*) echo "unknown mode: $MODE" >&2; exit 2 ;;
|
||||
profiles) ARGS="--only posts,reels,stories,highlights --abort 50" ;;
|
||||
# No --abort: the whole point of full-sweep is enumerating to the end, so it
|
||||
# is the only run that notices carousels edited after we archived them.
|
||||
full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
|
||||
# Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
|
||||
# "sweep" did not say it was the exhaustive one. Catch the old names rather
|
||||
# than failing with a bare error, in case something still passes them.
|
||||
full) echo "mode 'full' was renamed to 'profiles'" >&2; exit 2 ;;
|
||||
sweep) echo "mode 'sweep' was renamed to 'full-sweep'" >&2; exit 2 ;;
|
||||
*) echo "unknown mode: $MODE (want stories|profiles|full-sweep)" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
mkdir -p "$GDL_HOME/logs"
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
[Unit]
|
||||
Description=Full Instagram archive sweep, no abort (~420 requests)
|
||||
# The only run that enumerates every profile to the end, and so the only one
|
||||
# that notices carousels edited after we archived them (test case 15).
|
||||
#
|
||||
# It is also by far the most expensive thing here: ~420 requests to
|
||||
# instagram.com, the same order as the run that preceded the 2026-08-21
|
||||
# scraping warning, spent to catch a handful of retroactively edited posts.
|
||||
# Consider running it by hand when you mean to, rather than on a timer.
|
||||
|
||||
[Timer]
|
||||
# Month names are not valid in OnCalendar's date field -- numeric only.
|
||||
OnCalendar=*-01,04,07,10-07 04:00:00
|
||||
RandomizedDelaySec=45m
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=Monthly Instagram archive sync (all surfaces, --abort 50)
|
||||
Description=Monthly Instagram profile sync (all surfaces, --abort 50)
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-03 04:00:00
|
||||
@@ -1,13 +0,0 @@
|
||||
[Unit]
|
||||
Description=Quarterly full Instagram archive sweep (no abort)
|
||||
# The only run that enumerates every profile to the end, and so the only one
|
||||
# that notices carousels edited after we archived them (test case 15).
|
||||
|
||||
[Timer]
|
||||
# Month names are not valid in OnCalendar's date field -- numeric only.
|
||||
OnCalendar=*-01,04,07,10-07 04:00:00
|
||||
RandomizedDelaySec=45m
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user