ergosteur/instaarchive-viewer
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

refactor: rename the modes to say what they actually do

Browse Source 
`full` was the misleading one: it is the abort-LIMITED run, the one that
deliberately stops enumerating a profile as soon as it reaches content already
held. Calling it "full" invited exactly the wrong assumption about coverage.
And `sweep` gave no hint that it was the exhaustive one.

    full   -> profiles     every surface, --abort 50, ~40-60 requests
    sweep  -> full-sweep   every surface, no abort,   ~420 requests

The old names now exit 2 with a pointer to the new one rather than a bare
"unknown mode", since muscle memory and any stray crontab will still use them.

full-sweep's description now says what it costs. At ~420 requests it is the
same order as the run that preceded the 2026-08-21 scraping warning, spent to
catch a handful of retroactively edited posts, so the docs suggest running it
by hand when you mean to rather than leaving it on a timer. Its cadence was
never actually agreed.

Units renamed to match and re-verified with systemd-analyze; the old ones are
removed from the host. All three timers remain disabled.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
ergosteur
2026-08-22 11:31:45 -04:00
co-authored by Claude Opus 5
parent 9c574e02eb
commit 7c7ce70ff9
5 changed files with 60 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TOOLING.md
+21 -16
View File
@@ -63,7 +63,7 @@ every deletion because of it.
| path | what it is |
|---|---|
| `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `full` \| `sweep` |
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
| `scripts/systemd/` | the timers actually installed on the fetch host |
| `scripts/test_gdl_sync.py` | its tests |
| `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
@@ -103,7 +103,7 @@ session-hijack detection looks for.
```sh
ssh mattellite
~/gdl/gdl-cron.sh full # or: stories | sweep
~/gdl/gdl-cron.sh profiles # or: stories | full-sweep
```
That is the whole thing: it wipes staging, fetches, and publishes straight to
@@ -126,13 +126,17 @@ on cron's PATH and not on a non-login shell's either.
### The three modes
Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run —
and `sweep` did not convey that it is the exhaustive one. The wrapper rejects
the old names with a pointer rather than a bare error.
| mode | cadence | cost | why |
|---|---|---|---|
| `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped |
| `full` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating once it reaches content already held |
| `sweep` | quarterly | ~420 requests | no abort; the **only** run that notices carousels edited after we archived them (test case 15) |
| `profiles` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating a profile once it reaches content already held. Catches everything **new** |
| `full-sweep` | rarely, by hand | **~420 requests** | no abort; walks every profile to the end. The only run that notices posts **edited** after we archived them (test case 15) |
The skip-archive means an infrequent `full` costs barely more than a frequent
The skip-archive means an infrequent `profiles` run costs barely more than a frequent
one — it only fetches what is new. Frequency buys freshness, not completeness,
except for stories.
@@ -164,8 +168,8 @@ systemctl --user list-timers 'gdl-sync@*'
| unit | schedule | next fire (as installed) |
|---|---|---|
| `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working |
| `gdl-sync@full.timer` | 3rd of each month, 04:00 | 04:37:44 |
| `gdl-sync@sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
| `gdl-sync@profiles.timer` | 3rd of each month, 04:00 | 04:37:44 |
| `gdl-sync@full-sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
Unit files are version-controlled in `scripts/systemd/` and installed to
`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
@@ -186,9 +190,9 @@ Operating them:
```sh
export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed over non-interactive ssh
systemctl --user start gdl-sync@stories.service # run one now
systemctl --user status gdl-sync@full.timer
systemctl --user status gdl-sync@profiles.timer
journalctl --user -u 'gdl-sync@*' -n 50
systemctl --user disable --now gdl-sync@sweep.timer # stop one
systemctl --user disable --now gdl-sync@full-sweep.timer # stop one
```
`systemctl --user` fails with "Failed to connect to bus" over ssh unless
@@ -201,8 +205,8 @@ before installing it.
```cron
17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full
11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh profiles
11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full-sweep
```
cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a
@@ -234,7 +238,7 @@ highlights endpoint → suspension. If a run logs one, disable the timers and
stop for the day:
```sh
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@full.timer gdl-sync@sweep.timer
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
```
Then confirm the archive actually grew, from the workstation:
@@ -333,7 +337,7 @@ twelve sources across six profiles, `official_artms` included at 1829 posts
and 781 reels, finished in minutes for a few dozen requests where the old
behaviour would have spent ~400.
Note the gap this leaves: **the scheduled `full` mode still uses the default
Note the gap this leaves: **the scheduled `profiles` mode still uses the default
6-10s pacing**, not the 12-20s used here. Reconcile that before re-enabling
the timers, or the automation will be less careful than the hand runs that
followed a warning.
@@ -399,10 +403,11 @@ broken; these are decisions not yet made and cleanups not yet done.
fully-skipped stories run exits 75 with a warning instead of looking like a
success. The timers are still **disabled** — enabling them is a separate
decision about cadence, not about pacing.
- **`--abort 50` is opt-in.** `gdl-cron.sh full` passes it and the manual runs
used it; `sweep` deliberately does not. It stops noticing **edited
- **`--abort 50` is opt-in.** `gdl-cron.sh profiles` passes it and the manual
runs used it; `full-sweep` deliberately does not. It stops noticing **edited
carousels** (test case 15), which only a full enumeration finds — which is
what `sweep` is for. The quarterly cadence was proposed and never agreed.
what `full-sweep` is for. No cadence for it was ever agreed, and at ~420
requests it is the riskiest thing on the schedule; prefer running it by hand.
- **The `seeded` flags in `<db>.state.json` were hand-written**, reconstructed
from the 2026-08-17 log rather than derived from the archive DB. They assert
"the skip-archive already knows this source". If `artms.db` is ever rebuilt,