refactor: rename the modes to say what they actually do
`full` was the misleading one: it is the abort-LIMITED run, the one that
deliberately stops enumerating a profile as soon as it reaches content already
held. Calling it "full" invited exactly the wrong assumption about coverage.
And `sweep` gave no hint that it was the exhaustive one.
full -> profiles every surface, --abort 50, ~40-60 requests
sweep -> full-sweep every surface, no abort, ~420 requests
The old names now exit 2 with a pointer to the new one rather than a bare
"unknown mode", since muscle memory and any stray crontab will still use them.
full-sweep's description now says what it costs. At ~420 requests it is the
same order as the run that preceded the 2026-08-21 scraping warning, spent to
catch a handful of retroactively edited posts, so the docs suggest running it
by hand when you mean to rather than leaving it on a timer. Its cadence was
never actually agreed.
Units renamed to match and re-verified with systemd-analyze; the old ones are
removed from the host. All three timers remain disabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
+20
-11
@@ -1,15 +1,19 @@
|
||||
#!/bin/bash
|
||||
# Unattended wrapper around gdl-sync.py. One argument: the run mode.
|
||||
#
|
||||
# stories daily ~6 requests; the only surface that cannot be backfilled
|
||||
# full monthly every surface, --abort 50 (stops at already-held content)
|
||||
# sweep quarterly every surface, no abort -- the only run that notices
|
||||
# carousels edited after we archived them
|
||||
# stories daily ~6 requests; the only surface that cannot be backfilled
|
||||
# profiles monthly every surface, --abort 50: stops enumerating a profile
|
||||
# once it reaches content already held, so it costs
|
||||
# ~40-60 requests and catches everything NEW
|
||||
# full-sweep rarely every surface, no abort: walks each profile to the end
|
||||
# for ~420 requests. The only run that notices posts
|
||||
# EDITED after we archived them, and by far the most
|
||||
# expensive thing here -- see TOOLING.md before running.
|
||||
#
|
||||
# Exits non-zero if the sync does, so cron mails you. Everything is logged.
|
||||
set -eu
|
||||
|
||||
MODE="${1:?usage: gdl-cron.sh stories|full|sweep}"
|
||||
MODE="${1:?usage: gdl-cron.sh stories|profiles|full-sweep}"
|
||||
|
||||
GDL_HOME="${GDL_HOME:-$HOME/gdl}"
|
||||
INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
|
||||
@@ -38,12 +42,17 @@ case "$MODE" in
|
||||
# skipped all six sources and reported success. A stories fetch is one
|
||||
# request per profile, so the worst case 8h permits is roughly twelve
|
||||
# requests in a day instead of six.
|
||||
stories) ARGS="--only stories --min-interval 8" ;;
|
||||
full) ARGS="--only posts,reels,stories,highlights --abort 50" ;;
|
||||
# No --abort: the whole point of sweep is enumerating to the end, so it is
|
||||
# the only run that notices carousels edited after we archived them.
|
||||
sweep) ARGS="--only posts,reels,stories,highlights" ;;
|
||||
*) echo "unknown mode: $MODE" >&2; exit 2 ;;
|
||||
stories) ARGS="--only stories --min-interval 8" ;;
|
||||
profiles) ARGS="--only posts,reels,stories,highlights --abort 50" ;;
|
||||
# No --abort: the whole point of full-sweep is enumerating to the end, so it
|
||||
# is the only run that notices carousels edited after we archived them.
|
||||
full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
|
||||
# Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
|
||||
# "sweep" did not say it was the exhaustive one. Catch the old names rather
|
||||
# than failing with a bare error, in case something still passes them.
|
||||
full) echo "mode 'full' was renamed to 'profiles'" >&2; exit 2 ;;
|
||||
sweep) echo "mode 'sweep' was renamed to 'full-sweep'" >&2; exit 2 ;;
|
||||
*) echo "unknown mode: $MODE (want stories|profiles|full-sweep)" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
mkdir -p "$GDL_HOME/logs"
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
[Unit]
|
||||
Description=Full Instagram archive sweep, no abort (~420 requests)
|
||||
# The only run that enumerates every profile to the end, and so the only one
|
||||
# that notices carousels edited after we archived them (test case 15).
|
||||
#
|
||||
# It is also by far the most expensive thing here: ~420 requests to
|
||||
# instagram.com, the same order as the run that preceded the 2026-08-21
|
||||
# scraping warning, spent to catch a handful of retroactively edited posts.
|
||||
# Consider running it by hand when you mean to, rather than on a timer.
|
||||
|
||||
[Timer]
|
||||
# Month names are not valid in OnCalendar's date field -- numeric only.
|
||||
OnCalendar=*-01,04,07,10-07 04:00:00
|
||||
RandomizedDelaySec=45m
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=Monthly Instagram archive sync (all surfaces, --abort 50)
|
||||
Description=Monthly Instagram profile sync (all surfaces, --abort 50)
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-03 04:00:00
|
||||
@@ -1,13 +0,0 @@
|
||||
[Unit]
|
||||
Description=Quarterly full Instagram archive sweep (no abort)
|
||||
# The only run that enumerates every profile to the end, and so the only one
|
||||
# that notices carousels edited after we archived them (test case 15).
|
||||
|
||||
[Timer]
|
||||
# Month names are not valid in OnCalendar's date field -- numeric only.
|
||||
OnCalendar=*-01,04,07,10-07 04:00:00
|
||||
RandomizedDelaySec=45m
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user