ergosteurandClaude Opus 5
7c7ce70ff9
`full` was the misleading one: it is the abort-LIMITED run, the one that
deliberately stops enumerating a profile as soon as it reaches content already
held. Calling it "full" invited exactly the wrong assumption about coverage.
And `sweep` gave no hint that it was the exhaustive one.
full -> profiles every surface, --abort 50, ~40-60 requests
sweep -> full-sweep every surface, no abort, ~420 requests
The old names now exit 2 with a pointer to the new one rather than a bare
"unknown mode", since muscle memory and any stray crontab will still use them.
full-sweep's description now says what it costs. At ~420 requests it is the
same order as the run that preceded the 2026-08-21 scraping warning, spent to
catch a handful of retroactively edited posts, so the docs suggest running it
by hand when you mean to rather than leaving it on a timer. Its cadence was
never actually agreed.
Units renamed to match and re-verified with systemd-analyze; the old ones are
removed from the host. All three timers remain disabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
113 lines
5.3 KiB
Bash
Executable File
113 lines
5.3 KiB
Bash
Executable File
#!/bin/bash
|
|
# Unattended wrapper around gdl-sync.py. One argument: the run mode.
|
|
#
|
|
# stories daily ~6 requests; the only surface that cannot be backfilled
|
|
# profiles monthly every surface, --abort 50: stops enumerating a profile
|
|
# once it reaches content already held, so it costs
|
|
# ~40-60 requests and catches everything NEW
|
|
# full-sweep rarely every surface, no abort: walks each profile to the end
|
|
# for ~420 requests. The only run that notices posts
|
|
# EDITED after we archived them, and by far the most
|
|
# expensive thing here -- see TOOLING.md before running.
|
|
#
|
|
# Exits non-zero if the sync does, so cron mails you. Everything is logged.
|
|
set -eu
|
|
|
|
MODE="${1:?usage: gdl-cron.sh stories|profiles|full-sweep}"
|
|
|
|
GDL_HOME="${GDL_HOME:-$HOME/gdl}"
|
|
INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
|
|
PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
|
|
STAGING="$GDL_HOME/staging-$MODE"
|
|
# $$ in the name because two runs in the same SECOND would otherwise share a
|
|
# log file, and `tee -a` appends -- which made a test see the previous run.
|
|
LOG="$GDL_HOME/logs/$MODE-$(date +%Y%m%d-%H%M%S)-$$.log"
|
|
|
|
PATH="$HOME/.local/bin:$PATH"; export PATH
|
|
|
|
# Pacing. These are the values the 2026-08-22 runs used by hand, after the
|
|
# scraping warning -- roughly double the caution of gdl-sync.py's own defaults
|
|
# (6-10s / 3-6s / 1M). An archive sync has no deadline; being slow is free and
|
|
# being restricted is not. Override per-run with GDL_SLEEP_REQUEST etc. if you
|
|
# ever need to, but raise them rather than lower them.
|
|
SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
|
|
SLEEP="${GDL_SLEEP:-5 10}"
|
|
RATE="${GDL_RATE:-500K}"
|
|
|
|
case "$MODE" in
|
|
# --min-interval 8, not the 20h default. The floor exists to stop an ABORTED
|
|
# RESTART re-enumerating profiles -- a minutes-to-hours concern. At 20h the
|
|
# daily timer silently did nothing whenever a manual run had happened the
|
|
# previous afternoon, which is exactly what happened on 2026-08-21: it fired,
|
|
# skipped all six sources and reported success. A stories fetch is one
|
|
# request per profile, so the worst case 8h permits is roughly twelve
|
|
# requests in a day instead of six.
|
|
stories) ARGS="--only stories --min-interval 8" ;;
|
|
profiles) ARGS="--only posts,reels,stories,highlights --abort 50" ;;
|
|
# No --abort: the whole point of full-sweep is enumerating to the end, so it
|
|
# is the only run that notices carousels edited after we archived them.
|
|
full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
|
|
# Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
|
|
# "sweep" did not say it was the exhaustive one. Catch the old names rather
|
|
# than failing with a bare error, in case something still passes them.
|
|
full) echo "mode 'full' was renamed to 'profiles'" >&2; exit 2 ;;
|
|
sweep) echo "mode 'sweep' was renamed to 'full-sweep'" >&2; exit 2 ;;
|
|
*) echo "unknown mode: $MODE (want stories|profiles|full-sweep)" >&2; exit 2 ;;
|
|
esac
|
|
|
|
mkdir -p "$GDL_HOME/logs"
|
|
|
|
# Staging is wiped every run ON PURPOSE. What we already hold is decided by the
|
|
# skip-archive (--download-archive), never by which files happen to be sitting
|
|
# in staging, so starting empty is correct -- and it keeps the publish rsync
|
|
# to just the new files instead of re-walking gigabytes each time.
|
|
rm -rf "$STAGING"
|
|
|
|
echo "=== $MODE run $(date -Is) ===" | tee -a "$LOG"
|
|
|
|
# The exit status has to survive the pipe into tee. The left-hand side of a
|
|
# pipeline runs in a SUBSHELL, so an `exit` in there sets the subshell's status
|
|
# and the script goes on to return tee's, which is always 0. An earlier version
|
|
# of this file did exactly that and reported success no matter what the sync
|
|
# did -- which is why the shebang is bash: PIPESTATUS is the fix.
|
|
# This run's output only. The check below must never see a previous run's
|
|
# lines, so it reads this rather than the (appended-to) log.
|
|
RUNOUT=$(mktemp)
|
|
trap 'rm -f "$RUNOUT"' EXIT
|
|
|
|
set +e
|
|
# shellcheck disable=SC2086
|
|
"$GDL_HOME/gdl-sync.py" \
|
|
--index "$INDEX" \
|
|
--staging "$STAGING" \
|
|
--publish "$PUBLISH" \
|
|
--archive-db "$GDL_HOME/artms.db" \
|
|
--urls-file "$GDL_HOME/artms_account_links.txt" \
|
|
--sleep-request $SLEEP_REQUEST \
|
|
--sleep $SLEEP \
|
|
--rate "$RATE" \
|
|
$ARGS --execute 2>&1 | tee -a "$LOG" "$RUNOUT"
|
|
status=${PIPESTATUS[0]}
|
|
set -e
|
|
|
|
# A stories run that skipped every source is NOT a success. It means the
|
|
# min-interval floor blocked the one surface that cannot be backfilled, and
|
|
# without this it looks identical to a clean run: exit 0, "0 step(s) failed".
|
|
# Note this is not the same as "no stories today" -- that shows up as sources
|
|
# being fetched and returning no results, which is normal and stays quiet.
|
|
if [ "$MODE" = "stories" ] && grep -q "sources : 0 to sync" "$RUNOUT"; then
|
|
echo "WARNING: every stories source was skipped by --min-interval." | tee -a "$LOG" >&2
|
|
echo " Nothing was fetched. Stories expire in 24h and cannot be" | tee -a "$LOG" >&2
|
|
echo " backfilled, so this is a real loss, not a quiet no-op." | tee -a "$LOG" >&2
|
|
[ "$status" -eq 0 ] && status=75
|
|
fi
|
|
|
|
echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
|
|
|
|
# Keep the log directory from growing without bound.
|
|
ls -1t "$GDL_HOME/logs" | tail -n +30 | while read -r old; do
|
|
rm -f "$GDL_HOME/logs/$old"
|
|
done
|
|
|
|
exit $status
|