fix: make the scheduled runs as careful as the manual ones, and honest about failing

All three modes now use the pacing the 2026-08-22 runs used by hand after the
scraping warning -- 12-20s between requests, 5-10s between downloads, 500K --
against gdl-sync.py's defaults of 6-10 / 3-6 / 1M. Those values produced 0 400s
and 0 429s across 18 sources. An archive sync has no deadline; being slow is
free and being restricted is not.

Two things that made "scheduled" not mean much:

`stories` now uses --min-interval 8, not 20. At 20h the daily timer silently
did nothing whenever a manual run had happened the previous afternoon, which is
exactly what happened on 2026-08-21. The floor is there to stop an aborted
restart re-enumerating profiles -- minutes to hours -- and a stories fetch is
one request per profile, so 8h permits about twelve requests a day instead of
six. A run that skips every source now exits 75 and says so.

And the wrapper never reported failure at all. It ran the sync inside
`{ ...; exit $status; } | tee`, and the left side of a pipeline is a subshell,
so that exit set the subshell's status while the script returned tee's --
always 0. Every claim about cron mailing on failure was wrong. Fixed with
bash PIPESTATUS, hence the shebang change.

Verified with a stub sync across five cases: skipped-and-clean exits 75 with a
warning, a real failure propagates its own code, a normal run stays quiet, and
`full` skipping everything is not treated as an anomaly. Testing also caught
that the log name was only second-granular, so runs in the same second shared a
file and the skip check saw the previous run's output; the check now reads a
per-run temp file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF

TOOLING.md

@@ -136,6 +136,21 @@ The skip-archive means an infrequent `full` costs barely more than a frequent
 one — it only fetches what is new. Frequency buys freshness, not completeness,
 except for stories.
 
+**Pacing is deliberately slower than `gdl-sync.py`'s own defaults.** All three
+modes run at `--sleep-request 12 20 --sleep 5 10 --rate 500K`, against defaults
+of 6-10 / 3-6 / 1M. These are the values the 2026-08-22 runs used by hand after
+the scraping warning, and they produced 0 400s and 0 429s. An archive sync has
+no deadline: being slow is free, being restricted is not. Override with
+`GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE` — to raise them, not lower them.
+
+`stories` also runs at `--min-interval 8` rather than the 20h default, because
+at 20h the daily timer silently did nothing whenever a manual run had happened
+the previous afternoon. The floor exists to stop an *aborted restart*
+re-enumerating profiles, which is a minutes-to-hours concern; a stories fetch
+is one request per profile, so 8h permits at worst about twelve requests in a
+day instead of six. **A stories run that skips every source now exits 75 and
+prints a warning** rather than reporting success.
+
 ## Scheduling — installed on `mattellite`
 
 systemd **user** timers, running as `matt`, with lingering enabled so they fire
@@ -378,11 +393,12 @@ broken; these are decisions not yet made and cleanups not yet done.
   automation behind it. Every day nobody runs `gdl-cron.sh stories` is a day
   of stories gone. That is the central unresolved tension: the cadence that
   protects stories is also the most machine-like pattern here.
-- **The scheduled modes have not been reconciled with the pacing used by
+- ~~The scheduled modes have not been reconciled with the pacing used by
-  hand.** The 2026-08-22 runs used `--sleep-request 12 20 --rate 500K`;
+  hand.~~ **Done 2026-08-22**: all modes now pass `--sleep-request 12 20
-  `gdl-cron.sh` still passes the defaults (6-10s, 1M). Re-enabling the timers
+  --sleep 5 10 --rate 500K`, `stories` uses `--min-interval 8`, and a
-  as they stand would make the automation *less* careful than the manual runs
+  fully-skipped stories run exits 75 with a warning instead of looking like a
-  that followed a warning.
+  success. The timers are still **disabled** — enabling them is a separate
+  decision about cadence, not about pacing.
 - **`--abort 50` is opt-in.** `gdl-cron.sh full` passes it and the manual runs
   used it; `sweep` deliberately does not. It stops noticing **edited
   carousels** (test case 15), which only a full enumeration finds — which is

scripts/gdl-cron.sh

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 # Unattended wrapper around gdl-sync.py. One argument: the run mode.
 #
 #   stories   daily      ~6 requests; the only surface that cannot be backfilled
@@ -15,13 +15,33 @@ GDL_HOME="${GDL_HOME:-$HOME/gdl}"
 INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
 PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
 STAGING="$GDL_HOME/staging-$MODE"
-LOG="$GDL_HOME/logs/$MODE-$(date +%Y%m%d-%H%M%S).log"
+# $$ in the name because two runs in the same SECOND would otherwise share a
+# log file, and `tee -a` appends -- which made a test see the previous run.
+LOG="$GDL_HOME/logs/$MODE-$(date +%Y%m%d-%H%M%S)-$$.log"
 
 PATH="$HOME/.local/bin:$PATH"; export PATH
 
+# Pacing. These are the values the 2026-08-22 runs used by hand, after the
+# scraping warning -- roughly double the caution of gdl-sync.py's own defaults
+# (6-10s / 3-6s / 1M). An archive sync has no deadline; being slow is free and
+# being restricted is not. Override per-run with GDL_SLEEP_REQUEST etc. if you
+# ever need to, but raise them rather than lower them.
+SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
+SLEEP="${GDL_SLEEP:-5 10}"
+RATE="${GDL_RATE:-500K}"
+
 case "$MODE" in
-  stories) ARGS="--only stories" ;;
+  # --min-interval 8, not the 20h default. The floor exists to stop an ABORTED
+  # RESTART re-enumerating profiles -- a minutes-to-hours concern. At 20h the
+  # daily timer silently did nothing whenever a manual run had happened the
+  # previous afternoon, which is exactly what happened on 2026-08-21: it fired,
+  # skipped all six sources and reported success. A stories fetch is one
+  # request per profile, so the worst case 8h permits is roughly twelve
+  # requests in a day instead of six.
+  stories) ARGS="--only stories --min-interval 8" ;;
   full)    ARGS="--only posts,reels,stories,highlights --abort 50" ;;
+  # No --abort: the whole point of sweep is enumerating to the end, so it is
+  # the only run that notices carousels edited after we archived them.
   sweep)   ARGS="--only posts,reels,stories,highlights" ;;
   *) echo "unknown mode: $MODE" >&2; exit 2 ;;
 esac
@@ -34,8 +54,19 @@ mkdir -p "$GDL_HOME/logs"
 # to just the new files instead of re-walking gigabytes each time.
 rm -rf "$STAGING"
 
-{
+echo "=== $MODE run $(date -Is) ===" | tee -a "$LOG"
-  echo "=== $MODE run $(date -Is) ==="
+
+# The exit status has to survive the pipe into tee. The left-hand side of a
+# pipeline runs in a SUBSHELL, so an `exit` in there sets the subshell's status
+# and the script goes on to return tee's, which is always 0. An earlier version
+# of this file did exactly that and reported success no matter what the sync
+# did -- which is why the shebang is bash: PIPESTATUS is the fix.
+# This run's output only. The check below must never see a previous run's
+# lines, so it reads this rather than the (appended-to) log.
+RUNOUT=$(mktemp)
+trap 'rm -f "$RUNOUT"' EXIT
+
+set +e
 # shellcheck disable=SC2086
 "$GDL_HOME/gdl-sync.py" \
     --index "$INDEX" \
@@ -43,13 +74,30 @@ rm -rf "$STAGING"
     --publish "$PUBLISH" \
     --archive-db "$GDL_HOME/artms.db" \
     --urls-file "$GDL_HOME/artms_account_links.txt" \
-      $ARGS --execute
+    --sleep-request $SLEEP_REQUEST \
-  status=$?
+    --sleep $SLEEP \
-  echo "=== exit $status at $(date -Is) ==="
+    --rate "$RATE" \
-  exit $status
+    $ARGS --execute 2>&1 | tee -a "$LOG" "$RUNOUT"
-} 2>&1 | tee -a "$LOG"
+status=${PIPESTATUS[0]}
+set -e
+
+# A stories run that skipped every source is NOT a success. It means the
+# min-interval floor blocked the one surface that cannot be backfilled, and
+# without this it looks identical to a clean run: exit 0, "0 step(s) failed".
+# Note this is not the same as "no stories today" -- that shows up as sources
+# being fetched and returning no results, which is normal and stays quiet.
+if [ "$MODE" = "stories" ] && grep -q "sources  : 0 to sync" "$RUNOUT"; then
+  echo "WARNING: every stories source was skipped by --min-interval." | tee -a "$LOG" >&2
+  echo "         Nothing was fetched. Stories expire in 24h and cannot be" | tee -a "$LOG" >&2
+  echo "         backfilled, so this is a real loss, not a quiet no-op." | tee -a "$LOG" >&2
+  [ "$status" -eq 0 ] && status=75
+fi
+
+echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
 
 # Keep the log directory from growing without bound.
 ls -1t "$GDL_HOME/logs" | tail -n +30 | while read -r old; do
   rm -f "$GDL_HOME/logs/$old"
 done
+
+exit $status