All three modes now use the pacing the 2026-08-22 runs used by hand after the
scraping warning -- 12-20s between requests, 5-10s between downloads, 500K --
against gdl-sync.py's defaults of 6-10 / 3-6 / 1M. Those values produced 0 400s
and 0 429s across 18 sources. An archive sync has no deadline; being slow is
free and being restricted is not.
Two things that made "scheduled" not mean much:
`stories` now uses --min-interval 8, not 20. At 20h the daily timer silently
did nothing whenever a manual run had happened the previous afternoon, which is
exactly what happened on 2026-08-21. The floor is there to stop an aborted
restart re-enumerating profiles -- minutes to hours -- and a stories fetch is
one request per profile, so 8h permits about twelve requests a day instead of
six. A run that skips every source now exits 75 and says so.
And the wrapper never reported failure at all. It ran the sync inside
`{ ...; exit $status; } | tee`, and the left side of a pipeline is a subshell,
so that exit set the subshell's status while the script returned tee's --
always 0. Every claim about cron mailing on failure was wrong. Fixed with
bash PIPESTATUS, hence the shebang change.
Verified with a stub sync across five cases: skipped-and-clean exits 75 with a
warning, a real failure propagates its own code, a normal run stays quiet, and
`full` skipping everything is not treated as an anomaly. Testing also caught
that the log name was only second-granular, so runs in the same second shared a
file and the skip check saw the previous run's output; the check now reads a
per-run temp file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
gdl-sync.py always needed a fistful of paths on the command line, and the
2026-08-20 run published to a local directory and then rsynced to the NAS by
hand -- fine interactively, impossible from cron. gdl-cron.sh takes one
argument (stories|full|sweep) and does the whole thing.
Staging is wiped every run deliberately: what we already hold is decided by
the skip-archive, never by what happens to be sitting in staging, so starting
empty is correct and keeps the publish to just the new files.
Documents the three cadences and what each is for -- notably that only `sweep`
notices carousels edited after we archived them, and that only `stories` loses
content if it is skipped, since stories cannot be backfilled.
Cron notes the two things that would silently break it: cron runs /bin/sh so
$RANDOM does not exist (use shuf), and `%` in a crontab line means newline.
systemd is offered as the better option because RandomizedDelaySec and
Persistent=true are built in -- a host asleep at 09:00 silently skips a cron
job, and a skipped stories run is content gone for good.
The publish step is still blocked on mattellite having no key on the NAS; that
is recorded under Outstanding as the one thing between this and unattended.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
ergosteurandClaude Opus 5