fix: make the scheduled runs as careful as the manual ones, and honest about failing
All three modes now use the pacing the 2026-08-22 runs used by hand after the
scraping warning -- 12-20s between requests, 5-10s between downloads, 500K --
against gdl-sync.py's defaults of 6-10 / 3-6 / 1M. Those values produced 0 400s
and 0 429s across 18 sources. An archive sync has no deadline; being slow is
free and being restricted is not.
Two things that made "scheduled" not mean much:
`stories` now uses --min-interval 8, not 20. At 20h the daily timer silently
did nothing whenever a manual run had happened the previous afternoon, which is
exactly what happened on 2026-08-21. The floor is there to stop an aborted
restart re-enumerating profiles -- minutes to hours -- and a stories fetch is
one request per profile, so 8h permits about twelve requests a day instead of
six. A run that skips every source now exits 75 and says so.
And the wrapper never reported failure at all. It ran the sync inside
`{ ...; exit $status; } | tee`, and the left side of a pipeline is a subshell,
so that exit set the subshell's status while the script returned tee's --
always 0. Every claim about cron mailing on failure was wrong. Fixed with
bash PIPESTATUS, hence the shebang change.
Verified with a stub sync across five cases: skipped-and-clean exits 75 with a
warning, a real failure propagates its own code, a normal run stays quiet, and
`full` skipping everything is not treated as an anomaly. Testing also caught
that the log name was only second-granular, so runs in the same second shared a
file and the skip check saw the previous run's output; the check now reads a
per-run temp file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
+21
-5
@@ -136,6 +136,21 @@ The skip-archive means an infrequent `full` costs barely more than a frequent
|
||||
one — it only fetches what is new. Frequency buys freshness, not completeness,
|
||||
except for stories.
|
||||
|
||||
**Pacing is deliberately slower than `gdl-sync.py`'s own defaults.** All three
|
||||
modes run at `--sleep-request 12 20 --sleep 5 10 --rate 500K`, against defaults
|
||||
of 6-10 / 3-6 / 1M. These are the values the 2026-08-22 runs used by hand after
|
||||
the scraping warning, and they produced 0 400s and 0 429s. An archive sync has
|
||||
no deadline: being slow is free, being restricted is not. Override with
|
||||
`GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE` — to raise them, not lower them.
|
||||
|
||||
`stories` also runs at `--min-interval 8` rather than the 20h default, because
|
||||
at 20h the daily timer silently did nothing whenever a manual run had happened
|
||||
the previous afternoon. The floor exists to stop an *aborted restart*
|
||||
re-enumerating profiles, which is a minutes-to-hours concern; a stories fetch
|
||||
is one request per profile, so 8h permits at worst about twelve requests in a
|
||||
day instead of six. **A stories run that skips every source now exits 75 and
|
||||
prints a warning** rather than reporting success.
|
||||
|
||||
## Scheduling — installed on `mattellite`
|
||||
|
||||
systemd **user** timers, running as `matt`, with lingering enabled so they fire
|
||||
@@ -378,11 +393,12 @@ broken; these are decisions not yet made and cleanups not yet done.
|
||||
automation behind it. Every day nobody runs `gdl-cron.sh stories` is a day
|
||||
of stories gone. That is the central unresolved tension: the cadence that
|
||||
protects stories is also the most machine-like pattern here.
|
||||
- **The scheduled modes have not been reconciled with the pacing used by
|
||||
hand.** The 2026-08-22 runs used `--sleep-request 12 20 --rate 500K`;
|
||||
`gdl-cron.sh` still passes the defaults (6-10s, 1M). Re-enabling the timers
|
||||
as they stand would make the automation *less* careful than the manual runs
|
||||
that followed a warning.
|
||||
- ~~The scheduled modes have not been reconciled with the pacing used by
|
||||
hand.~~ **Done 2026-08-22**: all modes now pass `--sleep-request 12 20
|
||||
--sleep 5 10 --rate 500K`, `stories` uses `--min-interval 8`, and a
|
||||
fully-skipped stories run exits 75 with a warning instead of looking like a
|
||||
success. The timers are still **disabled** — enabling them is a separate
|
||||
decision about cadence, not about pacing.
|
||||
- **`--abort 50` is opt-in.** `gdl-cron.sh full` passes it and the manual runs
|
||||
used it; `sweep` deliberately does not. It stops noticing **edited
|
||||
carousels** (test case 15), which only a full enumeration finds — which is
|
||||
|
||||
Reference in New Issue
Block a user