The hand-run example had no pacing flags, so following it ran at gdl-sync.py's
defaults -- 6-10s and 1M -- which since yesterday is roughly half the caution
the wrapper applies. A documented manual procedure that is less careful than
the automation is backwards, and that is what this file said until now.
Adds the parts that were only ever in a transcript:
- how to detach a long run, and why `pkill -f <pattern>` over ssh kills your
own shell instead of the sync (the pattern matches the ssh command line --
this happened twice yesterday)
- that nothing reaches the archive until a run finishes, so killing one midway
is safe
- a "when a run fails" runbook in cost order: read the log, check the cookies,
then look at the browser. The first two are free.
The two results that look like failures and are not: rsync exit 23 is the
chown to rslsync failing because the ssh user is not root, and "No results"
just means a profile has no active story.
The cookie check is the one worth having written down -- exporting them and
looking at the shape of sessionid separates "Chrome decryption broke locally"
from "the account is in trouble" without spending a single request, and it is
what ruled out the wrong explanation yesterday.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
`full` was the misleading one: it is the abort-LIMITED run, the one that
deliberately stops enumerating a profile as soon as it reaches content already
held. Calling it "full" invited exactly the wrong assumption about coverage.
And `sweep` gave no hint that it was the exhaustive one.
full -> profiles every surface, --abort 50, ~40-60 requests
sweep -> full-sweep every surface, no abort, ~420 requests
The old names now exit 2 with a pointer to the new one rather than a bare
"unknown mode", since muscle memory and any stray crontab will still use them.
full-sweep's description now says what it costs. At ~420 requests it is the
same order as the run that preceded the 2026-08-21 scraping warning, spent to
catch a handful of retroactively edited posts, so the docs suggest running it
by hand when you mean to rather than leaving it on a timer. Its cadence was
never actually agreed.
Units renamed to match and re-verified with systemd-analyze; the old ones are
removed from the host. All three timers remain disabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
All three modes now use the pacing the 2026-08-22 runs used by hand after the
scraping warning -- 12-20s between requests, 5-10s between downloads, 500K --
against gdl-sync.py's defaults of 6-10 / 3-6 / 1M. Those values produced 0 400s
and 0 429s across 18 sources. An archive sync has no deadline; being slow is
free and being restricted is not.
Two things that made "scheduled" not mean much:
`stories` now uses --min-interval 8, not 20. At 20h the daily timer silently
did nothing whenever a manual run had happened the previous afternoon, which is
exactly what happened on 2026-08-21. The floor is there to stop an aborted
restart re-enumerating profiles -- minutes to hours -- and a stories fetch is
one request per profile, so 8h permits about twelve requests a day instead of
six. A run that skips every source now exits 75 and says so.
And the wrapper never reported failure at all. It ran the sync inside
`{ ...; exit $status; } | tee`, and the left side of a pipeline is a subshell,
so that exit set the subshell's status while the script returned tee's --
always 0. Every claim about cron mailing on failure was wrong. Fixed with
bash PIPESTATUS, hence the shebang change.
Verified with a stub sync across five cases: skipped-and-clean exits 75 with a
warning, a real failure propagates its own code, a normal run stays quiet, and
`full` skipping everything is not treated as an anomaly. Testing also caught
that the log name was only second-granular, so runs in the same second shared a
file and the skip check saw the previous run's output; the check now reads a
per-run temp file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Four claims went stale over two days and would each have misled someone
reading this cold:
- "the first unattended run has not happened yet" -- it has, and it did the
wrong thing: skipped every source on the 20h floor and reported success.
- "no daily stories run is scheduled" -- one exists, but is disabled after the
scraping warning, which is a materially different situation. Stories now
depend on someone remembering, and every day nobody runs it is a day gone.
- "--abort 50 is opt-in and nothing uses it yet" -- gdl-cron.sh full passes it
and both manual runs used it. sweep deliberately does not, which is the
point of sweep.
- the diff baseline in "Verifying a run" was pre-2026-08-22 and would have
made a correct run look like it had lost files.
Adds the gap that matters most for re-enabling: the scheduled modes still use
the default 6-10s pacing while the hand runs after the warning used 12-20s, so
turning the timers back on as they stand would make the automation less
careful than the humans were.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The pause section predicted official_artms and 0ct0ber19 stories would expire
uncollected. They did not: both surfaces were fetched by hand the next day at
roughly double the configured caution, with 0 400s and 0 429s. 16 story media
and 26 posts/reels media, 85 files into the archive.
That confirms the 400s were the challenge state rather than a block -- once
the interstitial was dismissed, the same endpoints served normally.
It does not retire the warning, and the section says so. Two hand-paced runs
are not evidence the old cadence was safe. It also flags the gap that matters
for whenever the timers go back on: the scheduled "full" mode still runs at
the default 6-10s pacing, so the automation would be less careful than the
manual runs that followed a warning.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Instagram flagged the account on 2026-08-21 -- not a suspension, an
interstitial at /accounts/scraping_warning/ saying it suspects automated
behaviour. Dismissed; account healthy. All three timers are disabled.
The sequence is worth keeping because each step misled differently. The daily
timer fired and exited 0 having fetched nothing: yesterday's manual run was
19h earlier, under the 20h floor, so a silent no-op on the one surface that
cannot be backfilled reported as success. Chrome was separately OOM-killed, so
no browser was running to notice. The recovery run then drew 400s from the
stories endpoint on all six profiles.
Cookies were exported and checked before assuming a block -- sessionid
well-formed and unexpired -- which cost nothing and ruled out the local
explanation. Only then did the browser reveal the interstitial.
Records the four things to fix before re-enabling: the floor defeats the daily
job, a skipped stories run must be loud rather than exit 0, the daily cadence
is itself the most machine-like pattern we run, and chrome-devtools.service
has Restart=no.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The first unattended run is tomorrow morning and nobody will be watching it,
so this writes down how to tell whether it worked -- entirely from the
journal, local logs and our own viewer's API, without spending a single
Instagram request to check.
Includes the per-profile file counts as of today so tomorrow's numbers have
something to diff against, and says plainly that a stories run adding zero
files is a normal result rather than a failure: profiles often have no active
story, and someone checking for the first time would reasonably read 0 as
broken.
Repeats the one rule that matters more than any tuning: a CDN 429 ends the
session. The warning order last time was 429 -> 400 on the highlights endpoint
-> suspension, so the runbook gives the disable command rather than leaving it
to judgement in the moment.
Also records what changed today and why, since none of it is recoverable from
the diffs: the run, the --abort measurement, the public/private split, and the
automation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
mattellite's key is now in the NAS's authorized_keys for agentapi, so
gdl-sync.py publishes straight to the archive. That was the last manual step:
the 2026-08-20 run published locally and rsynced by hand with sshpass, which
no scheduled job could do. Verified end to end -- key auth, then a dry-run
rsync from the fetch host into the archive root.
Three timers, installed and enabled: stories daily, full monthly, sweep
quarterly. A templated gdl-sync@.service takes the mode as its instance name.
Settings that are load-bearing rather than decoration:
RandomizedDelaySec=45m a job firing at exactly 09:00 daily is obviously a
machine; list-timers now shows 09:36, not 09:00
Persistent=true catch up a run the host slept through -- cron just
skips, and a skipped stories run is content gone
TimeoutStartSec=infinity a sweep runs for hours at this pacing; the 90s
default would kill it mid-fetch
Two things caught while installing, both documented: month names are invalid
in OnCalendar's date field (Jan,Apr,Jul,Oct-07 is rejected -- use numerics,
and check with systemd-analyze calendar), and `systemctl --user` over
non-interactive ssh needs XDG_RUNTIME_DIR or it cannot find the bus.
Smoke-tested by starting the stories service: it spent zero Instagram requests
because the 20h floor skipped all six sources, which is exactly what a job
firing twice should do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
gdl-sync.py always needed a fistful of paths on the command line, and the
2026-08-20 run published to a local directory and then rsynced to the NAS by
hand -- fine interactively, impossible from cron. gdl-cron.sh takes one
argument (stories|full|sweep) and does the whole thing.
Staging is wiped every run deliberately: what we already hold is decided by
the skip-archive, never by what happens to be sitting in staging, so starting
empty is correct and keeps the publish to just the new files.
Documents the three cadences and what each is for -- notably that only `sweep`
notices carousels edited after we archived them, and that only `stories` loses
content if it is skipped, since stories cannot be backfilled.
Cron notes the two things that would silently break it: cron runs /bin/sh so
$RANDOM does not exist (use shuf), and `%` in a crontab line means newline.
systemd is offered as the better option because RandomizedDelaySec and
Persistent=true are built in -- a host asleep at 09:00 silently skips a cron
job, and a skipped stories run is content gone for good.
The publish step is still blocked on mattellite having no key on the NAS; that
is recorded under Outstanding as the one thing between this and unattended.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Nothing here is broken — these are decisions not made and cleanups not done,
written down before the session's context is lost.
The two that can actually cost something: no daily stories run is scheduled,
and stories are the one surface that cannot be backfilled; and the `seeded`
flags in the state file were reconstructed by hand from a log rather than
derived from the archive DB, so losing artms.db without also clearing the
state file would leave those sources permanently unseeded and re-download
everything.
Also corrects "Scanner work (not done yet)", which shipped in 53b1f80 —
the three .json shapes are told apart structurally in gallery-dl-sidecar.ts
and isReel comes from the sidecar's type. That file lives on main: it parses
archives at display time and is viewer code, not tooling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Putting it at the top of README.md worked, but it diverged a shared file: a
probe showed main editing the adjacent line conflicts on every merge. The
warning survived the conflict, so nothing was ever silently lost, but a file
that only exists on this branch has no such cost at all.
TOOLING.md carries the warning plus what CLAUDE.md would have said if it could
— the commands, the remote policy, and the two guards, which are local and
unversioned and so are absent from every fresh clone.
Shared files are byte-identical to main again: CLAUDE.md, README.md and
package.json. The divergence is now only files main has never had.
Trade-off worth knowing: gitea renders README.md on the branch page and does
not render TOOLING.md, so this warning is one click less visible than it was.
The pre-push hook, not the documentation, remains the guard that actually
stops a mistake.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
ergosteurandClaude Opus 5