docs: document the manual procedures properly, pacing included

The hand-run example had no pacing flags, so following it ran at gdl-sync.py's
defaults -- 6-10s and 1M -- which since yesterday is roughly half the caution
the wrapper applies. A documented manual procedure that is less careful than
the automation is backwards, and that is what this file said until now.

Adds the parts that were only ever in a transcript:

- how to detach a long run, and why `pkill -f <pattern>` over ssh kills your
  own shell instead of the sync (the pattern matches the ssh command line --
  this happened twice yesterday)
- that nothing reaches the archive until a run finishes, so killing one midway
  is safe
- a "when a run fails" runbook in cost order: read the log, check the cookies,
  then look at the browser. The first two are free.

The two results that look like failures and are not: rsync exit 23 is the
chown to rslsync failing because the ssh user is not root, and "No results"
just means a profile has no active story.

The cookie check is the one worth having written down -- exporting them and
looking at the shape of sessionid separates "Chrome decryption broke locally"
from "the account is in trouble" without spending a single request, and it is
what ruled out the wrong explanation yesterday.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
2026-08-22 11:34:23 -04:00
co-authored by Claude Opus 5
parent 7c7ce70ff9
commit b2eae177b6
+103
View File
@@ -118,12 +118,47 @@ PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
--publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \ --publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
--archive-db ~/gdl/artms.db \ --archive-db ~/gdl/artms.db \
--urls-file ~/gdl/artms_account_links.txt \ --urls-file ~/gdl/artms_account_links.txt \
--sleep-request 12 20 --sleep 5 10 --rate 500K \
--abort 50 --dry-run # swap for --execute when the plan looks right --abort 50 --dry-run # swap for --execute when the plan looks right
``` ```
**Do not omit the pacing flags.** `gdl-sync.py`'s own defaults are 6-10s / 3-6s
/ 1M, which is roughly half the caution the wrapper applies. A hand-run that
leaves them off is *less* careful than the automation — which is backwards, and
was true of this very example until 2026-08-22. Prefer `gdl-cron.sh`; it
carries them for you.
`PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not `PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not
on cron's PATH and not on a non-login shell's either. on cron's PATH and not on a non-login shell's either.
Useful variations:
```sh
--profile 0ct0ber19 --profile kimxxlip # instead of --urls-file
--only stories # one surface
--max-sources 4 # hard ceiling on what one run touches
--force # ignore --min-interval AND the probe
# cache; almost never what you want
```
### Long runs
A sync runs for minutes to hours at this pacing, so detach it rather than
holding an ssh session open:
```sh
cd ~/gdl && setsid nohup ./gdl-cron.sh profiles > /dev/null 2>&1 &
tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)
```
**Do not kill it with `pkill -f <pattern>` over ssh.** The pattern matches your
own `ssh` command line, so you kill your own shell and the sync survives — this
happened twice on 2026-08-22. Use `pkill -x chrome` style exact-name matches,
or kill the pid: `pgrep -f 'only posts,reels' | tail -1`.
Nothing reaches the archive until the run finishes: `gdl-sync.py` publishes
once, at the end, so a run killed midway leaves the archive untouched.
### The three modes ### The three modes
Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run — Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run —
@@ -342,6 +377,74 @@ Note the gap this leaves: **the scheduled `profiles` mode still uses the default
the timers, or the automation will be less careful than the hand runs that the timers, or the automation will be less careful than the hand runs that
followed a warning. followed a warning.
## When a run fails
Worked through on 2026-08-22. Do these **in order** — the first two cost no
Instagram requests, and the third costs one page view.
### 1. Read the error, not the exit code
```sh
L=~/gdl/logs/$(ls -1t ~/gdl/logs | head -1); grep -vE '^(profiles|surfaces|sources|pacing|staging|publish|read )' "$L" | head -40
```
Two results are **not** failures, despite how they look:
| looks like | actually |
|---|---|
| `rsync error: some files/attrs were not transferred (code 23)` and `done; 1 step(s) failed` | the `chown` to `rslsync` failing because the ssh user is not root. Data landed. Confirm by re-running the rsync with `--dry-run`: an empty file list means everything arrived |
| a source reporting `No results` | that profile simply has no active story / no highlights |
### 2. Check the cookies before concluding you are blocked
Free, and it separates a local fault from a server-side one:
```sh
PATH=$HOME/.local/bin:$PATH gallery-dl \
--cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools" \
--cookies-export /tmp/ck.txt
grep instagram /tmp/ck.txt | awk '{print $6, length($7)}' # names + value lengths
rm -f /tmp/ck.txt
```
A healthy `sessionid` is ~77 chars, printable ASCII, colon-delimited, and
unexpired; `ds_user_id` should be an 11-digit number. Garbage or non-printable
values mean Chrome's cookie decryption failed locally — not that the account is
in trouble. **Never print the values into a transcript or a bug report.**
### 3. Look at the account in the browser
The session lives in a dedicated Chrome on `mattellite`, on display `:1` with
CDP on 9222, run by a systemd unit:
```sh
export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user status chrome-devtools.service
systemctl --user start chrome-devtools.service # it has Restart=no
```
Then VNC to `:1` and open `instagram.com`. What you are looking for:
- **`/accounts/scraping_warning/`** — *"We suspect automated behaviour on your
account"*. This is what `400 Bad Request` on `/api/v1/feed/reels_media/`
actually means; the session is in a challenge state, not banned. Dismissing
it in the browser restores API access immediately, verified 2026-08-22.
- a checkpoint or login page — the session is gone; re-log in the browser.
- a normal feed — the fault is elsewhere.
**Do not dismiss a warning and immediately resume.** The dismissal fixes the
symptom. The behaviour that caused it is the thing to change.
### 4. Stop, if it was a 429 or a repeated 400
```sh
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
```
The documented escalation is CDN 429 → 400 on a stories/highlights endpoint →
suspension. It has now run twice, and both times the 400 was the last warning
before something worse.
## What changed on 2026-08-20 ## What changed on 2026-08-20
One session, three separate pieces of work. Recorded because the reasons are One session, three separate pieces of work. Recorded because the reasons are