docs: document the manual procedures properly, pacing included
The hand-run example had no pacing flags, so following it ran at gdl-sync.py's defaults -- 6-10s and 1M -- which since yesterday is roughly half the caution the wrapper applies. A documented manual procedure that is less careful than the automation is backwards, and that is what this file said until now. Adds the parts that were only ever in a transcript: - how to detach a long run, and why `pkill -f <pattern>` over ssh kills your own shell instead of the sync (the pattern matches the ssh command line -- this happened twice yesterday) - that nothing reaches the archive until a run finishes, so killing one midway is safe - a "when a run fails" runbook in cost order: read the log, check the cookies, then look at the browser. The first two are free. The two results that look like failures and are not: rsync exit 23 is the chown to rslsync failing because the ssh user is not root, and "No results" just means a profile has no active story. The cookie check is the one worth having written down -- exporting them and looking at the shape of sessionid separates "Chrome decryption broke locally" from "the account is in trouble" without spending a single request, and it is what ruled out the wrong explanation yesterday. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
+103
@@ -118,12 +118,47 @@ PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
|
|||||||
--publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
|
--publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
|
||||||
--archive-db ~/gdl/artms.db \
|
--archive-db ~/gdl/artms.db \
|
||||||
--urls-file ~/gdl/artms_account_links.txt \
|
--urls-file ~/gdl/artms_account_links.txt \
|
||||||
|
--sleep-request 12 20 --sleep 5 10 --rate 500K \
|
||||||
--abort 50 --dry-run # swap for --execute when the plan looks right
|
--abort 50 --dry-run # swap for --execute when the plan looks right
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Do not omit the pacing flags.** `gdl-sync.py`'s own defaults are 6-10s / 3-6s
|
||||||
|
/ 1M, which is roughly half the caution the wrapper applies. A hand-run that
|
||||||
|
leaves them off is *less* careful than the automation — which is backwards, and
|
||||||
|
was true of this very example until 2026-08-22. Prefer `gdl-cron.sh`; it
|
||||||
|
carries them for you.
|
||||||
|
|
||||||
`PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not
|
`PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not
|
||||||
on cron's PATH and not on a non-login shell's either.
|
on cron's PATH and not on a non-login shell's either.
|
||||||
|
|
||||||
|
Useful variations:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
--profile 0ct0ber19 --profile kimxxlip # instead of --urls-file
|
||||||
|
--only stories # one surface
|
||||||
|
--max-sources 4 # hard ceiling on what one run touches
|
||||||
|
--force # ignore --min-interval AND the probe
|
||||||
|
# cache; almost never what you want
|
||||||
|
```
|
||||||
|
|
||||||
|
### Long runs
|
||||||
|
|
||||||
|
A sync runs for minutes to hours at this pacing, so detach it rather than
|
||||||
|
holding an ssh session open:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd ~/gdl && setsid nohup ./gdl-cron.sh profiles > /dev/null 2>&1 &
|
||||||
|
tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Do not kill it with `pkill -f <pattern>` over ssh.** The pattern matches your
|
||||||
|
own `ssh` command line, so you kill your own shell and the sync survives — this
|
||||||
|
happened twice on 2026-08-22. Use `pkill -x chrome` style exact-name matches,
|
||||||
|
or kill the pid: `pgrep -f 'only posts,reels' | tail -1`.
|
||||||
|
|
||||||
|
Nothing reaches the archive until the run finishes: `gdl-sync.py` publishes
|
||||||
|
once, at the end, so a run killed midway leaves the archive untouched.
|
||||||
|
|
||||||
### The three modes
|
### The three modes
|
||||||
|
|
||||||
Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run —
|
Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run —
|
||||||
@@ -342,6 +377,74 @@ Note the gap this leaves: **the scheduled `profiles` mode still uses the default
|
|||||||
the timers, or the automation will be less careful than the hand runs that
|
the timers, or the automation will be less careful than the hand runs that
|
||||||
followed a warning.
|
followed a warning.
|
||||||
|
|
||||||
|
## When a run fails
|
||||||
|
|
||||||
|
Worked through on 2026-08-22. Do these **in order** — the first two cost no
|
||||||
|
Instagram requests, and the third costs one page view.
|
||||||
|
|
||||||
|
### 1. Read the error, not the exit code
|
||||||
|
|
||||||
|
```sh
|
||||||
|
L=~/gdl/logs/$(ls -1t ~/gdl/logs | head -1); grep -vE '^(profiles|surfaces|sources|pacing|staging|publish|read )' "$L" | head -40
|
||||||
|
```
|
||||||
|
|
||||||
|
Two results are **not** failures, despite how they look:
|
||||||
|
|
||||||
|
| looks like | actually |
|
||||||
|
|---|---|
|
||||||
|
| `rsync error: some files/attrs were not transferred (code 23)` and `done; 1 step(s) failed` | the `chown` to `rslsync` failing because the ssh user is not root. Data landed. Confirm by re-running the rsync with `--dry-run`: an empty file list means everything arrived |
|
||||||
|
| a source reporting `No results` | that profile simply has no active story / no highlights |
|
||||||
|
|
||||||
|
### 2. Check the cookies before concluding you are blocked
|
||||||
|
|
||||||
|
Free, and it separates a local fault from a server-side one:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
PATH=$HOME/.local/bin:$PATH gallery-dl \
|
||||||
|
--cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools" \
|
||||||
|
--cookies-export /tmp/ck.txt
|
||||||
|
grep instagram /tmp/ck.txt | awk '{print $6, length($7)}' # names + value lengths
|
||||||
|
rm -f /tmp/ck.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
A healthy `sessionid` is ~77 chars, printable ASCII, colon-delimited, and
|
||||||
|
unexpired; `ds_user_id` should be an 11-digit number. Garbage or non-printable
|
||||||
|
values mean Chrome's cookie decryption failed locally — not that the account is
|
||||||
|
in trouble. **Never print the values into a transcript or a bug report.**
|
||||||
|
|
||||||
|
### 3. Look at the account in the browser
|
||||||
|
|
||||||
|
The session lives in a dedicated Chrome on `mattellite`, on display `:1` with
|
||||||
|
CDP on 9222, run by a systemd unit:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export XDG_RUNTIME_DIR=/run/user/$(id -u)
|
||||||
|
systemctl --user status chrome-devtools.service
|
||||||
|
systemctl --user start chrome-devtools.service # it has Restart=no
|
||||||
|
```
|
||||||
|
|
||||||
|
Then VNC to `:1` and open `instagram.com`. What you are looking for:
|
||||||
|
|
||||||
|
- **`/accounts/scraping_warning/`** — *"We suspect automated behaviour on your
|
||||||
|
account"*. This is what `400 Bad Request` on `/api/v1/feed/reels_media/`
|
||||||
|
actually means; the session is in a challenge state, not banned. Dismissing
|
||||||
|
it in the browser restores API access immediately, verified 2026-08-22.
|
||||||
|
- a checkpoint or login page — the session is gone; re-log in the browser.
|
||||||
|
- a normal feed — the fault is elsewhere.
|
||||||
|
|
||||||
|
**Do not dismiss a warning and immediately resume.** The dismissal fixes the
|
||||||
|
symptom. The behaviour that caused it is the thing to change.
|
||||||
|
|
||||||
|
### 4. Stop, if it was a 429 or a repeated 400
|
||||||
|
|
||||||
|
```sh
|
||||||
|
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
The documented escalation is CDN 429 → 400 on a stories/highlights endpoint →
|
||||||
|
suspension. It has now run twice, and both times the 400 was the last warning
|
||||||
|
before something worse.
|
||||||
|
|
||||||
## What changed on 2026-08-20
|
## What changed on 2026-08-20
|
||||||
|
|
||||||
One session, three separate pieces of work. Recorded because the reasons are
|
One session, three separate pieces of work. Recorded because the reasons are
|
||||||
|
|||||||
Reference in New Issue
Block a user