Compare commits
1
Commits
7638fa4a9a
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
  ergosteurandClaude Sonnet 5
|
600949e475 |
-751
@@ -1,751 +0,0 @@
|
||||
# Tooling branch
|
||||
|
||||
> [!CAUTION]
|
||||
> ## This branch must not be pushed to GitHub
|
||||
>
|
||||
> `tooling` is the only branch that still contains the archive-fetching
|
||||
> scripts and their docs, and those name things `main` was rewritten to
|
||||
> remove:
|
||||
>
|
||||
> - the fetch host's **public IP** (`docs/gallery-dl.md`)
|
||||
> - the browser profile the session cookie is read from
|
||||
> - the NAS archive path
|
||||
> - the **list of Instagram accounts being archived**
|
||||
>
|
||||
> On 2026-08-20 `main`'s entire history was rewritten with `git filter-repo`,
|
||||
> the GitHub repo was deleted and recreated, and 22 container images were
|
||||
> pruned from ghcr — all to get exactly this material out of public view.
|
||||
> **One push of this branch to GitHub undoes all of it.**
|
||||
>
|
||||
> A second cleanup would be harder than the first: after a force-push the old
|
||||
> commits stayed reachable by raw SHA, and only deleting the repository
|
||||
> outright removed them.
|
||||
|
||||
## Quick reference — getting new content onto the site
|
||||
|
||||
Timers are **disabled**, so this is manual today. Everything runs on
|
||||
`mattellite`; it fetches, publishes to the NAS itself, and the site picks it up
|
||||
with no deploy or restart.
|
||||
|
||||
```sh
|
||||
ssh mattellite
|
||||
|
||||
~/gdl/gdl-cron.sh stories # ~1 min. DO THIS OFTEN - stories die in 24h
|
||||
~/gdl/gdl-cron.sh profiles # ~5 min. posts, reels, highlights, stories
|
||||
```
|
||||
|
||||
`profiles` covers everything, so it is the one to run if you only run one — but
|
||||
it is no substitute for `stories`, because a story posted and expired between
|
||||
two `profiles` runs is simply gone. Both are safe to run back-to-back; the
|
||||
`--min-interval` floor makes a repeat a no-op rather than a re-fetch.
|
||||
|
||||
Then check <https://instaarchive.ergosteur.com> — new posts appear without a
|
||||
redeploy. The archive index re-reads a directory when its mtime changes.
|
||||
|
||||
**Reading the output.** These two look alarming and are fine:
|
||||
|
||||
- `rsync error: ... (code 23)` / `done; 1 step(s) failed` — the `chown` to
|
||||
`rslsync` failing because the ssh user is not root. Files landed.
|
||||
- `No results` for a profile — it has no active story right now.
|
||||
|
||||
**These mean stop**, and are covered in "When a run fails":
|
||||
|
||||
- `429` from `scontent-*.cdninstagram.com`
|
||||
- `400 Bad Request` on `/api/v1/feed/reels_media/` — the account is behind a
|
||||
scraping-warning interstitial; check the browser before anything else.
|
||||
|
||||
Long runs: `setsid nohup ~/gdl/gdl-cron.sh profiles >/dev/null 2>&1 &` and
|
||||
`tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`. Nothing reaches the
|
||||
archive until a run finishes, so killing one midway is safe.
|
||||
|
||||
## Safety net: a global gallery-dl config
|
||||
|
||||
`~/.config/gallery-dl/config.json` on `mattellite` exists so that a **plain
|
||||
`gallery-dl <url>` typed by hand** — for a quick manual check, outside
|
||||
`gdl-sync.py` entirely — still gets the hand-paced caution settings instead of
|
||||
gallery-dl's own faster defaults. It is loaded automatically; nothing needs to
|
||||
reference it. `gdl-sync.py`'s own `--sleep-request`/`--sleep`/`--rate` flags
|
||||
still override it as normal — this is only a floor for when nobody passed any.
|
||||
|
||||
```json
|
||||
{
|
||||
"extractor": {
|
||||
"instagram": {
|
||||
"api": "rest",
|
||||
"cookies": ["chrome", "/home/matt/.config/google-chrome-devtools"],
|
||||
"sleep-request": [12.0, 20.0],
|
||||
"sleep": [5.0, 10.0],
|
||||
"sleep-429": 120.0,
|
||||
"retries": 8,
|
||||
"videos": true
|
||||
}
|
||||
},
|
||||
"downloader": {
|
||||
"http": {
|
||||
"rate": "500K",
|
||||
"retries": 8
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Every value here mirrors `gdl-cron.sh`'s own hand-paced defaults (see its
|
||||
`SLEEP_REQUEST`/`SLEEP`/`RATE` comments) and `build_config()` in
|
||||
`gdl-sync.py` — `api: rest` matters most: the graphql backend issues one
|
||||
request PER POST for every video and carousel, the pattern that got this
|
||||
account banned once already. `cookies` here is the config-file equivalent of
|
||||
`--cookies-from-browser`, so a bare `gallery-dl <url>` is already
|
||||
authenticated as the archive account, not anonymous.
|
||||
|
||||
Verified with `gallery_dl.config.load()` + `config.get(...)` (zero live
|
||||
requests) — every value above loads correctly with no `--config` flag passed.
|
||||
|
||||
This file is **not tracked in the repo** — like `artms.db` and the state
|
||||
files, it is host-local runtime config, and it embeds the same real
|
||||
Chrome-profile path already documented above. Recreate it by hand (or from
|
||||
this section) after a fresh `mattellite` setup.
|
||||
|
||||
## Remotes
|
||||
|
||||
| remote | what goes there |
|
||||
|---|---|
|
||||
| `origin` → gitea | everything: `main`, `tooling`, tags, backups |
|
||||
| `github` | **`main` and the current release tag only** — it exists to run the CI/CD image build |
|
||||
|
||||
The other 22 release tags stay on gitea. Pushing them all to GitHub triggers
|
||||
one container build per tag, because each tag carries its own workflow file.
|
||||
|
||||
## Guards — recreate these after a fresh clone
|
||||
|
||||
Neither guard is versioned, so a new clone has **no protection at all**:
|
||||
|
||||
```sh
|
||||
git config remote.github.push refs/heads/main:refs/heads/main
|
||||
|
||||
cat > .git/hooks/pre-push <<'HOOK'
|
||||
#!/bin/sh
|
||||
remote_url="$2"
|
||||
case "$remote_url" in *github.com*) ;; *) exit 0 ;; esac
|
||||
while read -r _ _ remote_ref _; do
|
||||
[ -z "$remote_ref" ] && continue
|
||||
case "$remote_ref" in
|
||||
refs/heads/main|refs/tags/*) ;;
|
||||
*) echo "pre-push: refusing to push '$remote_ref' to GitHub." >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
exit 0
|
||||
HOOK
|
||||
chmod +x .git/hooks/pre-push
|
||||
```
|
||||
|
||||
Decide on the **remote** ref, not the local one: a delete push sends
|
||||
`(delete)` as the local ref, and an earlier version of this hook rejected
|
||||
every deletion because of it.
|
||||
|
||||
## What lives here
|
||||
|
||||
| path | what it is |
|
||||
|---|---|
|
||||
| `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
|
||||
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
|
||||
| `scripts/systemd/` | the timers actually installed on the fetch host |
|
||||
| `scripts/test_gdl_sync.py` | its tests |
|
||||
| `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
|
||||
| `docs/gallery-dl.md` | the measurements behind every option in the fetcher — **read before changing pacing** |
|
||||
| `docs/jdownloader.md` | the older JD2 flow |
|
||||
| `docs/artms-instagram-accounts.txt` | the profile list passed to `--urls-file` |
|
||||
|
||||
## Commands
|
||||
|
||||
```sh
|
||||
# fetch: always --dry-run first; it prints the plan and the publish step
|
||||
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \
|
||||
--staging <dir> --publish <user>@<nas>:<archives> \
|
||||
--archive-db <db> --urls-file artms_account_links.txt --abort 50 --dry-run
|
||||
|
||||
# crawljobs (no npm script — package.json is kept identical to main)
|
||||
npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run
|
||||
```
|
||||
|
||||
Run the fetcher from the host whose public IP matches the browser the cookie
|
||||
came from. `--abort 50` is the routine setting; omit it for a full sweep that
|
||||
also catches edited carousels.
|
||||
|
||||
## Why there is no CLAUDE.md entry for any of this
|
||||
|
||||
`CLAUDE.md`, `README.md` and `package.json` are kept **byte-identical** to
|
||||
`main` so that merging `main` into `tooling` never conflicts. The earlier
|
||||
attempt put tooling notes in `CLAUDE.md` and a `jd2` script in `package.json`;
|
||||
because `main` had *deleted* those lines, every merge re-applied the deletion.
|
||||
Keep branch-specific documentation in this file, which `main` does not have.
|
||||
|
||||
## Running it by hand
|
||||
|
||||
Everything happens on **`mattellite`** — the fetch host whose public IP matches
|
||||
the browser the cookie came from. Running it anywhere else is what
|
||||
session-hijack detection looks for.
|
||||
|
||||
```sh
|
||||
ssh mattellite
|
||||
~/gdl/gdl-cron.sh profiles # or: stories | full-sweep
|
||||
```
|
||||
|
||||
That is the whole thing: it wipes staging, fetches, and publishes straight to
|
||||
the NAS. To drive `gdl-sync.py` directly instead — always `--dry-run` first,
|
||||
which prints the plan and the exact rsync that would touch the archive:
|
||||
|
||||
```sh
|
||||
cd ~/gdl
|
||||
PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
|
||||
--index https://instaarchive.ergosteur.com \
|
||||
--staging ~/gdl/staging-manual \
|
||||
--publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
|
||||
--archive-db ~/gdl/artms.db \
|
||||
--urls-file ~/gdl/artms_account_links.txt \
|
||||
--sleep-request 12 20 --sleep 5 10 --rate 500K \
|
||||
--abort 50 --dry-run # swap for --execute when the plan looks right
|
||||
```
|
||||
|
||||
**Do not omit the pacing flags.** `gdl-sync.py`'s own defaults are 6-10s / 3-6s
|
||||
/ 1M, which is roughly half the caution the wrapper applies. A hand-run that
|
||||
leaves them off is *less* careful than the automation — which is backwards, and
|
||||
was true of this very example until 2026-08-22. Prefer `gdl-cron.sh`; it
|
||||
carries them for you.
|
||||
|
||||
`PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not
|
||||
on cron's PATH and not on a non-login shell's either.
|
||||
|
||||
Useful variations:
|
||||
|
||||
```sh
|
||||
--profile 0ct0ber19 --profile kimxxlip # instead of --urls-file
|
||||
--only stories # one surface
|
||||
--max-sources 4 # hard ceiling on what one run touches
|
||||
--force # ignore --min-interval AND the probe
|
||||
# cache; almost never what you want
|
||||
```
|
||||
|
||||
### Long runs
|
||||
|
||||
A sync runs for minutes to hours at this pacing, so detach it rather than
|
||||
holding an ssh session open:
|
||||
|
||||
```sh
|
||||
cd ~/gdl && setsid nohup ./gdl-cron.sh profiles > /dev/null 2>&1 &
|
||||
tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)
|
||||
```
|
||||
|
||||
**Do not kill it with `pkill -f <pattern>` over ssh.** The pattern matches your
|
||||
own `ssh` command line, so you kill your own shell and the sync survives — this
|
||||
happened twice on 2026-08-22. Use `pkill -x chrome` style exact-name matches,
|
||||
or kill the pid: `pgrep -f 'only posts,reels' | tail -1`.
|
||||
|
||||
Nothing reaches the archive until the run finishes: `gdl-sync.py` publishes
|
||||
once, at the end, so a run killed midway leaves the archive untouched.
|
||||
|
||||
### The three modes
|
||||
|
||||
Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run —
|
||||
and `sweep` did not convey that it is the exhaustive one. The wrapper rejects
|
||||
the old names with a pointer rather than a bare error.
|
||||
|
||||
| mode | cadence | cost | why |
|
||||
|---|---|---|---|
|
||||
| `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped |
|
||||
| `profiles` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating a profile once it reaches content already held. Catches everything **new** |
|
||||
| `full-sweep` | rarely, by hand | **~420 requests** | no abort; walks every profile to the end. The only run that notices posts **edited** after we archived them (test case 15) |
|
||||
|
||||
The skip-archive means an infrequent `profiles` run costs barely more than a frequent
|
||||
one — it only fetches what is new. Frequency buys freshness, not completeness,
|
||||
except for stories.
|
||||
|
||||
**Pacing is deliberately slower than `gdl-sync.py`'s own defaults.** All three
|
||||
modes run at `--sleep-request 12 20 --sleep 5 10 --rate 500K`, against defaults
|
||||
of 6-10 / 3-6 / 1M. These are the values the 2026-08-22 runs used by hand after
|
||||
the scraping warning, and they produced 0 400s and 0 429s. An archive sync has
|
||||
no deadline: being slow is free, being restricted is not. Override with
|
||||
`GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE` — to raise them, not lower them.
|
||||
|
||||
`stories` also runs at `--min-interval 8` rather than the 20h default, because
|
||||
at 20h the daily timer silently did nothing whenever a manual run had happened
|
||||
the previous afternoon. The floor exists to stop an *aborted restart*
|
||||
re-enumerating profiles, which is a minutes-to-hours concern; a stories fetch
|
||||
is one request per profile, so 8h permits at worst about twelve requests in a
|
||||
day instead of six. **A stories run that skips every source now exits 75 and
|
||||
prints a warning** rather than reporting success.
|
||||
|
||||
## Scheduling — installed on `mattellite`
|
||||
|
||||
systemd **user** timers, running as `matt`, with lingering enabled so they fire
|
||||
without a login session:
|
||||
|
||||
```sh
|
||||
loginctl show-user matt --property=Linger # Linger=yes
|
||||
systemctl --user list-timers 'gdl-sync@*'
|
||||
```
|
||||
|
||||
| unit | schedule | next fire (as installed) |
|
||||
|---|---|---|
|
||||
| `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working |
|
||||
| `gdl-sync@profiles.timer` | 3rd of each month, 04:00 | 04:37:44 |
|
||||
| `gdl-sync@full-sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
|
||||
|
||||
Unit files are version-controlled in `scripts/systemd/` and installed to
|
||||
`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
|
||||
the mode as its instance name and runs `gdl-cron.sh %i`.
|
||||
|
||||
Three settings are load-bearing:
|
||||
|
||||
- **`RandomizedDelaySec=45m`** — a job firing at exactly 09:00 daily is
|
||||
obviously a machine, and the entire safety model is about not looking like
|
||||
one. This is why the table above shows 09:36 rather than 09:00.
|
||||
- **`Persistent=true`** — catch up a run missed because the host was off.
|
||||
cron silently skips, and a skipped `stories` run is content gone for good.
|
||||
- **`TimeoutStartSec=infinity`** — a sweep can run for hours at this pacing.
|
||||
The default 90s would kill it mid-fetch.
|
||||
|
||||
Operating them:
|
||||
|
||||
```sh
|
||||
export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed over non-interactive ssh
|
||||
systemctl --user start gdl-sync@stories.service # run one now
|
||||
systemctl --user status gdl-sync@profiles.timer
|
||||
journalctl --user -u 'gdl-sync@*' -n 50
|
||||
systemctl --user disable --now gdl-sync@full-sweep.timer # stop one
|
||||
```
|
||||
|
||||
`systemctl --user` fails with "Failed to connect to bus" over ssh unless
|
||||
`XDG_RUNTIME_DIR` is set. Note also that **month names are not valid in
|
||||
`OnCalendar`'s date field** — `Jan,Apr,Jul,Oct-07` is rejected outright, hence
|
||||
`*-01,04,07,10-07`. Check any change with `systemd-analyze calendar '<expr>'`
|
||||
before installing it.
|
||||
|
||||
### cron, if you ever prefer it
|
||||
|
||||
```cron
|
||||
17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
|
||||
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh profiles
|
||||
11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full-sweep
|
||||
```
|
||||
|
||||
cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a
|
||||
crontab line means newline unless escaped, so avoid it entirely. cron has no
|
||||
equivalent of `Persistent=true`.
|
||||
|
||||
## Verifying a run
|
||||
|
||||
The first unattended run is **2026-08-21, around 09:36** (09:00 plus the
|
||||
randomised delay). Nothing below costs an Instagram request — every check is
|
||||
against the journal, local logs, or our own viewer's API.
|
||||
|
||||
```sh
|
||||
# 1. did it run, and did it exit 0?
|
||||
ssh mattellite
|
||||
export XDG_RUNTIME_DIR=/run/user/$(id -u)
|
||||
systemctl --user list-timers 'gdl-sync@*' # LAST/PASSED columns
|
||||
journalctl --user -u 'gdl-sync@stories.service' --since yesterday --no-pager
|
||||
|
||||
# 2. what did it actually fetch? (sidecars vastly outnumber media -- count media)
|
||||
ls -1t ~/gdl/logs | head -3
|
||||
grep -E '^(==>| FAILED|done;)' ~/gdl/logs/stories-*.log | tail -20
|
||||
grep -ic 429 ~/gdl/logs/stories-*.log # MUST be 0 -- see below
|
||||
```
|
||||
|
||||
**A `429` from `scontent-*.cdninstagram.com` ends the session, it is not a
|
||||
pacing knob to tune.** The warning order last time was CDN 429 → `400` on the
|
||||
highlights endpoint → suspension. If a run logs one, disable the timers and
|
||||
stop for the day:
|
||||
|
||||
```sh
|
||||
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
|
||||
```
|
||||
|
||||
Then confirm the archive actually grew, from the workstation:
|
||||
|
||||
```sh
|
||||
# 3. did the publish land? compare against yesterday's counts
|
||||
for u in 0ct0ber19 kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
|
||||
n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
|
||||
printf '%-18s %s\n' "$u" "$n"
|
||||
done
|
||||
```
|
||||
|
||||
Counts after the 2026-08-22 runs, to diff against:
|
||||
|
||||
| profile | files |
|
||||
|---|---:|
|
||||
| 0ct0ber19 | 3160 |
|
||||
| official_artms | 6760 |
|
||||
| cher_ryppo | 3056 |
|
||||
| kimxxlip | 3021 |
|
||||
| zindoriyam | 2256 |
|
||||
| withaseul | 1760 |
|
||||
|
||||
A stories-only run adds few files and often **none** — profiles frequently have
|
||||
no active story. "0 new" is a normal result, not a failure. `fileCount` in
|
||||
`/api/archives` is stale by design; use the per-profile `/files` listing.
|
||||
|
||||
## 2026-08-21 — scraping warning, automation stopped
|
||||
|
||||
**Instagram flagged the account.** Not suspended: an interstitial at
|
||||
`/accounts/scraping_warning/` reading *"We suspect automated behaviour on your
|
||||
account"*. It was dismissed in the browser and the account is healthy — feed
|
||||
loads, still signed in. **All three timers are disabled.** Do not re-enable
|
||||
them without deciding the cadence question below.
|
||||
|
||||
How it unfolded, because each step misled in a different way:
|
||||
|
||||
1. **09:13** the daily timer fired, exited 0 in one second, logged
|
||||
`done; 0 step(s) failed` — and fetched nothing. Yesterday's manual run was
|
||||
18.9–19.2h earlier, just under the `--min-interval 20` floor, so all six
|
||||
sources were skipped. **A silent no-op on the one surface that cannot be
|
||||
backfilled, reported as success.**
|
||||
2. **20:28** `chrome-devtools.service` was OOM-killed (5.1 GB peak, ~1w3d CPU).
|
||||
Unrelated to the above, and it does not break fetching — gallery-dl reads
|
||||
the cookie *file*, not a live browser — but it meant no browser was running
|
||||
to notice anything was wrong.
|
||||
3. **23:19** a manual recovery run passed the floor (33h) and every source
|
||||
failed with `400 Bad Request` on
|
||||
`/api/v1/feed/reels_media/?reel_ids=…`. Six identical failures across six
|
||||
profiles is not a per-profile fault.
|
||||
4. Cookies were **exported and checked before assuming a block**:
|
||||
`sessionid` 77 chars, printable, colon-delimited, 360 days to expiry;
|
||||
`ds_user_id` present. Decryption was fine, so the fault was server-side.
|
||||
This check costs no Instagram requests and should always come first.
|
||||
5. The browser then showed the interstitial. The 400s were the challenge
|
||||
state, not a ban.
|
||||
|
||||
### What has to change before automation is re-enabled
|
||||
|
||||
- **The `--min-interval` floor silently defeats the daily job.** Any manual run
|
||||
in the preceding 20h makes the scheduled one a no-op. The floor exists to
|
||||
stop an *aborted restart* re-enumerating profiles — a minutes-to-hours
|
||||
concern — and a stories fetch is one request per profile. `stories` should
|
||||
use something like `--min-interval 8`, not 20.
|
||||
- **A skipped stories run must be loud.** `0 to sync, 6 skipped` currently
|
||||
exits 0 and looks identical to success. On this surface a skip is a real
|
||||
loss, and it should be visible in the journal without reading the log.
|
||||
- **Reconsider the daily cadence itself.** A job hitting story endpoints for
|
||||
six profiles every morning is the most machine-like thing here, randomised
|
||||
delay or not, and it is what was flagged. Every-few-days, or on-demand, may
|
||||
be the honest answer even though stories will be missed.
|
||||
- **`chrome-devtools.service` has `Restart=no`** and died silently for three
|
||||
hours. It needs `Restart=on-failure` and probably a `MemoryMax=`, or it will
|
||||
be dead the next time the cookie needs refreshing.
|
||||
|
||||
### 2026-08-22 — caught up by hand, cleanly
|
||||
|
||||
Both surfaces were fetched manually the next day, on the owner's call, at
|
||||
**roughly double the configured caution**: `--sleep-request 12 20`,
|
||||
`--sleep 5 10`, `--rate 500K`, versus the defaults of 6-10 / 3-6 / 1M.
|
||||
|
||||
| run | result |
|
||||
|---|---|
|
||||
| stories, 6 profiles | 16 media, +21 files, **0 400s, 0 429s** |
|
||||
| posts+reels, 12 sources, `--abort 50` | 26 media, +64 files, **0 400s, 0 429s** |
|
||||
|
||||
So the 400s really were the challenge state and nothing more: once the
|
||||
interstitial was dismissed in the browser, the same endpoints served normally.
|
||||
The stories that looked lost — `official_artms`, `0ct0ber19`, `cher_ryppo`,
|
||||
`kimxxlip`, `zindoriyam` — were all captured before expiry.
|
||||
|
||||
**This does not retire the warning.** Two hand-paced runs a day later are not
|
||||
evidence that the previous cadence was safe; they are evidence that the
|
||||
account still works. What actually changed the request cost is `--abort 50`:
|
||||
twelve sources across six profiles, `official_artms` included at 1829 posts
|
||||
and 781 reels, finished in minutes for a few dozen requests where the old
|
||||
behaviour would have spent ~400.
|
||||
|
||||
Note the gap this leaves: **the scheduled `profiles` mode still uses the default
|
||||
6-10s pacing**, not the 12-20s used here. Reconcile that before re-enabling
|
||||
the timers, or the automation will be less careful than the hand runs that
|
||||
followed a warning.
|
||||
|
||||
## When a run fails
|
||||
|
||||
Worked through on 2026-08-22. Do these **in order** — the first two cost no
|
||||
Instagram requests, and the third costs one page view.
|
||||
|
||||
### 1. Read the error, not the exit code
|
||||
|
||||
```sh
|
||||
L=~/gdl/logs/$(ls -1t ~/gdl/logs | head -1); grep -vE '^(profiles|surfaces|sources|pacing|staging|publish|read )' "$L" | head -40
|
||||
```
|
||||
|
||||
Two results are **not** failures, despite how they look:
|
||||
|
||||
| looks like | actually |
|
||||
|---|---|
|
||||
| `rsync error: some files/attrs were not transferred (code 23)` and `done; 1 step(s) failed` | the `chown` to `rslsync` failing because the ssh user is not root. Data landed. Confirm by re-running the rsync with `--dry-run`: an empty file list means everything arrived |
|
||||
| a source reporting `No results` | that profile simply has no active story / no highlights |
|
||||
|
||||
### 2. Check the cookies before concluding you are blocked
|
||||
|
||||
Free, and it separates a local fault from a server-side one:
|
||||
|
||||
```sh
|
||||
PATH=$HOME/.local/bin:$PATH gallery-dl \
|
||||
--cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools" \
|
||||
--cookies-export /tmp/ck.txt
|
||||
grep instagram /tmp/ck.txt | awk '{print $6, length($7)}' # names + value lengths
|
||||
rm -f /tmp/ck.txt
|
||||
```
|
||||
|
||||
A healthy `sessionid` is ~77 chars, printable ASCII, colon-delimited, and
|
||||
unexpired; `ds_user_id` should be an 11-digit number. Garbage or non-printable
|
||||
values mean Chrome's cookie decryption failed locally — not that the account is
|
||||
in trouble. **Never print the values into a transcript or a bug report.**
|
||||
|
||||
### 3. Look at the account in the browser
|
||||
|
||||
The session lives in a dedicated Chrome on `mattellite`, on display `:1` with
|
||||
CDP on 9222, run by a systemd unit:
|
||||
|
||||
```sh
|
||||
export XDG_RUNTIME_DIR=/run/user/$(id -u)
|
||||
systemctl --user status chrome-devtools.service
|
||||
systemctl --user start chrome-devtools.service # it has Restart=no
|
||||
```
|
||||
|
||||
Then VNC to `:1` and open `instagram.com`. What you are looking for:
|
||||
|
||||
- **`/accounts/scraping_warning/`** — *"We suspect automated behaviour on your
|
||||
account"*. This is what `400 Bad Request` on `/api/v1/feed/reels_media/`
|
||||
actually means; the session is in a challenge state, not banned. Dismissing
|
||||
it in the browser restores API access immediately, verified 2026-08-22.
|
||||
- a checkpoint or login page — the session is gone; re-log in the browser.
|
||||
- a normal feed — the fault is elsewhere.
|
||||
|
||||
**Do not dismiss a warning and immediately resume.** The dismissal fixes the
|
||||
symptom. The behaviour that caused it is the thing to change.
|
||||
|
||||
### 4. Stop, if it was a 429 or a repeated 400
|
||||
|
||||
```sh
|
||||
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
|
||||
```
|
||||
|
||||
The documented escalation is CDN 429 → 400 on a stories/highlights endpoint →
|
||||
suspension. It has now run twice, and both times the 400 was the last warning
|
||||
before something worse.
|
||||
|
||||
## Reels: the API is blocked, scrape by scrolling instead
|
||||
|
||||
As of 2026-08-26/27, `gallery-dl`'s dedicated reels extractor fails on every
|
||||
profile with `HTTP redirect to home page` — confirmed hours apart, with a
|
||||
freshly-warmed session and a correct `X-IG-WWW-Claim` header (that was the
|
||||
first suspect; ruled out by tracing the raw HTTP exchange). It is not a
|
||||
scraping-warning interstitial — the reels tab loads completely normally in a
|
||||
real, already-signed-in browser — so this is Meta blocking the specific
|
||||
`/api/v1/clips/user/` endpoint gallery-dl calls, not an account-health issue.
|
||||
Posts, highlights and stories are unaffected; only reels breaks.
|
||||
|
||||
`scripts/reels-scrape.py` works around it by never calling that endpoint:
|
||||
it drives the same signed-in Chrome via its loopback CDP port (`:9222`,
|
||||
already exposed for MCP automation), scrolls the reels tab like a person
|
||||
would, and scrapes `/reel/<code>/` links out of the rendered page. It only
|
||||
finds shortcodes — nothing is downloaded until the fetch step — and dedupes
|
||||
against the **whole archive**, not just the profile being scraped (see why
|
||||
below), so re-running it costs nothing for reels already held anywhere.
|
||||
|
||||
### The easy way: `reels-sync.sh`
|
||||
|
||||
```sh
|
||||
ssh mattellite
|
||||
~/gdl/reels-sync.sh zindoriyam
|
||||
# or a full URL: ~/gdl/reels-sync.sh https://www.instagram.com/zindoriyam/
|
||||
```
|
||||
|
||||
Runs both steps (scrape, then fetch + publish whatever's new) with the same
|
||||
hand-paced settings `gdl-cron.sh` uses, logs to `~/gdl/logs/reels-<profile>-*`,
|
||||
and exits 0 with "no new reels" printed when a profile is already caught up
|
||||
— `gdl-sync.py` never even gets invoked in that case. Override pacing the
|
||||
same way as `gdl-cron.sh` (`GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE`), plus
|
||||
`GDL_SCROLL_PAUSE` and `GDL_MAX_IDLE_ROUNDS` for the scrape step. Staging
|
||||
(`~/gdl/staging-reels-<profile>`) and the scraped URL list
|
||||
(`~/gdl/<profile>-reels.txt`) are wiped at the START of the next run, not
|
||||
after — left behind for inspection, same as `gdl-cron.sh`'s `staging-*`.
|
||||
|
||||
Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the
|
||||
page, 10 new ones fetched and published cleanly on the first run.
|
||||
|
||||
**Not wired into `gdl-cron.sh` or the timers.** Both scripts drive your
|
||||
actual browser session rather than firing a background request, and are
|
||||
slower by design (real scrolling, not an API call) — both good reasons not
|
||||
to run this unattended without deciding that deliberately. Today it is a
|
||||
per-profile, by-hand tool only.
|
||||
|
||||
### Where a shared reel lands
|
||||
|
||||
A reel found on one profile's `/reels/` page is not necessarily *owned* by
|
||||
that profile — reposts and collabs between tracked accounts show up there
|
||||
too. It always gets filed under its **true owner**, per Instagram's own
|
||||
metadata on the post, never under whichever profile's page you happened to
|
||||
scrape it from — `gdl-sync.py`'s directory template for a scraped reel is
|
||||
`{username}` filled in from that metadata, the same mechanism highlights
|
||||
already used for their own directory. So:
|
||||
|
||||
- Scrape order doesn't matter. Run `reels-sync.sh` on `0ct0ber19` or
|
||||
`zindoriyam` first, whichever — a reel they share lands in the same place
|
||||
either way, and running it on the other one afterward just sees that
|
||||
shortcode as already archived (dedup checks every profile, not only the
|
||||
one being scraped) and skips it.
|
||||
- It never gets duplicated into both accounts' directories, and it never
|
||||
gets misattributed to the profile you scraped instead of who actually
|
||||
posted it.
|
||||
|
||||
This is also literally why the whole-archive dedup fix above exists: a
|
||||
zindoriyam-scraped shortcode that turned out to belong to `0ct0ber19` was
|
||||
filed under `0ct0ber19/`, not `zindoriyam/` — dedup that only checked
|
||||
`zindoriyam`'s own directory would never have found it there.
|
||||
|
||||
### What it's doing, if you want to run the two steps separately
|
||||
|
||||
```sh
|
||||
PROFILE=someuser
|
||||
|
||||
# 1. Scrape by scrolling; dedupe against the archive; write new URLs to a file.
|
||||
# Needs gallery-dl's own pipx venv python -- that's where websocket-client
|
||||
# (the one extra dependency this needs) got injected.
|
||||
~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/reels-scrape.py \
|
||||
--profile "$PROFILE" \
|
||||
--index https://instaarchive.ergosteur.com \
|
||||
--out ~/gdl/"$PROFILE"-reels.txt
|
||||
|
||||
# 2. Fetch whatever's new -- dry run first, same as any other gdl-sync.py call.
|
||||
cd ~/gdl && PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
|
||||
--publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
|
||||
--staging ~/gdl/staging-reels-scraped \
|
||||
--sleep-request 12 20 --sleep 5 10 --rate 500K \
|
||||
--post-urls-file ~/gdl/"$PROFILE"-reels.txt --dry-run
|
||||
# ...then swap --dry-run for --execute once the plan looks right.
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `reels-scrape.py` fails fast if Chrome/CDP isn't up
|
||||
(`curl -s http://localhost:9222/json/version` to check first).
|
||||
- If it finds nothing new, `gdl-sync.py --post-urls-file` refuses to run
|
||||
("no usable URLs") rather than doing nothing quietly — expected when a
|
||||
profile is already caught up.
|
||||
- `--max-idle-rounds` (default 3) and `--scroll-pause MIN MAX` (default
|
||||
`2.0 3.5`) are worth raising for an unusually large or slow-loading reels
|
||||
tab; the default stops once 3 consecutive scrolls find nothing new.
|
||||
|
||||
### Why dedup checks the whole archive, not just the scraped profile
|
||||
|
||||
First version deduped only against the scraped profile's own directories.
|
||||
Re-running it on `zindoriyam` minutes after a successful fetch found "9 new"
|
||||
reels again — all reposts/collabs originally by `0ct0ber19` and
|
||||
`official_artms` (also tracked profiles). The *old*, broken direct-reels-tab
|
||||
fetch had left orphaned sidecar-only remnants (`.json`/`.txt`, no media) for
|
||||
them, misfiled under `zindoriyam - reels/` with the true owner's name baked
|
||||
into the filename stem — `index_existing()` correctly does not count a
|
||||
sidecar-only entry as "held", so they looked new. `gdl-sync.py` re-fetched
|
||||
them, filed them correctly under `{username}` (the post's *true* owner, per
|
||||
its own metadata) — where they already existed from that profile's own
|
||||
regular sync — and `rsync --ignore-existing` silently skipped every one, so
|
||||
nothing was lost or duplicated. But 9 Instagram requests were spent finding
|
||||
that out. Since a shortcode is globally unique, `reels-scrape.py` now checks
|
||||
every archived profile's listing, not just the one being scraped — all local
|
||||
requests to the viewer's own API, never to `instagram.com`, so checking all
|
||||
16 profiles costs nothing on the budget that actually matters. Confirmed
|
||||
fixed the same day: re-running against `zindoriyam` immediately afterward
|
||||
found "26 already archived, 0 new" and exited clean.
|
||||
|
||||
## What changed on 2026-08-20
|
||||
|
||||
One session, three separate pieces of work. Recorded because the reasons are
|
||||
not recoverable from the diffs.
|
||||
|
||||
**The sync run.** First incremental fetch in four days: 184 new media, 299
|
||||
files published, 0 failures, 0 CDN 429s. 20 story items, which are the part
|
||||
that could not have been recovered later. Cost about half what it would have,
|
||||
because the archive DB was already seeded and the state file was primed by hand
|
||||
so no probe passes ran.
|
||||
|
||||
**`--abort 50`.** The skip-archive suppresses *downloads*, which spends the
|
||||
CDN; it does nothing about the *listing pass*, which spends `instagram.com` and
|
||||
scales with how big a profile is rather than how much is new. Measured from
|
||||
sidecar write times mid-run: 3 new posts took ~100s each, the other 2272 were
|
||||
written in one second. Enumerating `cher_ryppo` fell from 2151 posts to 7.
|
||||
|
||||
**The repo split.** `main` is public and now carries none of the fetching
|
||||
tooling, no host details, and no real account names — its entire history was
|
||||
rewritten, the GitHub repo deleted and recreated to clear force-push residue,
|
||||
and 22 container images pruned from ghcr because the server bundle had been
|
||||
shipping source comments naming real accounts. This branch holds everything
|
||||
that was removed. See the caution at the top.
|
||||
|
||||
**Automation.** mattellite got a key on the NAS, closing the last manual step,
|
||||
and three systemd timers now run the sync unattended.
|
||||
|
||||
## Outstanding
|
||||
|
||||
State as of 2026-08-20, after the sync run and the repo split. Nothing here is
|
||||
broken; these are decisions not yet made and cleanups not yet done.
|
||||
|
||||
### Fetching
|
||||
|
||||
- **The timers are DISABLED** after the 2026-08-21 scraping warning; see that
|
||||
section. Their one unattended firing did the wrong thing — it skipped every
|
||||
source on the 20h floor and reported success — so re-enabling should wait on
|
||||
the four fixes listed there, not just on the account settling.
|
||||
- `mattellite`'s `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys` for
|
||||
`agentapi` (added 2026-08-20, alongside the workstation's existing key), so
|
||||
the fetch host publishes straight to the archive and no `sshpass` step is
|
||||
needed. **That key is what makes the timers work** — remove it and every
|
||||
scheduled run will fetch successfully and then fail at publish.
|
||||
|
||||
- **5.4 GB of stale staging on `mattellite`** (`~/gdl`, 44 GB free): `staging`
|
||||
and `out` at 2.2 GB each from 2026-08-17, `staging-0820` / `out-0820` at
|
||||
446 MB each, plus `staging-full` (78 MB) and `staging-stories` (66 MB) from
|
||||
2026-08-22. All of it was verified published, so all of it is safe to delete.
|
||||
Staging is wiped per-run by `gdl-cron.sh`, but the `out-*` publish targets and
|
||||
anything created by a direct `gdl-sync.py` call are never cleaned up.
|
||||
- **Stories currently depend on someone remembering.** The daily timer exists
|
||||
but is disabled, so the one surface that cannot be backfilled has no
|
||||
automation behind it. Every day nobody runs `gdl-cron.sh stories` is a day
|
||||
of stories gone. That is the central unresolved tension: the cadence that
|
||||
protects stories is also the most machine-like pattern here.
|
||||
- ~~The scheduled modes have not been reconciled with the pacing used by
|
||||
hand.~~ **Done 2026-08-22**: all modes now pass `--sleep-request 12 20
|
||||
--sleep 5 10 --rate 500K`, `stories` uses `--min-interval 8`, and a
|
||||
fully-skipped stories run exits 75 with a warning instead of looking like a
|
||||
success. The timers are still **disabled** — enabling them is a separate
|
||||
decision about cadence, not about pacing.
|
||||
- **`--abort 50` is opt-in.** `gdl-cron.sh profiles` passes it and the manual
|
||||
runs used it; `full-sweep` deliberately does not. It stops noticing **edited
|
||||
carousels** (test case 15), which only a full enumeration finds — which is
|
||||
what `full-sweep` is for. No cadence for it was ever agreed, and at ~420
|
||||
requests it is the riskiest thing on the schedule; prefer running it by hand.
|
||||
- **The `seeded` flags in `<db>.state.json` were hand-written**, reconstructed
|
||||
from the 2026-08-17 log rather than derived from the archive DB. They assert
|
||||
"the skip-archive already knows this source". If `artms.db` is ever rebuilt,
|
||||
moved or lost, **clear the state file too** — otherwise those sources will
|
||||
never re-seed and a fetch into empty staging re-downloads everything.
|
||||
- **`~/gdl/gdl-sync.py` on the fetch host is a copy, not a checkout.** It
|
||||
currently matches this branch (`96e5694e…`), but nothing keeps them in sync;
|
||||
`scp` it after any change and re-check the hash.
|
||||
- The 2026-08-20 run is split across two logs — `artms-run3.log` (12 sources,
|
||||
no abort) and `artms-run4.log` (12 sources, `--abort 50`) — because it was
|
||||
stopped midway to pick up the new flag.
|
||||
|
||||
### Repo and infrastructure
|
||||
|
||||
- **The `pre-rewrite-*` branches on gitea hold the unredacted history** — real
|
||||
account names, the fetch host's IP, and the tooling, as it was before the
|
||||
rewrite. They are deliberate backups. Decide whether they expire; the
|
||||
`pre-push` hook does cover them (it allows only `main` and tags to GitHub).
|
||||
- **The `pre-rewrite-full.bundle` backup is in a session scratchpad** and will
|
||||
be deleted with it. If a durable backup outside gitea is wanted, move it now.
|
||||
- **Only one container image exists.** 22 versions were pruned, so rolling back
|
||||
to an older release means checking out its tag from gitea and pushing that
|
||||
tag to GitHub to rebuild it — the old images are gone, not archived.
|
||||
- **CI warns that the Node 20 actions are deprecated.** `actions/checkout@v4`,
|
||||
`docker/login-action@v3`, `docker/metadata-action@v5` and
|
||||
`docker/build-push-action@v5` are being forced onto Node 24. They work today;
|
||||
bump when convenient.
|
||||
- **`review-fixes`** on gitea is a stale v1.3.0-era branch, never merged,
|
||||
published only because the whole local repo was pushed. Probably deletable.
|
||||
- GitHub Actions run history was lost when the repo was recreated. Cosmetic.
|
||||
@@ -1,6 +0,0 @@
|
||||
https://www.instagram.com/0ct0ber19/
|
||||
https://www.instagram.com/kimxxlip/
|
||||
https://www.instagram.com/withaseul/
|
||||
https://www.instagram.com/cher_ryppo/
|
||||
https://www.instagram.com/zindoriyam/
|
||||
https://www.instagram.com/official_artms/
|
||||
@@ -1,610 +0,0 @@
|
||||
# gallery-dl — a CLI replacement for JDownloader2
|
||||
|
||||
Status: **in production.** All six ARTMS profiles are synced with
|
||||
`scripts/gdl-sync.py`; JD2 is no longer used for them.
|
||||
|
||||
Everything below was measured against the live site and the real archive on
|
||||
2026-08-16 and 2026-08-20, not inferred from documentation.
|
||||
|
||||
## Why gallery-dl and not a hand-rolled script
|
||||
|
||||
The hard parts of fetching Instagram are pagination, cookie handling, CDN URL
|
||||
expiry and resumption. gallery-dl already has all of them, plus extractors that
|
||||
map 1:1 onto our sidecar directory layout (`posts`, `reels`, `stories`,
|
||||
`highlights`). Rolling our own would mean reimplementing the ban-sensitive part
|
||||
by hand.
|
||||
|
||||
## The account was suspended on 2026-08-17 — read this first
|
||||
|
||||
The account used for all of the below was suspended the same day this tooling
|
||||
was built, for "activity that doesn't follow our Community Standards on spam".
|
||||
The fetching was not the expensive part. **Verification was.**
|
||||
|
||||
**It was restored, and synced normally again on 2026-08-20** — a full run
|
||||
across all six profiles with 0 failures and 0 CDN 429s. That is not evidence
|
||||
the limits were imagined; it is one data point on a restored account that has
|
||||
been treated carefully since. Everything below still applies, and the budget is
|
||||
still per session rather than per command.
|
||||
|
||||
What was actually spent against `instagram.com` in a few hours, from one
|
||||
session and one IP:
|
||||
|
||||
| activity | rough requests | downloaded |
|
||||
|---|---:|---|
|
||||
| enumerating a profile grid by scrolling it in an automated browser | ~18 pages | nothing |
|
||||
| the same profile again, after a bug in the scraping selector | ~18 pages | nothing |
|
||||
| a Reels tab enumerated the same way | ~9 pages | nothing |
|
||||
| full `-j` metadata dumps of one profile, twice | ~16 pages | nothing |
|
||||
| `--simulate` runs over the same profile, three times | ~24 pages | nothing |
|
||||
| single-post `/p/<code>/` fetches while testing filename formats | ~8 | a handful |
|
||||
| an aborted sync that re-ran every listing pass before dying | ~40 pages | ~270 MB |
|
||||
| the real sync, 24 sources across 6 profiles | ~150 pages | 2.2 GB |
|
||||
|
||||
The two rows that actually mattered to the archive are the last one and part of
|
||||
the second-to-last. **Everything above them produced no files at all**, and
|
||||
together they were a comparable number of requests.
|
||||
|
||||
The warnings arrived in this order and were each rationalised:
|
||||
|
||||
1. `429 Too Many Requests` from `scontent-*.cdninstagram.com`, losing two
|
||||
videos. Treated as a pacing problem — pacing was lowered and the run
|
||||
continued.
|
||||
2. `400 Bad Request` from `/api/v1/highlights/<id>/highlights_tray/`, on an
|
||||
endpoint that had worked hours earlier. Correctly read as a possible block;
|
||||
requests stopped.
|
||||
3. Suspension.
|
||||
|
||||
**Treat the first CDN 429 as a stop signal for the session, not a tuning
|
||||
parameter.** It is the tolerant surface complaining; if that surface is
|
||||
complaining, the rate-limited one has been unhappy for a while.
|
||||
|
||||
### Rules that follow from this
|
||||
|
||||
- **Count verification requests against the same budget as fetching.** A
|
||||
`--simulate`, a `-j` dump and a browser scroll all hit `instagram.com` and
|
||||
download nothing. Being read-only does not make them free; it makes them
|
||||
invisible, which is worse.
|
||||
- **Never enumerate the live site with an automated browser.** Scrolling a
|
||||
214-post grid is ~18 paginated GraphQL loads at machine speed with no dwell
|
||||
time between them. It is the most obviously non-human thing in this whole
|
||||
document, and it was done here twice on one profile.
|
||||
- **Verify against the archive, not against Instagram.** Every naming, dating
|
||||
and classification question answered in this file could have been answered
|
||||
from files already on disk plus a single listing pass.
|
||||
- **`probe_live` is not cached, so every restart re-enumerates everything.**
|
||||
The aborted run cost a full duplicate set of listing passes for five
|
||||
profiles. Cache probe output to disk before running anything twice.
|
||||
- **Budget per session, not per command.** Nothing in the tooling knows what
|
||||
the last command spent.
|
||||
|
||||
### For a replacement account
|
||||
|
||||
- Let it exist and be used normally for a while before pointing any tool at it.
|
||||
- Keep the cookie on one machine and one public IP, as before.
|
||||
- Start with a single small profile and stop for the day afterwards.
|
||||
- Prefer Instagram's own "Download a copy" export where possible: it is
|
||||
first-party, costs no scraping requests, and carries the metadata this whole
|
||||
document works around not having.
|
||||
|
||||
## The safety model — read this before changing any option
|
||||
|
||||
The ban vector is **requests to `instagram.com`**, not bandwidth. See
|
||||
`docs/jdownloader.md` for the history; Instaloader got this account banned by
|
||||
asking `instagram.com` a question *per post*.
|
||||
|
||||
gallery-dl has two API backends and the difference is exactly that vector:
|
||||
|
||||
```python
|
||||
if self.config("api") == "graphql":
|
||||
self.api = InstagramGraphqlAPI(self) # per-post api.media() for every
|
||||
else: # video and every carousel
|
||||
self.api = InstagramRestAPI(self) # <- default, listing-only
|
||||
```
|
||||
|
||||
The REST backend paginates at `count: 30` (feed) / `page_size: 50` (clips), and
|
||||
those responses already carry `carousel_media`, `image_versions2`,
|
||||
`video_versions` and `product_type`. **No per-post request.** A 300-post
|
||||
profile costs roughly 10 requests to `instagram.com`.
|
||||
|
||||
Rules, in order of importance:
|
||||
|
||||
1. **`"api": "rest"` always.** Never `graphql`. This is the whole ballgame.
|
||||
2. **Never enable `metadata`-style options that trigger extra calls.** If a
|
||||
field is not already in the listing response, it is not worth a request.
|
||||
3. **Pace it.** `"sleep-request": [4.0, 7.0]` — a randomised gap, not a fixed
|
||||
one. Also `"sleep": [1.0, 3.0]` between downloads.
|
||||
4. **Cap the download rate** (`downloader.http.rate`) so the CDN side looks like
|
||||
a person, not a mirror.
|
||||
5. **Run from the same public IP as the browser the cookie came from.** At time
|
||||
of writing that is `mattellite` (`66.23.52.196`); the dev workstation is a
|
||||
*different* public IP and using the cookie from there is precisely what
|
||||
session-hijack detection looks for.
|
||||
6. **No programmatic login, ever.** gallery-dl's username/password path is
|
||||
disabled upstream anyway; use `--cookies-from-browser`.
|
||||
|
||||
Do not add proxy rotation, fingerprint spoofing or account rotation. Throttling
|
||||
and request-avoidance are welcome; evasion is not.
|
||||
|
||||
### Cookies
|
||||
|
||||
The logged-in Chrome on `mattellite` runs with a non-default profile:
|
||||
|
||||
```
|
||||
--user-data-dir=/home/matt/.config/google-chrome-devtools
|
||||
```
|
||||
|
||||
so the cookie flag is:
|
||||
|
||||
```
|
||||
--cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools"
|
||||
```
|
||||
|
||||
Plain `--cookies-from-browser chrome` fails with "Unable to find chrome cookies
|
||||
database" because it looks in `~/.config/google-chrome/`.
|
||||
|
||||
Anonymous access is **not** a viable fallback: it serves lower-resolution media,
|
||||
caps profile pagination at 12 posts, and returns `AuthRequired` for stories and
|
||||
highlights.
|
||||
|
||||
## Output format
|
||||
|
||||
The viewer's parser is the contract, not JD2's exact bytes. `EXPORT_RE` in
|
||||
`src/lib/archive-patterns.ts` accepts all of these, and normalises the index
|
||||
with `parseInt`, so **JD2 and gallery-dl naming interoperate**:
|
||||
|
||||
```
|
||||
"… - CrORBIcJJbM.mp4" -> postId=CrORBIcJJbM index=1
|
||||
"… - CrORBIcJJbM - 1.mp4" -> postId=CrORBIcJJbM index=1
|
||||
"… - C53YPQzp7Wj - 09.jpg" -> postId=C53YPQzp7Wj index=9
|
||||
```
|
||||
|
||||
That means zero-padding and the presence/absence of ` - N` on single-media posts
|
||||
are cosmetic. Don't spend effort forcing them.
|
||||
|
||||
### Directory layout
|
||||
|
||||
| kind | directory | note |
|
||||
|---|---|---|
|
||||
| posts | `<user>` | |
|
||||
| reels | `<user> - reels` | |
|
||||
| stories | `story - <user>` | |
|
||||
| highlights | `story highlights - <user> - <title>` | |
|
||||
|
||||
**Force the directory with `-D`; never use `{username}` for it.** A profile's
|
||||
reels tab returns *collab reels owned by other accounts* — `/0ct0ber19/reels/`
|
||||
served 6 reels owned by `official_artms` and 1 by `chuuo3o`. With
|
||||
`{username}` those would scatter into `official_artms - reels/`. JD2 got this
|
||||
right and the archive proves it: `chuuo3o` and `official_artms` filenames sit
|
||||
inside `0ct0ber19 - reels/`.
|
||||
|
||||
So: **owner in the filename, crawl scope in the directory.**
|
||||
|
||||
### Filenames
|
||||
|
||||
```jsonc
|
||||
"filename": {
|
||||
"sidecar_shortcode and count >= 10":
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num:02}.{extension}",
|
||||
"sidecar_shortcode":
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num}.{extension}",
|
||||
"":
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.{extension}"
|
||||
}
|
||||
```
|
||||
|
||||
`sidecar_shortcode` is set only when the post is a carousel, so it is the
|
||||
carousel discriminator. Conditions are evaluated in order, first match wins
|
||||
(`path.py:265`).
|
||||
|
||||
Stories and highlights use the per-item `{shortcode}`, not `{post_shortcode}`
|
||||
(which is the *reel's* id, shared by every item in it):
|
||||
|
||||
```
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {shortcode}.{extension}"
|
||||
```
|
||||
|
||||
`{date}` on a story/highlight file is the **per-item** `taken_at`
|
||||
(`instagram.py:337` prefers `item["taken_at"]`), verified on a 154-item
|
||||
highlight whose items carried distinct times while `post_date` stayed pinned to
|
||||
the reel. Highlights therefore gain real dates — today they fall back to
|
||||
directory mtime.
|
||||
|
||||
### The timezone is not UTC
|
||||
|
||||
JD2 stamped filenames in **desktop local time (US Eastern)**. Measured across
|
||||
212 comparable posts:
|
||||
|
||||
| model | mismatches |
|
||||
|---|---:|
|
||||
| UTC | 19 |
|
||||
| UTC−5 (EST) | 10 |
|
||||
| UTC−4 (EDT) | **0** |
|
||||
| America/New_York (DST-aware) | **0** |
|
||||
|
||||
`{date:Olocal/%Y-%m-%d}` uses the machine's local zone with per-timestamp DST
|
||||
awareness, which reproduces it — `mattellite` is `America/Toronto`, the same
|
||||
offsets. Note the **trailing `/` must be omitted**: `Olocal/%Y-%m-%d/` puts the
|
||||
separator into the strftime format and it sanitises to an underscore, giving
|
||||
`2026-08-15__0ct0ber19`.
|
||||
|
||||
If the sync ever moves to a host in another timezone, set an explicit
|
||||
`{date:O-4/…}` or the dates will silently shift for ~9% of posts.
|
||||
|
||||
### Caption sidecars
|
||||
|
||||
JD2 writes one `.txt` per post, named without the index, containing the caption
|
||||
with **no trailing newline**, and writes nothing when the caption is empty
|
||||
(measured: 197 of 217 posts, 86 of 86 reels, 0 of 10 stories, 0 of 16
|
||||
highlights). gallery-dl reproduces this exactly with the default
|
||||
`"empty": false`:
|
||||
|
||||
```jsonc
|
||||
{ "name": "metadata", "event": "post", "mode": "custom",
|
||||
"content-format": "{description}", "extension": "txt",
|
||||
"filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.txt" }
|
||||
```
|
||||
|
||||
`"event": "post"` is what makes it one file per post rather than per media file.
|
||||
|
||||
### Metadata sidecar (new — JD2 had no equivalent)
|
||||
|
||||
```jsonc
|
||||
{ "name": "metadata", "event": "post", "mode": "json",
|
||||
"filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.json",
|
||||
"include": ["post_shortcode","post_id","type","date","post_date","username",
|
||||
"fullname","owner_id","description","count","likes","post_url",
|
||||
"sidecar_shortcode"] }
|
||||
```
|
||||
|
||||
Use **`include`**, not `fields` — `fields` is for `mode: custom` and silently
|
||||
does nothing here, leaving `audio_user` blobs (including another user's profile
|
||||
picture URL) in the output.
|
||||
|
||||
The payoff is `type`, which is Instagram's own classification:
|
||||
|
||||
```json
|
||||
{ "post_shortcode": "DbdG9L9jU4m", "type": "post", "count": 2 } // feed video
|
||||
{ "post_shortcode": "Db-lNCoib9m", "type": "reel", "count": 1 } // real reel
|
||||
```
|
||||
|
||||
This is the `product_type: "clips"` signal, delivered free in the listing
|
||||
response. It is the authoritative answer to "is this a reel", and would let the
|
||||
viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
|
||||
"Scanner work" below.
|
||||
|
||||
**`type` is only populated by listing extractors.** Extracting a single
|
||||
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
|
||||
only matters when testing by hand.
|
||||
|
||||
## Cadence, and the budget that enforces it
|
||||
|
||||
**Monthly for everything, daily for stories only.** Stories expire in 24h and
|
||||
cannot be backfilled, so they are the one surface where missing a day means
|
||||
losing the content permanently. Everything else can wait — the skip-archive
|
||||
means an infrequent full sync costs barely more than a frequent one, because it
|
||||
only fetches what is new.
|
||||
|
||||
```
|
||||
# monthly, everything
|
||||
gdl-sync.py --index <viewer-url> --staging ~/gdl/staging \
|
||||
--publish <user>@<nas>:<archives> --archive-db ~/gdl/artms.db \
|
||||
--urls-file artms_account_links.txt --execute
|
||||
|
||||
# daily, stories only -- one request per profile
|
||||
gdl-sync.py ... --only stories --execute
|
||||
```
|
||||
|
||||
A stories-only run is one source per profile and **never seeds**, because a
|
||||
story cannot be in the archive before it is fetched; probing would double the
|
||||
cost of the cheapest surface for no benefit. Six profiles is a handful of
|
||||
requests.
|
||||
|
||||
When scheduling it, **randomise the minute and avoid the hour boundary**. A job
|
||||
that fires at exactly 09:00 every day is a machine; one that fires somewhere in
|
||||
a window looks like someone opening the app.
|
||||
|
||||
The tool now refuses to repeat itself:
|
||||
|
||||
| flag | default | what it prevents |
|
||||
|---|---|---|
|
||||
| `--min-interval` | 20h | re-fetching a source touched recently — the aborted-restart case that re-enumerated five profiles |
|
||||
| `--probe-ttl` | 24h | paying for a listing pass twice within a run cycle |
|
||||
| `--max-sources` | off | a runaway list touching more than intended |
|
||||
| `--force` | off | (escape hatch: ignores both guards) |
|
||||
|
||||
State lives beside the archive DB as `<db>.state.json`, recording per source
|
||||
when it was seeded and last fetched. **Seeding is a one-time bootstrap**: after
|
||||
the first successful sync the archive DB records everything gallery-dl has
|
||||
seen, so the source is never probed again. That is the single biggest saving
|
||||
here — a second full sync costs roughly half what the first did.
|
||||
|
||||
## Incremental sync — why the fetch host needs no copy of the archive
|
||||
|
||||
gallery-dl can skip already-held media two ways, and the difference decides
|
||||
whether the fetcher needs the archive mounted:
|
||||
|
||||
- **By file existence** (default). Needs the destination to already contain the
|
||||
files, so it only works if the archive is mounted where gallery-dl writes.
|
||||
- **By skip-archive** (`--download-archive`). A sqlite DB of ids. Needs nothing
|
||||
on disk.
|
||||
|
||||
We use the second, so the fetch host can write to **local disk and rsync
|
||||
afterwards**. That avoids writing tens of thousands of small files over CIFS,
|
||||
and keeps a mid-sync failure from leaving partial files on the live Resilio
|
||||
share.
|
||||
|
||||
The key is `archive_prefix + archive_fmt`, which for this extractor is the
|
||||
literal `instagram` plus the per-media numeric pk (`instagram.py:25`,
|
||||
`job.py:713-719`). Verified: a 3-image carousel produced
|
||||
|
||||
```
|
||||
instagram3079387627521318672
|
||||
instagram3079387627521429433
|
||||
instagram3079387627529716672
|
||||
```
|
||||
|
||||
and a second run skipped every media file, rewriting only the idempotent
|
||||
`.txt`/`.json` sidecars.
|
||||
|
||||
**Seeding.** `media_id` is not in our filenames, so the DB cannot be built from
|
||||
names alone — but one listing pass (the pass we make anyway) maps every live
|
||||
item to its `media_id`, and the archive's *file listing* says which we already
|
||||
hold. No extra Instagram requests, and no archive content — a listing is
|
||||
enough, which `GET /api/archives/:name/files` already serves.
|
||||
|
||||
Measured on `0ct0ber19`: 2275 live media items, 2248 seeded from the existing
|
||||
listing, **27 left to download** — precisely the media of the two posts added
|
||||
since the last crawl.
|
||||
|
||||
The one trap, which silently seeds almost nothing if you get it backwards:
|
||||
|
||||
| surface | filed under | why |
|
||||
|---|---|---|
|
||||
| posts, reels | `post_shortcode` | carousel children each have their own `shortcode`, which never appears in a filename |
|
||||
| stories, highlights | `shortcode` (per item) | `post_shortcode` is the containing reel's id, shared by every item |
|
||||
|
||||
`live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
|
||||
|
||||
### The skip-archive saves the CDN, not `instagram.com`
|
||||
|
||||
Worth being exact about, because the two costs land on different surfaces and
|
||||
only one of them bans accounts:
|
||||
|
||||
| what | which surface | scales with |
|
||||
|---|---|---|
|
||||
| downloading media | `scontent-*.cdninstagram.com` | how much is **new** |
|
||||
| enumerating the profile to find it | `instagram.com` | how **big** the profile is |
|
||||
|
||||
The skip-archive suppresses the first. It does nothing about the second, so a
|
||||
2275-post profile costs ~76 pages of pagination every run, forever, whether it
|
||||
has three new posts or none. Seeding (above) saved a *second* full pass, not
|
||||
the first.
|
||||
|
||||
Measured on the 2026-08-20 run, from sidecar write times in staging — free,
|
||||
since the run was paying for the listing anyway:
|
||||
|
||||
```
|
||||
1787248852 2026-08-19 … DcOeoVxkthi new, +0s
|
||||
1787248944 2026-08-18 … DcLpfoJCZtp new, +92s
|
||||
1787249058 2026-08-17 … DcIlGbxCUk0 new, +114s
|
||||
1787249162 2026-07-24 … DbKr1TxlPSX ┐ all one second: nothing
|
||||
1787249162 2026-08-15 … DcD-FdBCYGm ┘ downloaded, sidecars only
|
||||
```
|
||||
|
||||
Three posts took ~100s each; the remaining 2272 were enumeration with nothing
|
||||
to show for it.
|
||||
|
||||
**Pinned posts do not break early abort.** Test case 16 previously claimed
|
||||
`0ct0ber19` returns its 3 pinned posts out of date order — that is true of the
|
||||
*web grid*, but the REST `/posts/` listing came back strictly
|
||||
reverse-chronological, newest first, no hoisting. That matters because
|
||||
front-loaded old posts are the one thing that would make `skip: abort:N`
|
||||
dangerous: it would trip on them and abort before reaching anything new.
|
||||
|
||||
So `skip: abort:N` is viable, and cuts ~420 requests per run to ~40-60:
|
||||
|
||||
| surface | live items | pages | with `abort:50` |
|
||||
|---|---:|---:|---:|
|
||||
| posts, 6 profiles | 11,248 | ~377 | ~12 |
|
||||
| reels, 6 profiles | 1,080 | ~24 | ~8 |
|
||||
| stories + highlights | — | ~20 | ~20 |
|
||||
|
||||
N counts consecutive skipped **files**, not posts, so it must clear the largest
|
||||
already-held carousel — `DcD-FdBCYGm` alone is 22 media. 50 is comfortable; 5
|
||||
would not be.
|
||||
|
||||
**The tradeoff is edited carousels.** Test case 15 is a post that gained items
|
||||
after we archived it, and only a full enumeration finds those. Suggested
|
||||
policy: `abort:50` for routine runs, a full sweep occasionally.
|
||||
|
||||
Measured the same day, resuming a stopped run with `--abort 50`:
|
||||
|
||||
| source | live items | enumerated |
|
||||
|---|---:|---:|
|
||||
| `cher_ryppo` posts | 2,151 | **7** |
|
||||
| `cher_ryppo` reels | 92 | 53 |
|
||||
|
||||
One page instead of 72, and every new post was still caught. The 7 is roughly
|
||||
3 new posts plus 4 already-held carousels making up the 50 skipped files.
|
||||
Reels need 53 because they are single-media, so 50 consecutive skips really is
|
||||
50 reels — another reminder that N counts files, and that the same N behaves
|
||||
very differently on a carousel-heavy surface than on a reels tab.
|
||||
|
||||
## Publishing
|
||||
|
||||
The fetch host stages to local disk and rsyncs afterwards. `rsync
|
||||
--ignore-existing` is not an optimisation but the safety property: the archive
|
||||
deliberately outlives Instagram, so publishing must only ever **add**. No
|
||||
`--delete`, and nothing already present is overwritten — including sidecars,
|
||||
which are rewritten every run and would otherwise churn the synced share.
|
||||
|
||||
Publishing happens once at the end of a run, so a profile that fails midway
|
||||
never reaches the archive half-written.
|
||||
|
||||
## Status
|
||||
|
||||
In use for all six ARTMS profiles.
|
||||
|
||||
`withaseul` first — 322 files added (74 media, 241 `.json`, 7 `.txt`), nothing
|
||||
overwritten or deleted. Of the 74 new media, **zero** duplicated media already
|
||||
held under a different name, which is the check that says JD2 and gallery-dl
|
||||
naming really do converge.
|
||||
|
||||
**2026-08-20**, the first full incremental sync, four days after the previous
|
||||
one. 184 new media, 299 files published, 0 failures and **0 CDN 429s**:
|
||||
|
||||
| profile | posts | reels | stories | files added |
|
||||
|---|---:|---:|---:|---:|
|
||||
| 0ct0ber19 | 58 | 2 | 4 | +77 |
|
||||
| official_artms | 12 | — | 2 | +85 |
|
||||
| cher_ryppo | 41 | 1 | 8 | +63 |
|
||||
| zindoriyam | 23 | — | 4 | +35 |
|
||||
| kimxxlip | 16 | — | 2 | +23 |
|
||||
| withaseul | 10 | — | — | +16 |
|
||||
|
||||
The 20 story items are the part that could not have been recovered later.
|
||||
|
||||
Two things made it cheap, and both are worth keeping:
|
||||
|
||||
- The archive DB was already seeded from the previous run, so `--min-interval`
|
||||
and the recorded `seeded` state meant **no probe passes at all**. A state
|
||||
file has to exist for this; if one is missing after a manual run, write it
|
||||
rather than letting the tool re-seed 24 sources.
|
||||
- `--abort 50` (see above) cut the remaining listing cost by roughly 85%.
|
||||
|
||||
The run was deliberately **stopped and resumed** halfway to pick up `--abort`.
|
||||
That is safe precisely because of the state file: the 12 finished sources were
|
||||
already marked `fetched`, so the 20h floor skipped them and only the remaining
|
||||
12 re-ran. Stopping a run is cheap now; it was not before.
|
||||
|
||||
Published files land owned by the SSH user rather than `rslsync`. The viewer
|
||||
reads them fine (world-readable), but Resilio does not own what it syncs; worth
|
||||
a `chown` if that ever matters. This also makes **`rsync` exit 23**
|
||||
("some files/attrs were not transferred") the *normal* outcome of a publish —
|
||||
it is the failed `chown`, not lost data. Confirm by re-running the same rsync
|
||||
with `--dry-run`: an empty file list means everything arrived.
|
||||
|
||||
The profiles to fetch live in `artms_account_links.txt` at the archive root,
|
||||
passed with `--urls-file`.
|
||||
|
||||
## Verified run
|
||||
|
||||
`withaseul`, all four surfaces, staged locally and published to a scratch
|
||||
directory before the live publish above:
|
||||
|
||||
```
|
||||
==> withaseul / posts seeded 915 of 984 live items
|
||||
==> withaseul / reels seeded 28 of 34 live items
|
||||
==> withaseul / stories no results (none active)
|
||||
==> withaseul / highlights no results
|
||||
```
|
||||
|
||||
Output landed correctly, including the collab-reel case — `withaseul - reels`
|
||||
contains 53 files owned by `withaseul`, 10 by `cher_ryppo`, 3 by `0ct0ber19`
|
||||
and 2 by `official_artms`, all with the owner in the filename and the crawl
|
||||
scope as the directory.
|
||||
|
||||
### The CDN rate-limits, and the first run tripped it
|
||||
|
||||
At `rate: 3M` with `sleep: [1.0, 3.0]`, `scontent-*.cdninstagram.com` returned
|
||||
**`429 Too Many Requests`** and two videos were lost (gallery-dl retried, then
|
||||
gave up with exit 4). This is the *tolerant* surface complaining, which is a
|
||||
clear signal the pacing was too aggressive.
|
||||
|
||||
Defaults are now:
|
||||
|
||||
| option | value |
|
||||
|---|---|
|
||||
| `--rate` | `1M` |
|
||||
| `--sleep-request` | 6–10 s |
|
||||
| `--sleep` | 3–6 s |
|
||||
| `sleep-429` | 120 s |
|
||||
| `retries` (extractor and downloader) | 8 |
|
||||
|
||||
Re-running with those recovered both videos and produced **0 failures and 0
|
||||
429s**. Do not raise them for speed; an archive sync has no deadline.
|
||||
|
||||
### yt-dlp is worth installing
|
||||
|
||||
Without it, gallery-dl logs `Cannot import yt-dlp or youtube-dl` and falls back
|
||||
to a progressive URL for DASH videos. The fallback mostly works but is what the
|
||||
429s hit hardest.
|
||||
|
||||
**`pipx install yt-dlp` does not work** — it was the advice here until
|
||||
2026-08-20, and it is wrong. It gives yt-dlp its own venv, so the binary lands
|
||||
on `PATH` while gallery-dl, in a *different* venv, still cannot `import yt_dlp`.
|
||||
The symptom is that everything looks installed and the log keeps saying
|
||||
`Cannot import yt-dlp`. gallery-dl needs it importable, not runnable:
|
||||
|
||||
```sh
|
||||
pipx inject gallery-dl yt-dlp
|
||||
```
|
||||
|
||||
Verify by asking gallery-dl's own interpreter, not the shell:
|
||||
|
||||
```sh
|
||||
/home/matt/.local/share/pipx/venvs/gallery-dl/bin/python -c 'import yt_dlp'
|
||||
```
|
||||
|
||||
## Known quirks
|
||||
|
||||
- **`count` is not the emitted file count.** For 135 of 214 posts it was exactly
|
||||
one higher than the number of files written. This makes the `count >= 10`
|
||||
padding condition mis-pad a handful of 9-item posts (10 of 214 measured). Since
|
||||
the parser normalises the index, this is cosmetic — but it means a re-fetch
|
||||
over an existing JD2 tree writes `- 01.jpg` beside an existing `- 1.jpg`.
|
||||
- **Carousels get edited.** Two posts had a different media count live than on
|
||||
disk. Padding width follows the count *at download time*, so a grown carousel
|
||||
produces mixed widths — the archive already contains one such post from JD2.
|
||||
- **Highlights already have two naming styles on disk**, and every undated file
|
||||
has a dated twin. The scanner dedupes by index so they render once; it is
|
||||
wasted disk, not a display bug.
|
||||
|
||||
## Scanner work — done
|
||||
|
||||
Shipped in `53b1f80` ("read gallery-dl sidecars for reel type and post dates").
|
||||
`useArchiveScanner` tells the three `.json` shapes apart **structurally**, not
|
||||
by filename, in `src/lib/gallery-dl-sidecar.ts`:
|
||||
|
||||
1. Instagram export manifests (`posts_1.json`) — top-level `media` array.
|
||||
2. Instaloader `.json.xz` — GraphQL node under `node` / `__typename`.
|
||||
3. gallery-dl `.json` — flat, `post_shortcode` + `type`, none of the above.
|
||||
|
||||
`post.isReel` now comes from the sidecar's `type`, which is Instagram's own
|
||||
classification, and beats every fallback in `post-tabs.ts`. Post dates are
|
||||
ranked rather than last-write-wins (`src/lib/post-dates.ts`): sidecar beats
|
||||
filename beats mtime.
|
||||
|
||||
Note those files live on **`main`** — they parse the archive at display time
|
||||
and are viewer code, not fetching tooling.
|
||||
|
||||
## Test cases
|
||||
|
||||
Real subjects, all present in the archive today. See
|
||||
`scripts/gdl-sync.py --selftest` for the harness.
|
||||
|
||||
| # | case | shortcode | expected |
|
||||
|---|---|---|---|
|
||||
| 1 | single image | `CwcXnQhOqFG` | one `.jpg`, no index |
|
||||
| 2 | single feed video | `DbdG9L9jU4m` | one `.mp4`, `type: post` |
|
||||
| 3 | carousel, images only | `Cq8LrxSJAJE` | `- 1 … - 3` |
|
||||
| 4 | carousel, image + video | `CtohvHxLnWO` | `- 1.jpg … - 4.mp4`, **no `.txt`** |
|
||||
| 5 | carousel of exactly 9 | `Cv2Hb_brx_N` | 1-digit index |
|
||||
| 6 | carousel of 10+ | `CzM8Uf6B6H_` | 2-digit index `- 01 … - 10` |
|
||||
| 7 | reel shown on the posts grid | `C8FHM6EJl15` | in `<user>`, `type: reel` |
|
||||
| 8 | reel on the reels tab | `Db-lNCoib9m` | in `<user> - reels`, `type: reel` |
|
||||
| 9 | collab reel (other owner) | `DYcZOb0h6Sv` | dir `0ct0ber19 - reels`, filename `chuuo3o` |
|
||||
| 10 | story | live only | `story - <user>`, per-item shortcode + date |
|
||||
| 11 | story highlight | `C-IImhvpFuk` | `story highlights - <user> - <title>` |
|
||||
| 12 | highlight, unicode title | `Drawheeing⠀` | trailing U+2800 preserved in dirname |
|
||||
| 13 | empty caption | `CrdsY5CrSsO` | media written, `.txt` absent |
|
||||
| 14 | deleted post | `C0TgI7sphfZ` | on disk, absent live — must not be removed |
|
||||
| 15 | edited carousel | `C7zG7-jJMlq` | 18 on disk, 8 live — must not be removed |
|
||||
| 16 | pinned posts | `0ct0ber19` | REST listing is strictly reverse-chronological; see below |
|
||||
| 17 | profile avatar | `0ct0ber19.jpg` | base dir, undated |
|
||||
|
||||
Cases 14–16 are reconciliation, not naming: **a sync must never delete**, since
|
||||
the archive deliberately outlives Instagram.
|
||||
|
||||
Not covered, decide before relying on them: the `/reposts/` tab (`0ct0ber19`
|
||||
has one) and `/tagged/`. Neither is fetched today.
|
||||
@@ -1,194 +0,0 @@
|
||||
# JDownloader2 — archive fetching
|
||||
|
||||
> **This file lives only on the `tooling` branch.** `main` is published to
|
||||
> GitHub and deliberately carries none of this — not the host details, not the
|
||||
> IPs, and not the account names. `main`'s history was redacted on 2026-08-20;
|
||||
> real names exist only here.
|
||||
>
|
||||
> There is no `npm run jd2` script — `package.json` and `CLAUDE.md` are kept
|
||||
> byte-identical to `main` so that merging `main` into `tooling` never
|
||||
> conflicts. Run the crawljob generator directly:
|
||||
>
|
||||
> ```sh
|
||||
> npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run
|
||||
> ```
|
||||
|
||||
How content gets into this archive, and why the setup is shaped the way it is.
|
||||
|
||||
## Why JDownloader and not Instaloader
|
||||
|
||||
There are two surfaces, and they're treated very differently:
|
||||
|
||||
| Surface | What hits it | Risk |
|
||||
|---|---|---|
|
||||
| `instagram.com` | profile pages, GraphQL/API metadata | Tied to your session, heavily rate-limited. **This is where bans come from.** |
|
||||
| `scontent*.cdninstagram.com` | the actual media | Signed URLs, CDN-served, tolerant. Mostly a bandwidth question. |
|
||||
|
||||
JDownloader does nearly all its work on the CDN. Instaloader's value — the rich
|
||||
`.json.xz` metadata — comes from asking `instagram.com` a question *per post*.
|
||||
|
||||
Concretely, from this archive: `rivvsofficial` has 188 post-metadata files, so
|
||||
backfilling it cost 188 API requests for one 605-file profile. That's the ban
|
||||
vector. Downloading the 238 photos was never the problem.
|
||||
|
||||
Instaloader got this account banned once. JDownloader with throttling did not.
|
||||
|
||||
> **The account was suspended anyway, on 2026-08-17, for "spam".** Not by
|
||||
> JDownloader, and not by downloading. It was suspended during a day of
|
||||
> *building and verifying* the gallery-dl replacement — automated browser
|
||||
> scrolling to enumerate profile grids, repeated `--simulate` and `-j` metadata
|
||||
> passes, and one aborted sync that re-ran every listing pass before dying.
|
||||
>
|
||||
> The framing above is right about which surface is dangerous and wrong about
|
||||
> what reaches it. **Every read of `instagram.com` counts, including the ones
|
||||
> that download nothing** — and read-only work is easy not to count precisely
|
||||
> because it leaves no files behind. See the post-mortem at the top of
|
||||
> `docs/gallery-dl.md`.
|
||||
>
|
||||
> The rule that would have prevented it: *verify against the archive, never
|
||||
> against the live site*, and treat the first CDN `429` as the end of the
|
||||
> session rather than a pacing knob.
|
||||
|
||||
### What the metadata gap actually costs
|
||||
|
||||
Comparing a JDownloader profile against an Instaloader one:
|
||||
|
||||
| | JDownloader | Instaloader |
|
||||
|---|---|---|
|
||||
| Media | ✅ | ✅ |
|
||||
| Captions (`.txt`) | ✅ | ✅ |
|
||||
| Dates (from filenames) | ✅ | ✅ |
|
||||
| Bio / full name | ❌ | ✅ |
|
||||
| Follower counts | ❌ | ✅ |
|
||||
| External URL | ❌ | ✅ |
|
||||
|
||||
Captions already work — the viewer reads the `.txt` sidecars. Everything missing
|
||||
lives in a *single* profile-level record, not the per-post ones. That's why
|
||||
JDownloader-sourced profiles show "0 followers" and a placeholder bio.
|
||||
|
||||
Not worth extra requests. If you ever want it, the zero-request option is a
|
||||
hand-written `profile.json` sidecar (not implemented yet — ask).
|
||||
|
||||
## Settings that matter
|
||||
|
||||
**Chunks per download → 1.** The single most important one. JDownloader splits
|
||||
each file into multiple ranged requests by default; that `Range` pattern looks
|
||||
nothing like a browser or the app. One chunk = one sequential GET per file.
|
||||
`jd2-sync` sets `chunks=1` per job, so no global change is needed — but set it
|
||||
globally too if you ever add links by hand.
|
||||
|
||||
**Max simultaneous downloads → 2–3**, connections-per-host low. Concurrency is
|
||||
what turns "a user" into a statistic.
|
||||
|
||||
**Leave reconnect / IP-change features off.** A mid-session IP change on a live
|
||||
cookie is a *stronger* anomaly signal than the request rate you'd be avoiding.
|
||||
|
||||
## The cookie
|
||||
|
||||
Exported manually from a real browser session. This is the right approach — no
|
||||
programmatic login anywhere, which is the thing that actually gets flagged.
|
||||
|
||||
- Use it from the **same public IP** as the browser it came from. A cookie used
|
||||
from a different network is what session-hijack detection looks for.
|
||||
- When it expires, **re-export from the browser**. Never add a login step to a tool.
|
||||
- It's a full account credential. Keep it off the NAS share and out of the repo.
|
||||
|
||||
## Workflow
|
||||
|
||||
Two URLs per profile, because the profile grid misses some reels:
|
||||
|
||||
```
|
||||
https://www.instagram.com/<user>/
|
||||
https://www.instagram.com/<user>/reels/
|
||||
```
|
||||
|
||||
They overlap slightly — a reel caught by both lands in each directory and shows
|
||||
up twice in the viewer. That's correct and matches Instagram, which also shows
|
||||
reels in the profile grid *and* the Reels tab.
|
||||
|
||||
## Generating jobs
|
||||
|
||||
Instead of pasting URLs and setting output folders by hand:
|
||||
|
||||
```bash
|
||||
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives --dry-run
|
||||
```
|
||||
|
||||
Review, then write it into JDownloader's folder-watch directory:
|
||||
|
||||
```bash
|
||||
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives \
|
||||
--out ~/.jd2/folderwatch
|
||||
```
|
||||
|
||||
JDownloader runs on the desktop while the archive lives on the NAS, so tell it
|
||||
the path *it* sees:
|
||||
|
||||
```bash
|
||||
npm run jd2 -- --archives /mnt/nas/Instagram-archive/archives \
|
||||
--download-base 'Z:\Instagram-archive\archives' \
|
||||
--out ~/.jd2/folderwatch
|
||||
```
|
||||
|
||||
| Flag | Purpose |
|
||||
|---|---|
|
||||
| `--archives <dir>` | Archive root to scan (or `$ARCHIVES_DIR`) |
|
||||
| `--out <dir>` | JDownloader folder-watch directory |
|
||||
| `--download-base <dir>` | Root path as JDownloader sees it (Windows paths fine) |
|
||||
| `--user <name>` | Just this profile (repeatable) |
|
||||
| `--skip <name>` | Never emit jobs for this directory (repeatable) |
|
||||
| `--chunks <n>` | Connections per file (default 1) |
|
||||
| `--auto-start` | Start immediately instead of parking in LinkGrabber |
|
||||
| `--all-reels` | Emit a reels job even where no reels directory exists |
|
||||
| `--dry-run` | Print instead of writing |
|
||||
|
||||
Defaults are deliberately conservative: `chunks=1`, and links park in the
|
||||
LinkGrabber for review rather than auto-starting.
|
||||
|
||||
Only posts and reels are emitted. Highlight URLs need a numeric id and story
|
||||
URLs expire, so those stay manual.
|
||||
|
||||
Directories that aren't Instagram profiles are skipped by username shape
|
||||
(letters, digits, dots, underscores, ≤30 chars) — pointing a crawl at those
|
||||
spends `instagram.com` requests to be told the profile doesn't exist. For names
|
||||
that *look* like usernames but aren't, use `--skip` or a `.jd2ignore` file in
|
||||
the archive root, one name per line.
|
||||
|
||||
Format reference: `src/org/jdownloader/extensions/folderwatchV2/explain.txt`.
|
||||
JDownloader develops on SVN — read it via the daily mirror at
|
||||
<https://github.com/mycodedoesnotcompile2/jdownloader_mirror> (`svn_trunk/`),
|
||||
not one of the abandoned GitHub copies.
|
||||
|
||||
## Expected layout
|
||||
|
||||
Everything downloads into `<archives>/`, one directory per source:
|
||||
|
||||
```
|
||||
archives/
|
||||
0ct0ber19/ posts
|
||||
0ct0ber19 - reels/ reels
|
||||
story - 0ct0ber19/ stories
|
||||
story highlights - 0ct0ber19 - Heestory/ a highlight
|
||||
```
|
||||
|
||||
Non-archive directories (tool output, exports from elsewhere) live *outside*
|
||||
`archives/` so they never reach the viewer.
|
||||
|
||||
The server picks up changes automatically — its index is keyed on directory
|
||||
mtime, so a new file invalidates only that directory.
|
||||
|
||||
## If something goes wrong
|
||||
|
||||
**429 / rate limited** — stop for hours, not seconds. Retrying into a limit is
|
||||
what converts a soft throttle into something worse.
|
||||
|
||||
**Cookie stops working** — re-export from the browser. Don't add a login step.
|
||||
|
||||
**Files land in the wrong folder** — a Packagizer rule is overriding the job.
|
||||
Generated jobs set `overwritePackagizerEnabled=TRUE` to prevent this; check that
|
||||
rules aren't set to run after it.
|
||||
|
||||
**Viewer doesn't show new posts** — check the file is in the right directory and
|
||||
matches the naming pattern (`YYYY-MM-DD_<user> - <shortcode>[ - NN].<ext>`).
|
||||
The index refreshes on directory mtime, so a genuinely new file is picked up on
|
||||
the next request.
|
||||
@@ -1,112 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Unattended wrapper around gdl-sync.py. One argument: the run mode.
|
||||
#
|
||||
# stories daily ~6 requests; the only surface that cannot be backfilled
|
||||
# profiles monthly every surface, --abort 50: stops enumerating a profile
|
||||
# once it reaches content already held, so it costs
|
||||
# ~40-60 requests and catches everything NEW
|
||||
# full-sweep rarely every surface, no abort: walks each profile to the end
|
||||
# for ~420 requests. The only run that notices posts
|
||||
# EDITED after we archived them, and by far the most
|
||||
# expensive thing here -- see TOOLING.md before running.
|
||||
#
|
||||
# Exits non-zero if the sync does, so cron mails you. Everything is logged.
|
||||
set -eu
|
||||
|
||||
MODE="${1:?usage: gdl-cron.sh stories|profiles|full-sweep}"
|
||||
|
||||
GDL_HOME="${GDL_HOME:-$HOME/gdl}"
|
||||
INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
|
||||
PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
|
||||
STAGING="$GDL_HOME/staging-$MODE"
|
||||
# $$ in the name because two runs in the same SECOND would otherwise share a
|
||||
# log file, and `tee -a` appends -- which made a test see the previous run.
|
||||
LOG="$GDL_HOME/logs/$MODE-$(date +%Y%m%d-%H%M%S)-$$.log"
|
||||
|
||||
PATH="$HOME/.local/bin:$PATH"; export PATH
|
||||
|
||||
# Pacing. These are the values the 2026-08-22 runs used by hand, after the
|
||||
# scraping warning -- roughly double the caution of gdl-sync.py's own defaults
|
||||
# (6-10s / 3-6s / 1M). An archive sync has no deadline; being slow is free and
|
||||
# being restricted is not. Override per-run with GDL_SLEEP_REQUEST etc. if you
|
||||
# ever need to, but raise them rather than lower them.
|
||||
SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
|
||||
SLEEP="${GDL_SLEEP:-5 10}"
|
||||
RATE="${GDL_RATE:-500K}"
|
||||
|
||||
case "$MODE" in
|
||||
# --min-interval 8, not the 20h default. The floor exists to stop an ABORTED
|
||||
# RESTART re-enumerating profiles -- a minutes-to-hours concern. At 20h the
|
||||
# daily timer silently did nothing whenever a manual run had happened the
|
||||
# previous afternoon, which is exactly what happened on 2026-08-21: it fired,
|
||||
# skipped all six sources and reported success. A stories fetch is one
|
||||
# request per profile, so the worst case 8h permits is roughly twelve
|
||||
# requests in a day instead of six.
|
||||
stories) ARGS="--only stories --min-interval 8" ;;
|
||||
profiles) ARGS="--only posts,reels,stories,highlights --abort 50" ;;
|
||||
# No --abort: the whole point of full-sweep is enumerating to the end, so it
|
||||
# is the only run that notices carousels edited after we archived them.
|
||||
full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
|
||||
# Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
|
||||
# "sweep" did not say it was the exhaustive one. Catch the old names rather
|
||||
# than failing with a bare error, in case something still passes them.
|
||||
full) echo "mode 'full' was renamed to 'profiles'" >&2; exit 2 ;;
|
||||
sweep) echo "mode 'sweep' was renamed to 'full-sweep'" >&2; exit 2 ;;
|
||||
*) echo "unknown mode: $MODE (want stories|profiles|full-sweep)" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
mkdir -p "$GDL_HOME/logs"
|
||||
|
||||
# Staging is wiped every run ON PURPOSE. What we already hold is decided by the
|
||||
# skip-archive (--download-archive), never by which files happen to be sitting
|
||||
# in staging, so starting empty is correct -- and it keeps the publish rsync
|
||||
# to just the new files instead of re-walking gigabytes each time.
|
||||
rm -rf "$STAGING"
|
||||
|
||||
echo "=== $MODE run $(date -Is) ===" | tee -a "$LOG"
|
||||
|
||||
# The exit status has to survive the pipe into tee. The left-hand side of a
|
||||
# pipeline runs in a SUBSHELL, so an `exit` in there sets the subshell's status
|
||||
# and the script goes on to return tee's, which is always 0. An earlier version
|
||||
# of this file did exactly that and reported success no matter what the sync
|
||||
# did -- which is why the shebang is bash: PIPESTATUS is the fix.
|
||||
# This run's output only. The check below must never see a previous run's
|
||||
# lines, so it reads this rather than the (appended-to) log.
|
||||
RUNOUT=$(mktemp)
|
||||
trap 'rm -f "$RUNOUT"' EXIT
|
||||
|
||||
set +e
|
||||
# shellcheck disable=SC2086
|
||||
"$GDL_HOME/gdl-sync.py" \
|
||||
--index "$INDEX" \
|
||||
--staging "$STAGING" \
|
||||
--publish "$PUBLISH" \
|
||||
--archive-db "$GDL_HOME/artms.db" \
|
||||
--urls-file "$GDL_HOME/artms_account_links.txt" \
|
||||
--sleep-request $SLEEP_REQUEST \
|
||||
--sleep $SLEEP \
|
||||
--rate "$RATE" \
|
||||
$ARGS --execute 2>&1 | tee -a "$LOG" "$RUNOUT"
|
||||
status=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
# A stories run that skipped every source is NOT a success. It means the
|
||||
# min-interval floor blocked the one surface that cannot be backfilled, and
|
||||
# without this it looks identical to a clean run: exit 0, "0 step(s) failed".
|
||||
# Note this is not the same as "no stories today" -- that shows up as sources
|
||||
# being fetched and returning no results, which is normal and stays quiet.
|
||||
if [ "$MODE" = "stories" ] && grep -q "sources : 0 to sync" "$RUNOUT"; then
|
||||
echo "WARNING: every stories source was skipped by --min-interval." | tee -a "$LOG" >&2
|
||||
echo " Nothing was fetched. Stories expire in 24h and cannot be" | tee -a "$LOG" >&2
|
||||
echo " backfilled, so this is a real loss, not a quiet no-op." | tee -a "$LOG" >&2
|
||||
[ "$status" -eq 0 ] && status=75
|
||||
fi
|
||||
|
||||
echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
|
||||
|
||||
# Keep the log directory from growing without bound.
|
||||
ls -1t "$GDL_HOME/logs" | tail -n +30 | while read -r old; do
|
||||
rm -f "$GDL_HOME/logs/$old"
|
||||
done
|
||||
|
||||
exit $status
|
||||
-1011
File diff suppressed because it is too large
Load Diff
@@ -1,277 +0,0 @@
|
||||
/**
|
||||
* Generate JDownloader2 .crawljob files for the archives on disk.
|
||||
*
|
||||
* The manual flow is: paste a profile URL into JDownloader, paste the /reels
|
||||
* URL separately (the profile page misses some reels), and set the output
|
||||
* folder by hand — times however many profiles you keep. This emits one
|
||||
* crawljob per source with the folder already pointed at the right directory,
|
||||
* so JDownloader's folder-watch picks the whole batch up at once.
|
||||
*
|
||||
* Profiles and their sidecar directories are derived with the same grouping
|
||||
* logic the server uses, so the output folders always match what the viewer
|
||||
* expects to find.
|
||||
*
|
||||
* Only posts and reels are emitted. Story and highlight URLs can't be rebuilt
|
||||
* from a directory name — highlights need their numeric id and stories expire —
|
||||
* so those stay manual.
|
||||
*
|
||||
* Crawljob format verified against JDownloader's own docs for the extension:
|
||||
* src/org/jdownloader/extensions/folderwatchV2/explain.txt. JDownloader
|
||||
* develops on SVN; read it via the daily mirror at
|
||||
* https://github.com/mycodedoesnotcompile2/jdownloader_mirror (svn_trunk/),
|
||||
* not one of the abandoned GitHub copies — several are a decade stale.
|
||||
*
|
||||
* Entries are separated by `->NEW ENTRY<-` and any property may be omitted.
|
||||
* There is also a `setBeforePackagizerEnabled` companion to
|
||||
* `overwritePackagizerEnabled`, if the Packagizer ever needs to see these
|
||||
* values before they're applied.
|
||||
*
|
||||
* Usage:
|
||||
* npx tsx scripts/jd2-sync.ts --archives <dir> [options]
|
||||
*
|
||||
* --archives <dir> Archive root to scan (default: $ARCHIVES_DIR)
|
||||
* --out <dir> JDownloader folder-watch directory to write into
|
||||
* --download-base <dir> Root path as *JDownloader* sees it, when it runs on
|
||||
* a different machine than this script (e.g. a mapped
|
||||
* drive). Defaults to --archives.
|
||||
* --user <name> Only this profile (repeatable)
|
||||
* --skip <name> Never emit jobs for this directory (repeatable).
|
||||
* Also read from a `.jd2ignore` file in the archive
|
||||
* root, one name per line.
|
||||
* --chunks <n> Connections per file (default 1: multi-chunk ranged
|
||||
* requests are the one CDN pattern that doesn't look
|
||||
* like a browser)
|
||||
* --auto-start Start downloads immediately instead of parking them
|
||||
* in the LinkGrabber for review
|
||||
* --all-reels Emit a reels job even where no reels directory
|
||||
* exists yet
|
||||
* --dry-run Print the crawljob instead of writing it
|
||||
*/
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { groupArchiveDirectories, ArchiveSource } from '../src/lib/archive-grouping.js';
|
||||
|
||||
interface Options {
|
||||
archives: string;
|
||||
out: string | null;
|
||||
downloadBase: string;
|
||||
users: string[];
|
||||
skip: Set<string>;
|
||||
chunks: number;
|
||||
autoStart: boolean;
|
||||
allReels: boolean;
|
||||
dryRun: boolean;
|
||||
}
|
||||
|
||||
const parseArgs = (argv: string[]): Options => {
|
||||
const opts: Options = {
|
||||
archives: process.env.ARCHIVES_DIR ?? '',
|
||||
out: null,
|
||||
downloadBase: '',
|
||||
users: [],
|
||||
skip: new Set(),
|
||||
chunks: 1,
|
||||
autoStart: false,
|
||||
allReels: false,
|
||||
dryRun: false,
|
||||
};
|
||||
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const arg = argv[i];
|
||||
const next = () => argv[++i];
|
||||
switch (arg) {
|
||||
case '--archives': opts.archives = path.resolve(next()); break;
|
||||
case '--out': opts.out = path.resolve(next()); break;
|
||||
case '--download-base': opts.downloadBase = next(); break;
|
||||
case '--user': opts.users.push(next()); break;
|
||||
case '--skip': opts.skip.add(next()); break;
|
||||
case '--chunks': opts.chunks = parseInt(next(), 10); break;
|
||||
case '--auto-start': opts.autoStart = true; break;
|
||||
case '--all-reels': opts.allReels = true; break;
|
||||
case '--dry-run': opts.dryRun = true; break;
|
||||
case '--help': case '-h': printUsage(); process.exit(0);
|
||||
default:
|
||||
console.error(`Unknown argument: ${arg}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
if (!opts.archives) {
|
||||
console.error('No archive root. Pass --archives <dir> or set ARCHIVES_DIR.');
|
||||
process.exit(1);
|
||||
}
|
||||
if (!opts.downloadBase) opts.downloadBase = opts.archives;
|
||||
if (!opts.out && !opts.dryRun) {
|
||||
console.error('No destination. Pass --out <folder-watch dir>, or --dry-run to preview.');
|
||||
process.exit(1);
|
||||
}
|
||||
return opts;
|
||||
};
|
||||
|
||||
const printUsage = () => {
|
||||
const header = readHeaderComment();
|
||||
console.log(header);
|
||||
};
|
||||
|
||||
/** Print the usage block from this file's own header comment. */
|
||||
const readHeaderComment = () => {
|
||||
try {
|
||||
const self = fs.readFileSync(new URL(import.meta.url), 'utf8');
|
||||
const usage = self.slice(self.indexOf(' * Usage:'), self.indexOf(' */'));
|
||||
return usage.split('\n').map(l => l.replace(/^ \* ?/, '')).join('\n');
|
||||
} catch {
|
||||
return 'See the comment at the top of scripts/jd2-sync.ts';
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* JDownloader escapes nothing in crawljob values, so a stray newline would
|
||||
* silently split a property. Paths with spaces are fine as-is.
|
||||
*/
|
||||
const sanitise = (value: string) => value.replace(/[\r\n]+/g, ' ').trim();
|
||||
|
||||
/**
|
||||
* Instagram usernames are 1–30 characters of letters, digits, dots and
|
||||
* underscores. Archive roots also collect directories that aren't profiles at
|
||||
* all — tool output, exports from other services — and pointing a crawl at
|
||||
* those spends requests on instagram.com to be told the profile doesn't exist.
|
||||
* That's the exact traffic worth not spending.
|
||||
*/
|
||||
const USERNAME_RE = /^[A-Za-z0-9._]{1,30}$/;
|
||||
|
||||
/** Directory names to skip, from `.jd2ignore` in the archive root. */
|
||||
const readIgnoreFile = (archives: string): string[] => {
|
||||
try {
|
||||
return fs.readFileSync(path.join(archives, '.jd2ignore'), 'utf8')
|
||||
.split('\n').map(l => l.trim()).filter(l => l && !l.startsWith('#'));
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
};
|
||||
|
||||
interface Job {
|
||||
user: string;
|
||||
kind: 'posts' | 'reels';
|
||||
url: string;
|
||||
packageName: string;
|
||||
downloadFolder: string;
|
||||
fileCount: number | null;
|
||||
}
|
||||
|
||||
const buildJobs = (opts: Options): Job[] => {
|
||||
const dirNames = fs.readdirSync(opts.archives, { withFileTypes: true })
|
||||
.filter(e => e.isDirectory() && !/^[.@_]/.test(e.name))
|
||||
.map(e => e.name);
|
||||
|
||||
const groups = groupArchiveDirectories(dirNames);
|
||||
const jobs: Job[] = [];
|
||||
const skipped: string[] = [];
|
||||
|
||||
for (const name of readIgnoreFile(opts.archives)) opts.skip.add(name);
|
||||
|
||||
const countFiles = (dir: string): number | null => {
|
||||
try {
|
||||
return fs.readdirSync(path.join(opts.archives, dir)).length;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
// JDownloader must be given the path *it* can see, which differs from the
|
||||
// scan path whenever the archive lives on a share.
|
||||
const downloadFolderFor = (dir: string) =>
|
||||
opts.downloadBase.includes('\\')
|
||||
? `${opts.downloadBase.replace(/\\$/, '')}\\${dir}`
|
||||
: path.posix.join(opts.downloadBase, dir);
|
||||
|
||||
for (const [user, sources] of [...groups].sort(([a], [b]) => a.localeCompare(b))) {
|
||||
if (opts.users.length && !opts.users.includes(user)) continue;
|
||||
|
||||
if (opts.skip.has(user)) { skipped.push(`${user} (ignored)`); continue; }
|
||||
if (!USERNAME_RE.test(user)) { skipped.push(`${user} (not a username)`); continue; }
|
||||
|
||||
const has = (kind: ArchiveSource['kind']) => sources.find(s => s.kind === kind);
|
||||
const base = has('posts');
|
||||
if (!base) continue; // sidecar-only group: nothing sensible to point a URL at
|
||||
|
||||
jobs.push({
|
||||
user, kind: 'posts',
|
||||
url: `https://www.instagram.com/${encodeURIComponent(user)}/`,
|
||||
packageName: base.dir,
|
||||
downloadFolder: downloadFolderFor(base.dir),
|
||||
fileCount: countFiles(base.dir),
|
||||
});
|
||||
|
||||
const reels = has('reels');
|
||||
if (reels || opts.allReels) {
|
||||
const dir = reels?.dir ?? `${user} - reels`;
|
||||
jobs.push({
|
||||
user, kind: 'reels',
|
||||
url: `https://www.instagram.com/${encodeURIComponent(user)}/reels/`,
|
||||
packageName: dir,
|
||||
downloadFolder: downloadFolderFor(dir),
|
||||
fileCount: reels ? countFiles(dir) : null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (skipped.length) {
|
||||
console.error(`Skipped ${skipped.length} director${skipped.length === 1 ? 'y' : 'ies'}:`);
|
||||
for (const s of skipped) console.error(` - ${s}`);
|
||||
console.error('');
|
||||
}
|
||||
|
||||
return jobs;
|
||||
};
|
||||
|
||||
const renderCrawljob = (jobs: Job[], opts: Options): string =>
|
||||
jobs.map(job => [
|
||||
`text=${sanitise(job.url)}`,
|
||||
`packageName=${sanitise(job.packageName)}`,
|
||||
`downloadFolder=${sanitise(job.downloadFolder)}`,
|
||||
`chunks=${opts.chunks}`,
|
||||
// Without this a Packagizer rule can override downloadFolder and scatter
|
||||
// files away from the directory the viewer reads.
|
||||
'overwritePackagizerEnabled=TRUE',
|
||||
`autoStart=${opts.autoStart ? 'TRUE' : 'FALSE'}`,
|
||||
`autoConfirm=${opts.autoStart ? 'TRUE' : 'FALSE'}`,
|
||||
'enabled=TRUE',
|
||||
`comment=instaarchive jd2-sync (${job.kind})`,
|
||||
].join('\n')).join('\n->NEW ENTRY<-\n');
|
||||
|
||||
const main = () => {
|
||||
const opts = parseArgs(process.argv.slice(2));
|
||||
const jobs = buildJobs(opts);
|
||||
|
||||
if (!jobs.length) {
|
||||
console.error('No profiles matched.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.error(`Archive root : ${opts.archives}`);
|
||||
console.error(`JD sees root : ${opts.downloadBase}`);
|
||||
console.error(`Jobs : ${jobs.length} (${new Set(jobs.map(j => j.user)).size} profiles)\n`);
|
||||
for (const job of jobs) {
|
||||
const count = job.fileCount === null ? 'new' : `${job.fileCount} files`;
|
||||
console.error(` ${job.kind.padEnd(5)} ${job.user.padEnd(24)} -> ${job.packageName} (${count})`);
|
||||
}
|
||||
console.error('');
|
||||
|
||||
const body = renderCrawljob(jobs, opts);
|
||||
|
||||
if (opts.dryRun || !opts.out) {
|
||||
console.log(body);
|
||||
return;
|
||||
}
|
||||
|
||||
fs.mkdirSync(opts.out, { recursive: true });
|
||||
const file = path.join(opts.out, `instaarchive-${new Date().toISOString().replace(/[:.]/g, '-')}.crawljob`);
|
||||
fs.writeFileSync(file, body, 'utf8');
|
||||
console.error(`Wrote ${file}`);
|
||||
console.error(opts.autoStart
|
||||
? 'Downloads will start automatically.'
|
||||
: 'Links land in the LinkGrabber for review; start them when ready.');
|
||||
};
|
||||
|
||||
main();
|
||||
@@ -1,237 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Find a profile's reels by scrolling the real page, not calling the API.
|
||||
|
||||
gallery-dl's dedicated reels extractor POSTs to /api/v1/clips/user/, which
|
||||
Instagram now 302-redirects to the home page for this account -- confirmed on
|
||||
2026-08-26/27 across multiple profiles, hours apart, with a freshly-warmed
|
||||
session and a correct X-IG-WWW-Claim header. The reels tab itself loads fine
|
||||
in a real browser, so this drives the SAME logged-in Chrome (already exposed
|
||||
on loopback CDP for MCP automation -- see TOOLING.md) via the DevTools
|
||||
protocol, scrolls it like a person would, and scrapes `/reel/<code>/` links
|
||||
out of the rendered page instead.
|
||||
|
||||
This only finds shortcodes; it never downloads anything itself. What comes
|
||||
out the other end is deduped against the archive (via the same --index used
|
||||
elsewhere) and printed as plain post URLs -- feed them to gdl-sync.py:
|
||||
|
||||
./scripts/reels-scrape.py --profile someuser \\
|
||||
--index https://instaarchive.ergosteur.com > /tmp/someuser-reels.txt
|
||||
|
||||
./scripts/gdl-sync.py --publish user@host:/path --staging /var/tmp/gdl \\
|
||||
--post-urls-file /tmp/someuser-reels.txt \\
|
||||
--sleep-request 12 20 --sleep 5 10 --rate 500K --execute
|
||||
|
||||
Requires the `websocket-client` package (only imported inside scrape_reels,
|
||||
so everything else here stays importable -- and testable -- without it).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import itertools
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from urllib.request import urlopen
|
||||
|
||||
RE_REEL_HREF = re.compile(r"/reel/([^/?#]+)")
|
||||
|
||||
# The reels tab has finished loading once a scroll round finds no NEW
|
||||
# shortcodes this many times in a row -- lazy-loaded feeds sometimes stall
|
||||
# for a round or two before producing more, so one dry round is not enough.
|
||||
DEFAULT_MAX_IDLE_ROUNDS = 3
|
||||
DEFAULT_MAX_SCROLLS = 200
|
||||
DEFAULT_SCROLL_PAUSE = (2.0, 3.5)
|
||||
|
||||
SCRAPE_JS = (
|
||||
"(() => {"
|
||||
"window.scrollTo(0, document.body.scrollHeight);"
|
||||
"return Array.from(document.querySelectorAll('a[href*=\"/reel/\"]'))"
|
||||
".map(a => a.getAttribute('href'));"
|
||||
"})()"
|
||||
)
|
||||
|
||||
|
||||
def extract_shortcodes(hrefs: list[str]) -> list[str]:
|
||||
codes = []
|
||||
for href in hrefs:
|
||||
if m := RE_REEL_HREF.search(href):
|
||||
codes.append(m.group(1))
|
||||
return codes
|
||||
|
||||
|
||||
class CDPError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class CDP:
|
||||
"""
|
||||
A deliberately minimal synchronous DevTools Protocol client: one request
|
||||
in flight at a time, which is all a linear scroll-and-scrape loop needs.
|
||||
Anything fancier (concurrent requests, event subscriptions) is scope this
|
||||
script has no reason to carry.
|
||||
"""
|
||||
|
||||
def __init__(self, ws_url: str, timeout: float = 30.0):
|
||||
import websocket # local: keep this importable without the package
|
||||
self.ws = websocket.create_connection(ws_url, timeout=timeout)
|
||||
self._ids = itertools.count(1)
|
||||
|
||||
def send(self, method: str, params: dict | None = None,
|
||||
session_id: str | None = None) -> dict:
|
||||
msg_id = next(self._ids)
|
||||
payload = {"id": msg_id, "method": method, "params": params or {}}
|
||||
if session_id:
|
||||
payload["sessionId"] = session_id
|
||||
self.ws.send(json.dumps(payload))
|
||||
while True:
|
||||
msg = json.loads(self.ws.recv())
|
||||
if msg.get("id") != msg_id:
|
||||
continue # an event notification, not our reply -- ignore
|
||||
if "error" in msg:
|
||||
raise CDPError(f"{method}: {msg['error']}")
|
||||
return msg.get("result", {})
|
||||
|
||||
def close(self) -> None:
|
||||
self.ws.close()
|
||||
|
||||
|
||||
def scrape_reels(user: str, cdp_port: int = 9222,
|
||||
scroll_pause: tuple[float, float] = DEFAULT_SCROLL_PAUSE,
|
||||
max_idle_rounds: int = DEFAULT_MAX_IDLE_ROUNDS,
|
||||
max_scrolls: int = DEFAULT_MAX_SCROLLS,
|
||||
log=lambda msg: None) -> list[str]:
|
||||
"""
|
||||
Open the profile's reels tab in a NEW tab of the already-signed-in Chrome,
|
||||
scroll it to the bottom repeatedly, and collect every unique `/reel/`
|
||||
shortcode that appears. Closes the tab when done either way.
|
||||
"""
|
||||
version = json.loads(urlopen(f"http://localhost:{cdp_port}/json/version",
|
||||
timeout=10).read())
|
||||
browser = CDP(version["webSocketDebuggerUrl"])
|
||||
target_id = None
|
||||
try:
|
||||
target = browser.send("Target.createTarget", {
|
||||
"url": f"https://www.instagram.com/{user}/reels/"})
|
||||
target_id = target["targetId"]
|
||||
attach = browser.send("Target.attachToTarget", {
|
||||
"targetId": target_id, "flatten": True})
|
||||
session_id = attach["sessionId"]
|
||||
browser.send("Page.enable", session_id=session_id)
|
||||
browser.send("Runtime.enable", session_id=session_id)
|
||||
|
||||
time.sleep(4.0) # initial page load, before the first scroll
|
||||
|
||||
seen: set[str] = set()
|
||||
idle = 0
|
||||
for i in range(max_scrolls):
|
||||
result = browser.send(
|
||||
"Runtime.evaluate",
|
||||
{"expression": SCRAPE_JS, "returnByValue": True},
|
||||
session_id=session_id)
|
||||
hrefs = result.get("result", {}).get("value") or []
|
||||
codes = extract_shortcodes(hrefs)
|
||||
new = [c for c in codes if c not in seen]
|
||||
seen.update(new)
|
||||
log(f" scroll {i + 1}: {len(seen)} unique reels so far (+{len(new)})")
|
||||
|
||||
if new:
|
||||
idle = 0
|
||||
else:
|
||||
idle += 1
|
||||
if idle >= max_idle_rounds:
|
||||
break
|
||||
|
||||
time.sleep(random.uniform(*scroll_pause))
|
||||
|
||||
return sorted(seen)
|
||||
finally:
|
||||
if target_id:
|
||||
try:
|
||||
browser.send("Target.closeTarget", {"targetId": target_id})
|
||||
except CDPError:
|
||||
pass # best-effort cleanup; a leftover tab is harmless
|
||||
browser.close()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--profile", required=True,
|
||||
help="Instagram username to scrape reels for")
|
||||
ap.add_argument("--index", required=True,
|
||||
help="existing archive listing, same as gdl-sync.py's "
|
||||
"--index: a local root, or the viewer's base URL. "
|
||||
"Used only to dedupe -- already-archived shortcodes "
|
||||
"are dropped before anything is printed")
|
||||
ap.add_argument("--cdp-port", type=int, default=9222,
|
||||
help="Chrome's loopback DevTools port (see TOOLING.md)")
|
||||
ap.add_argument("--scroll-pause", nargs=2, type=float,
|
||||
default=list(DEFAULT_SCROLL_PAUSE), metavar=("MIN", "MAX"),
|
||||
help="random pause between scrolls, seconds")
|
||||
ap.add_argument("--max-idle-rounds", type=int, default=DEFAULT_MAX_IDLE_ROUNDS,
|
||||
help="stop after this many consecutive scrolls with no "
|
||||
"new reels")
|
||||
ap.add_argument("--max-scrolls", type=int, default=DEFAULT_MAX_SCROLLS,
|
||||
help="hard ceiling on scroll rounds, in case a page never "
|
||||
"goes idle")
|
||||
ap.add_argument("--out", type=Path,
|
||||
help="write new reel URLs here, one per line (default: "
|
||||
"stdout)")
|
||||
args = ap.parse_args()
|
||||
|
||||
import importlib.util
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"gdl_sync", Path(__file__).with_name("gdl-sync.py"))
|
||||
gdl = importlib.util.module_from_spec(spec)
|
||||
sys.modules["gdl_sync"] = gdl
|
||||
spec.loader.exec_module(gdl)
|
||||
|
||||
index = gdl.ArchiveIndex(args.index)
|
||||
# Deduped against the WHOLE archive, not just this profile's own
|
||||
# directories: a shortcode is globally unique, and a reels tab commonly
|
||||
# surfaces reposts/collabs by OTHER tracked accounts. Missing that let a
|
||||
# 2026-08-27 run re-fetch 9 reels already held under their true owner's
|
||||
# directory -- gallery-dl filed them correctly there (by the post's real
|
||||
# `username`, not the scraped profile), so nothing was lost, but it spent
|
||||
# 9 avoidable Instagram requests to find that out. All of this is local
|
||||
# requests to our own viewer, never to instagram.com, so checking every
|
||||
# profile costs nothing on the budget that actually matters.
|
||||
profiles = index.profiles()
|
||||
have: set[str] = set()
|
||||
for profile in profiles:
|
||||
have.update(code for code, _ in gdl.index_existing(index.listing(profile)))
|
||||
print(f"archive already holds {len(have)} shortcode(s) across "
|
||||
f"{len(profiles)} profile(s)", file=sys.stderr)
|
||||
|
||||
print(f"scraping https://www.instagram.com/{args.profile}/reels/ ...",
|
||||
file=sys.stderr)
|
||||
codes = scrape_reels(args.profile, cdp_port=args.cdp_port,
|
||||
scroll_pause=tuple(args.scroll_pause),
|
||||
max_idle_rounds=args.max_idle_rounds,
|
||||
max_scrolls=args.max_scrolls,
|
||||
log=lambda msg: print(msg, file=sys.stderr))
|
||||
|
||||
new_codes = [c for c in codes if c not in have]
|
||||
print(f"found {len(codes)} reel(s) on the page, {len(codes) - len(new_codes)} "
|
||||
f"already archived, {len(new_codes)} new", file=sys.stderr)
|
||||
|
||||
urls = [f"https://www.instagram.com/{args.profile}/reel/{c}/"
|
||||
for c in new_codes]
|
||||
text = "\n".join(urls)
|
||||
if args.out:
|
||||
args.out.write_text(text + ("\n" if text else ""))
|
||||
print(f"wrote {len(urls)} URL(s) to {args.out}", file=sys.stderr)
|
||||
else:
|
||||
if text:
|
||||
print(text)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,114 +0,0 @@
|
||||
#!/bin/bash
|
||||
# The full reels pipeline for one profile: scrape by scrolling the real page,
|
||||
# then fetch and publish whatever's new. See TOOLING.md ("Reels: the API is
|
||||
# blocked, scrape by scrolling instead") for why this exists at all -- the
|
||||
# dedicated reels API is blocked, this drives the actual signed-in browser
|
||||
# session instead, and is slower by design.
|
||||
#
|
||||
# Usage: ./reels-sync.sh <username-or-profile-url>
|
||||
# ./reels-sync.sh zindoriyam
|
||||
# ./reels-sync.sh https://www.instagram.com/zindoriyam/
|
||||
#
|
||||
# Deliberately NOT wired into gdl-cron.sh or the timers -- see TOOLING.md.
|
||||
# Exits non-zero if either step does. Everything is logged.
|
||||
set -eu
|
||||
|
||||
RAW="${1:?usage: reels-sync.sh <username-or-profile-url>}"
|
||||
|
||||
GDL_HOME="${GDL_HOME:-$HOME/gdl}"
|
||||
GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
|
||||
INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
|
||||
PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
|
||||
|
||||
# Same hand-paced pacing gdl-cron.sh uses -- see its comment for why. Override
|
||||
# per-run with GDL_SLEEP_REQUEST etc. if you ever need to, but raise them
|
||||
# rather than lower them.
|
||||
SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
|
||||
SLEEP="${GDL_SLEEP:-5 10}"
|
||||
RATE="${GDL_RATE:-500K}"
|
||||
SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}"
|
||||
MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}"
|
||||
|
||||
PATH="$HOME/.local/bin:$PATH"; export PATH
|
||||
|
||||
if [ ! -x "$GDL_PYTHON" ]; then
|
||||
echo "reels-scrape.py needs gallery-dl's own pipx venv python (websocket-client" >&2
|
||||
echo "was injected there, not into the system python): $GDL_PYTHON not found" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Same username-from-URL parsing gdl-sync.py already does for --urls-file,
|
||||
# reused rather than re-implemented so the two never drift apart.
|
||||
PROFILE=$("$GDL_PYTHON" -c "
|
||||
import re, sys, importlib.util
|
||||
from pathlib import Path
|
||||
spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
|
||||
gdl = importlib.util.module_from_spec(spec)
|
||||
sys.modules['gdl_sync'] = gdl
|
||||
spec.loader.exec_module(gdl)
|
||||
raw = '$RAW'
|
||||
m = gdl.RE_PROFILE_URL.match(raw)
|
||||
user = m.group('user') if m else raw.strip('/')
|
||||
if not user or '/' in user or ' ' in user:
|
||||
print(f'cannot read a username from {raw!r}', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(user)
|
||||
")
|
||||
|
||||
mkdir -p "$GDL_HOME/logs"
|
||||
LOG="$GDL_HOME/logs/reels-$PROFILE-$(date +%Y%m%d-%H%M%S)-$$.log"
|
||||
URLS_FILE="$GDL_HOME/$PROFILE-reels.txt"
|
||||
STAGING="$GDL_HOME/staging-reels-$PROFILE"
|
||||
|
||||
echo "=== reels-sync $PROFILE $(date -Is) ===" | tee -a "$LOG"
|
||||
|
||||
# The exit status has to survive the pipe into tee -- see gdl-cron.sh's
|
||||
# comment on PIPESTATUS for why this needs to be bash, not sh.
|
||||
set +e
|
||||
"$GDL_PYTHON" "$GDL_HOME/reels-scrape.py" \
|
||||
--profile "$PROFILE" \
|
||||
--index "$INDEX" \
|
||||
--scroll-pause $SCROLL_PAUSE \
|
||||
--max-idle-rounds "$MAX_IDLE_ROUNDS" \
|
||||
--out "$URLS_FILE" 2>&1 | tee -a "$LOG"
|
||||
scrape_status=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
if [ "$scrape_status" -ne 0 ]; then
|
||||
echo "=== exit $scrape_status (scrape failed) at $(date -Is) ===" | tee -a "$LOG"
|
||||
exit "$scrape_status"
|
||||
fi
|
||||
|
||||
if [ ! -s "$URLS_FILE" ]; then
|
||||
echo "no new reels for $PROFILE; nothing to fetch" | tee -a "$LOG"
|
||||
echo "=== exit 0 at $(date -Is) ===" | tee -a "$LOG"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Staging is wiped every run on purpose -- same reasoning as gdl-cron.sh: what
|
||||
# we already hold is decided by the archive dedupe in reels-scrape.py, not by
|
||||
# what happens to be sitting in staging.
|
||||
rm -rf "$STAGING"
|
||||
|
||||
set +e
|
||||
# shellcheck disable=SC2086
|
||||
"$GDL_HOME/gdl-sync.py" \
|
||||
--publish "$PUBLISH" \
|
||||
--staging "$STAGING" \
|
||||
--post-urls-file "$URLS_FILE" \
|
||||
--sleep-request $SLEEP_REQUEST \
|
||||
--sleep $SLEEP \
|
||||
--rate "$RATE" \
|
||||
--execute 2>&1 | tee -a "$LOG"
|
||||
status=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
|
||||
|
||||
# Keep the log directory from growing without bound -- scoped to this
|
||||
# script's own logs so it never touches gdl-cron.sh's rotation.
|
||||
ls -1t "$GDL_HOME/logs" | grep '^reels-' | tail -n +30 | while read -r old; do
|
||||
rm -f "$GDL_HOME/logs/$old"
|
||||
done
|
||||
|
||||
exit "$status"
|
||||
@@ -1,16 +0,0 @@
|
||||
[Unit]
|
||||
# One templated service for all three modes; the instance name (%i) is the
|
||||
# mode: stories, full or sweep.
|
||||
Description=Instagram archive sync (%i)
|
||||
Documentation=file:%h/gdl/TOOLING.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=%h/gdl/gdl-cron.sh %i
|
||||
# A sync has no deadline and the pacing is deliberately slow; a full sweep can
|
||||
# run for hours. Never let systemd kill one midway -- a half-published run is
|
||||
# the one state the publish step is designed to avoid.
|
||||
TimeoutStartSec=infinity
|
||||
Nice=10
|
||||
@@ -1,18 +0,0 @@
|
||||
[Unit]
|
||||
Description=Full Instagram archive sweep, no abort (~420 requests)
|
||||
# The only run that enumerates every profile to the end, and so the only one
|
||||
# that notices carousels edited after we archived them (test case 15).
|
||||
#
|
||||
# It is also by far the most expensive thing here: ~420 requests to
|
||||
# instagram.com, the same order as the run that preceded the 2026-08-21
|
||||
# scraping warning, spent to catch a handful of retroactively edited posts.
|
||||
# Consider running it by hand when you mean to, rather than on a timer.
|
||||
|
||||
[Timer]
|
||||
# Month names are not valid in OnCalendar's date field -- numeric only.
|
||||
OnCalendar=*-01,04,07,10-07 04:00:00
|
||||
RandomizedDelaySec=45m
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,10 +0,0 @@
|
||||
[Unit]
|
||||
Description=Monthly Instagram profile sync (all surfaces, --abort 50)
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-03 04:00:00
|
||||
RandomizedDelaySec=45m
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,15 +0,0 @@
|
||||
[Unit]
|
||||
Description=Daily Instagram stories sync
|
||||
# Stories expire in 24h and cannot be backfilled. This is the only timer whose
|
||||
# missed run means content is gone for good, which is what Persistent= is for.
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 09:00:00
|
||||
# Not a fixed time: a job firing at exactly 09:00 every day is obviously a
|
||||
# machine, and the whole safety model is about not looking like one.
|
||||
RandomizedDelaySec=45m
|
||||
# Catch up after the host was asleep or off. cron would silently skip.
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,272 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Tests for the request-budget logic in gdl-sync.py.
|
||||
|
||||
python3 -m unittest discover -s scripts -p 'test_*.py'
|
||||
|
||||
Deliberately stdlib-only, so it runs anywhere the sync itself runs. What is
|
||||
covered here is the part that decides whether to spend a request — the part
|
||||
whose absence got the archive's Instagram account suspended.
|
||||
"""
|
||||
|
||||
import datetime as dt
|
||||
import importlib.util
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"gdl_sync", Path(__file__).with_name("gdl-sync.py"))
|
||||
gdl = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["gdl_sync"] = gdl
|
||||
_spec.loader.exec_module(gdl)
|
||||
|
||||
NOW = dt.datetime(2026, 8, 18, 12, 0, tzinfo=dt.timezone.utc)
|
||||
NOW_TS = NOW.timestamp()
|
||||
|
||||
|
||||
def ago(hours: float) -> str:
|
||||
return (NOW - dt.timedelta(hours=hours)).isoformat()
|
||||
|
||||
|
||||
class SourceSelection(unittest.TestCase):
|
||||
def test_only_stories_is_a_single_cheap_source(self):
|
||||
srcs = gdl.Profile("u").sources(gdl.STORIES_ONLY)
|
||||
self.assertEqual([s.kind for s in srcs], ["stories"])
|
||||
self.assertEqual(srcs[0].directory, "story - u")
|
||||
|
||||
def test_full_sync_covers_every_surface(self):
|
||||
srcs = gdl.Profile("u").sources(set(gdl.ALL_KINDS))
|
||||
self.assertEqual([s.kind for s in srcs], list(gdl.ALL_KINDS))
|
||||
|
||||
def test_reels_and_stories_go_to_their_own_directories(self):
|
||||
by_kind = {s.kind: s for s in gdl.Profile("u").sources(set(gdl.ALL_KINDS))}
|
||||
self.assertEqual(by_kind["posts"].directory, "u")
|
||||
self.assertEqual(by_kind["reels"].directory, "u - reels")
|
||||
# Highlights derive their directory from the title mid-extraction.
|
||||
self.assertEqual(by_kind["highlights"].directory, "")
|
||||
|
||||
|
||||
class PlanSource(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.state = gdl.SyncState(Path(self.tmp.name) / "state.json")
|
||||
self.posts = gdl.Profile("u").sources({"posts"})[0]
|
||||
self.stories = gdl.Profile("u").sources({"stories"})[0]
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_first_run_seeds(self):
|
||||
fetch, seed, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
|
||||
self.assertTrue(fetch)
|
||||
self.assertTrue(seed)
|
||||
|
||||
def test_seeding_happens_only_once(self):
|
||||
self.state.mark_seeded(self.posts.url, ago(720))
|
||||
fetch, seed, reason = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
|
||||
self.assertTrue(fetch)
|
||||
self.assertFalse(seed, "a seeded source must never be re-probed")
|
||||
self.assertIn("already seeded", reason)
|
||||
|
||||
def test_stories_never_seed(self):
|
||||
# A story cannot be in the archive before it is fetched, so probing
|
||||
# would double the cost of the cheapest surface for no benefit.
|
||||
_, seed, reason = gdl.plan_source(self.stories, self.state, NOW_TS, 20)
|
||||
self.assertFalse(seed)
|
||||
self.assertIn("no seed", reason)
|
||||
|
||||
def test_recent_fetch_is_refused(self):
|
||||
self.state.mark_fetched(self.posts.url, ago(3))
|
||||
fetch, _, reason = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
|
||||
self.assertFalse(fetch)
|
||||
self.assertIn("under the", reason)
|
||||
|
||||
def test_an_old_fetch_is_allowed_again(self):
|
||||
self.state.mark_fetched(self.posts.url, ago(30))
|
||||
fetch, _, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
|
||||
self.assertTrue(fetch)
|
||||
|
||||
def test_daily_stories_pass_a_20h_floor(self):
|
||||
# The cadence this is built for: once a day, every day.
|
||||
self.state.mark_fetched(self.stories.url, ago(24))
|
||||
fetch, _, _ = gdl.plan_source(self.stories, self.state, NOW_TS, 20)
|
||||
self.assertTrue(fetch)
|
||||
|
||||
def test_force_disables_the_floor(self):
|
||||
self.state.mark_fetched(self.posts.url, ago(1))
|
||||
fetch, _, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 0.0)
|
||||
self.assertTrue(fetch)
|
||||
|
||||
def test_the_aborted_run_scenario(self):
|
||||
"""
|
||||
Yesterday's failure: a run died mid-way and the restart re-enumerated
|
||||
every profile. Seeded-but-not-fetched must not re-probe.
|
||||
"""
|
||||
self.state.mark_seeded(self.posts.url, ago(0.5))
|
||||
fetch, seed, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
|
||||
self.assertTrue(fetch, "the fetch still needs to happen")
|
||||
self.assertFalse(seed, "but the listing pass must not be paid for twice")
|
||||
|
||||
|
||||
class StatePersistence(unittest.TestCase):
|
||||
def test_state_survives_a_reload(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
path = Path(d) / "state.json"
|
||||
a = gdl.SyncState(path)
|
||||
a.mark_seeded("https://x/", ago(1))
|
||||
a.mark_fetched("https://x/", ago(1))
|
||||
a.save()
|
||||
b = gdl.SyncState(path)
|
||||
self.assertFalse(b.needs_seed("https://x/"))
|
||||
self.assertEqual(b.last_fetch("https://x/"), ago(1))
|
||||
|
||||
def test_a_corrupt_state_file_never_blocks_a_sync(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
path = Path(d) / "state.json"
|
||||
path.write_text("{ not json")
|
||||
self.assertTrue(gdl.SyncState(path).needs_seed("https://x/"))
|
||||
|
||||
|
||||
class ProbeCaching(unittest.TestCase):
|
||||
def test_fresh_entries_are_reused_and_stale_ones_are_not(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
cache = gdl.ProbeCache(Path(d) / "p.json", ttl_hours=24)
|
||||
cache.put("https://x/", [{"shortcode": "A", "post_shortcode": "A",
|
||||
"num": 1, "media_id": "1"}], ago(1))
|
||||
self.assertEqual(len(cache.get("https://x/", NOW_TS)), 1)
|
||||
|
||||
cache.put("https://y/", [{"shortcode": "B", "post_shortcode": "B",
|
||||
"num": 1, "media_id": "2"}], ago(48))
|
||||
self.assertIsNone(cache.get("https://y/", NOW_TS))
|
||||
|
||||
def test_cache_keeps_only_the_fields_seeding_needs(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
path = Path(d) / "p.json"
|
||||
cache = gdl.ProbeCache(path, ttl_hours=24)
|
||||
cache.put("https://x/", [{"shortcode": "A", "post_shortcode": "A",
|
||||
"num": 1, "media_id": "1",
|
||||
"description": "x" * 5000}], ago(0))
|
||||
cache.save()
|
||||
self.assertNotIn("description", path.read_text())
|
||||
|
||||
def test_a_miss_is_reported_rather_than_guessed(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
cache = gdl.ProbeCache(Path(d) / "p.json", ttl_hours=24)
|
||||
self.assertIsNone(cache.get("https://never-seen/", NOW_TS))
|
||||
|
||||
|
||||
class Seeding(unittest.TestCase):
|
||||
"""The bug that seeded 5 of 2275: matching the wrong shortcode field."""
|
||||
|
||||
def test_posts_are_keyed_by_post_shortcode(self):
|
||||
item = {"shortcode": "childcode", "post_shortcode": "POSTCODE",
|
||||
"num": 2, "media_id": "9"}
|
||||
self.assertEqual(gdl.live_key(item, "posts"), ("POSTCODE", 2))
|
||||
|
||||
def test_stories_are_keyed_by_the_per_item_shortcode(self):
|
||||
item = {"shortcode": "ITEMCODE", "post_shortcode": "reelid",
|
||||
"num": 3, "media_id": "9"}
|
||||
self.assertEqual(gdl.live_key(item, "stories"), ("ITEMCODE", 1))
|
||||
self.assertEqual(gdl.live_key(item, "highlights"), ("ITEMCODE", 1))
|
||||
|
||||
def test_index_existing_normalises_a_missing_index_to_one(self):
|
||||
held = gdl.index_existing([
|
||||
"u/2023-04-19_u - ABC.mp4",
|
||||
"u/2023-04-12_u - DEF - 3.jpg",
|
||||
"u/2023-04-12_u - DEF.txt", # sidecars are not media
|
||||
"u/2023-04-12_u - DEF.json",
|
||||
])
|
||||
self.assertEqual(held, {("ABC", 1), ("DEF", 3)})
|
||||
|
||||
def test_seeding_marks_only_what_is_already_held(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
db = Path(d) / "a.db"
|
||||
live = [
|
||||
{"post_shortcode": "HELD", "shortcode": "x", "num": 1, "media_id": "11"},
|
||||
{"post_shortcode": "NEW", "shortcode": "y", "num": 1, "media_id": "22"},
|
||||
]
|
||||
n = gdl.seed_archive_db(db, {("HELD", 1)}, live, "posts")
|
||||
self.assertEqual(n, 1)
|
||||
import sqlite3
|
||||
rows = {r[0] for r in sqlite3.connect(db).execute(
|
||||
"SELECT entry FROM archive")}
|
||||
self.assertEqual(rows, {"instagram11"})
|
||||
|
||||
|
||||
class Publishing(unittest.TestCase):
|
||||
def test_publish_only_ever_adds(self):
|
||||
cmd = gdl.rsync_command(Path("/stage"), "host:/archives", dry_run=False)
|
||||
self.assertIn("--ignore-existing", cmd)
|
||||
self.assertNotIn("--delete", cmd)
|
||||
|
||||
def test_tooling_files_are_excluded_from_the_archive(self):
|
||||
cmd = " ".join(gdl.rsync_command(Path("/stage"), "/dest", dry_run=True))
|
||||
for pattern in ("gdl-sync*.json", "*.db"):
|
||||
self.assertIn(pattern, cmd)
|
||||
self.assertIn("--dry-run", cmd)
|
||||
|
||||
|
||||
class PostUrl(unittest.TestCase):
|
||||
"""--post-url: a one-off fetch outside the tracked profile list, whose
|
||||
owning account is only known mid-extraction -- same reasoning as
|
||||
highlights, so it must be exempted from the same forced-destination rule."""
|
||||
|
||||
def test_config_keys_a_username_directory(self):
|
||||
config = gdl.build_config("1M", [6.0, 10.0], [3.0, 6.0])
|
||||
for kind in ("post", "reel"):
|
||||
self.assertEqual(
|
||||
config["extractor"]["instagram"][kind]["directory"],
|
||||
["{username}"])
|
||||
|
||||
def test_destination_is_not_forced_like_posts_and_reels(self):
|
||||
staging = Path("/stage")
|
||||
for subcategory in ("post", "reel"):
|
||||
src = gdl.Source(subcategory, "https://www.instagram.com/p/ABC/",
|
||||
"", subcategory)
|
||||
cmd = gdl.gdl_command(src, staging, Path("/cfg.json"), "chrome:x",
|
||||
None)
|
||||
self.assertIn(str(staging), cmd)
|
||||
self.assertNotIn(str(staging / subcategory), cmd)
|
||||
|
||||
|
||||
class UrlsFile(unittest.TestCase):
|
||||
def test_reads_every_form_a_person_might_paste(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
p = Path(d) / "urls.txt"
|
||||
p.write_text(
|
||||
"# comment\n"
|
||||
"https://www.instagram.com/a/\n"
|
||||
"https://instagram.com/b\n"
|
||||
"www.instagram.com/c/\n"
|
||||
"d\n"
|
||||
" e # trailing\n"
|
||||
"\n"
|
||||
"https://www.instagram.com/a/\n" # duplicate
|
||||
"https://www.instagram.com/p/ABC123/\n" # a post, not a profile
|
||||
"not a username\n")
|
||||
self.assertEqual(gdl.read_urls_file(p), ["a", "b", "c", "d", "e"])
|
||||
|
||||
|
||||
class PostUrlsFile(unittest.TestCase):
|
||||
"""The format reels-scrape.py writes: whole URLs, not usernames."""
|
||||
|
||||
def test_skips_comments_blanks_and_duplicates(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
p = Path(d) / "reels.txt"
|
||||
p.write_text(
|
||||
"# scraped 2026-08-27\n"
|
||||
"https://www.instagram.com/u/reel/AAA/\n"
|
||||
"\n"
|
||||
"https://www.instagram.com/u/reel/BBB/\n"
|
||||
"https://www.instagram.com/u/reel/AAA/\n") # duplicate
|
||||
self.assertEqual(gdl.read_post_urls_file(p), [
|
||||
"https://www.instagram.com/u/reel/AAA/",
|
||||
"https://www.instagram.com/u/reel/BBB/",
|
||||
])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -1,40 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Tests for the pure parts of reels-scrape.py -- everything except the actual
|
||||
CDP session, which needs a live signed-in Chrome and is exercised by hand.
|
||||
|
||||
python3 -m unittest discover -s scripts -p 'test_*.py'
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"reels_scrape", Path(__file__).with_name("reels-scrape.py"))
|
||||
scrape = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["reels_scrape"] = scrape
|
||||
_spec.loader.exec_module(scrape)
|
||||
|
||||
|
||||
class ExtractShortcodes(unittest.TestCase):
|
||||
def test_reads_relative_and_absolute_hrefs(self):
|
||||
hrefs = [
|
||||
"/someuser/reel/ABC123/",
|
||||
"https://www.instagram.com/someuser/reel/DEF456/",
|
||||
"/reel/GHI789/?img_index=1",
|
||||
]
|
||||
self.assertEqual(scrape.extract_shortcodes(hrefs),
|
||||
["ABC123", "DEF456", "GHI789"])
|
||||
|
||||
def test_ignores_non_reel_links(self):
|
||||
hrefs = ["/someuser/", "/someuser/p/ABC123/", "/explore/tags/foo/"]
|
||||
self.assertEqual(scrape.extract_shortcodes(hrefs), [])
|
||||
|
||||
def test_empty_input(self):
|
||||
self.assertEqual(scrape.extract_shortcodes([]), [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
+56
-2
@@ -73,6 +73,17 @@ export default function App() {
|
||||
*/
|
||||
const initialRouteRef = useRef(parseRoute(window.location.pathname, window.location.search));
|
||||
|
||||
/**
|
||||
* Back-button support for the post view. Every other URL change
|
||||
* (`replaceState`s the tab/archive) is intentionally NOT pushed — only
|
||||
* opening a post gets its own history entry, matching Instagram's own
|
||||
* back-button behaviour: Back closes the post instead of leaving the app.
|
||||
*/
|
||||
const pushedPostRef = useRef(false);
|
||||
/** Set while reacting to a popstate, so the URL-sync effect below does not
|
||||
* try to push/replace/back() again for a change the browser already made. */
|
||||
const suppressNextSyncRef = useRef(false);
|
||||
|
||||
const isMobile = useIsMobile();
|
||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
||||
const profilePicInputRef = useRef<HTMLInputElement>(null);
|
||||
@@ -367,6 +378,13 @@ export default function App() {
|
||||
// loader below is waiting to read.
|
||||
if (!hasInitialLoaded) return;
|
||||
|
||||
// Consumed exactly once per popstate, regardless of what happens below:
|
||||
// a popstate-driven change often already matches the URL (the browser
|
||||
// already moved the pointer), which used to leave this flag stuck true
|
||||
// and silently no-op the NEXT real close/open until something reset it.
|
||||
const wasPopState = suppressNextSyncRef.current;
|
||||
suppressNextSyncRef.current = false;
|
||||
|
||||
const archive = currentArchive?.name ?? (allPosts.length > 0 ? username : null) ?? null;
|
||||
const nextPath = buildPath({
|
||||
archive,
|
||||
@@ -374,12 +392,48 @@ export default function App() {
|
||||
post: selectedPost ? postSlug(selectedPost) : null,
|
||||
});
|
||||
|
||||
if (nextPath !== window.location.pathname + window.location.search) {
|
||||
console.log(`[Permalink] Updating URL to: ${nextPath}`);
|
||||
if (nextPath === window.location.pathname + window.location.search) return;
|
||||
if (wasPopState) return; // the browser already navigated; nothing to add
|
||||
|
||||
console.log(`[Permalink] Updating URL to: ${nextPath}`);
|
||||
|
||||
if (selectedPost && !pushedPostRef.current) {
|
||||
// Opening a post: push, so Back closes it instead of leaving the app.
|
||||
window.history.pushState(null, '', nextPath);
|
||||
pushedPostRef.current = true;
|
||||
} else if (!selectedPost && pushedPostRef.current) {
|
||||
// Closing a post that was pushed for: consume that entry rather than
|
||||
// piling a new one on top of it, so Back still means "one step".
|
||||
pushedPostRef.current = false;
|
||||
window.history.back();
|
||||
} else {
|
||||
window.history.replaceState(null, '', nextPath);
|
||||
}
|
||||
}, [hasInitialLoaded, currentArchive?.name, username, allPosts.length, activeTab, selectedPost?.id]);
|
||||
|
||||
/**
|
||||
* Back/forward support for the post view. Only a post push (above) ever
|
||||
* creates an entry, so this only ever needs to open or close a post —
|
||||
* never re-derive the tab or archive, which stayed on replaceState.
|
||||
*/
|
||||
useEffect(() => {
|
||||
const onPopState = () => {
|
||||
suppressNextSyncRef.current = true;
|
||||
const route = parseRoute(window.location.pathname, window.location.search);
|
||||
const post = route.post ? findPostBySlug(allPosts, route.post) : null;
|
||||
if (post) {
|
||||
setActiveTab(tabForSource(post.source));
|
||||
setSelectedPost(post);
|
||||
pushedPostRef.current = true; // forward navigation can land back here
|
||||
} else {
|
||||
setSelectedPost(null);
|
||||
pushedPostRef.current = false;
|
||||
}
|
||||
};
|
||||
window.addEventListener('popstate', onPopState);
|
||||
return () => window.removeEventListener('popstate', onPopState);
|
||||
}, [allPosts]);
|
||||
|
||||
useEffect(() => {
|
||||
if (hasInitialLoaded) return;
|
||||
|
||||
|
||||
Reference in New Issue
Block a user