ergosteur/instaarchive-viewer
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

tooling: instagram-chrome-scrape.py (was reels-scrape.py) gains --tab posts with early stop; chrome-sync.sh + reels/posts-sync.sh wrappers; MANUAL-RUN.md; atomic-deploy note

Browse Source
This commit is contained in:
ergosteur
2026-10-22 23:53:35 -04:00
parent 30c9fbe611
commit 1b67765d8b
10 changed files with 457 additions and 193 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TOOLING.md
+46 -7
View File
@@ -23,6 +23,9 @@
## Quick reference — getting new content onto the site
For the whole thing by hand, step by step (including the DH2300 / `cosmo_web` hop), see
[MANUAL-RUN.md](MANUAL-RUN.md).
The three systemd timers (see "Scheduling" below) run these unattended; run
them by hand only to get content sooner. Everything runs on `mattellite`; it
fetches, publishes to the NAS itself, and the site picks it up with no deploy
@@ -290,8 +293,11 @@ restored, `posts` listings for `heejin`/`kimxxlip` returned `No results` in ~3 s
profile skips; it does not say that), while stories worked and the posts opened fine in the
browser. The host was on gallery-dl 1.32.14; 1.32.17 (2026-10-17) lists "instagram: update
`user` & `posts` query parameters". After upgrading, the profile lookup failed instead
(`KeyError: 'data'`, `Requested user could not be found`) -- **unresolved when written**, see
the end of this section.
(`KeyError: 'data'`, `Requested user could not be found`). That is upstream
[gh#9829](https://github.com/mikf/gallery-dl/issues/9829) -- Instagram changed the query again
on ~10-21; a fix is merged upstream but **not in a release** as of 10-22. Single-post fetches
(`/p/<code>/`) still work, so the gap is the *listing*, which the browser covers -- see
"Posts grid by browser" below.
`scripts/gdl-upgrade.sh` (`pipx upgrade --include-injected gallery-dl`, so yt-dlp too) now runs:
@@ -303,6 +309,35 @@ the end of this section.
It costs no Instagram requests. The trade-off: a bad upstream release is picked up
automatically; roll back with `pipx install --force gallery-dl==<old>` and re-inject yt-dlp.
### Posts grid by browser: `posts-sync.sh`
The reels workaround from August, generalised. `reels-scrape.py` is now
**`instagram-chrome-scrape.py`** with `--tab reels|posts`, and `reels-sync.sh`'s body is now
**`chrome-sync.sh`** (`GDL_TAB=reels|posts`); `reels-sync.sh` and the new `posts-sync.sh` are
two-line wrappers that set the tab, so every older command still works.
```sh
ssh mattellite
~/gdl/posts-sync.sh heejin # scroll /heejin/, fetch new /p/ links as single posts
```
`--tab posts` scrolls the main grid (`/<user>/`), collects `/p/` and `/reel/` links, and dedupes
against the whole archive like the reels tab does. Unlike reels it **stops early**: the grid is
newest-first, so after 2 scroll rounds in a row that turned up nothing unarchived
(`--max-known-rounds`, round 1 exempt because pinned posts sit on top) there is nothing newer to
find. First run, `heejin`: 3 scrolls, 36 posts seen, 30 already held, 6 new.
Same caveats as reels: it drives the signed-in Chrome on `:9222`, so it is by hand, not on a
timer. Once a gallery-dl release fixes the listing, `run-profiles.sh` covers posts again and
this goes back to being a fallback.
**Deploy scripts atomically, never in place.** bash reads a script as it runs, so overwriting
`gdl-cron.sh` under a running sync makes bash resume at the same byte offset in the *new* text.
On 10-22 an `scp` over it mid-run made the wrapper die after the sync with
`line 163: status: unbound variable`. The fetch and publish had finished, but the exit line and
the Healthchecks ping were skipped. Copy to a temp name and `mv` it into place (a new inode; the
running bash keeps the old one), or check `pgrep -af gdl-` first.
**A `--min-interval` trap:** a run that returns `No results` still counts as a fetch, so the
next run skips the source for 20 h. Re-run after a bad result with `--min-interval 0`
(not `--force`, which also drops the probe cache).
@@ -339,6 +374,9 @@ every deletion because of it.
| path | what it is |
|---|---|
| `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
| `scripts/gdl-upgrade.sh` | `pipx upgrade` gallery-dl + yt-dlp; run by `gdl-cron.sh` and `gdl-upgrade.timer` |
| `scripts/instagram-chrome-scrape.py` | list a profile's reels tab or posts grid by scrolling the signed-in Chrome over CDP |
| `scripts/chrome-sync.sh` | scrape + fetch new posts for one profile; `reels-sync.sh` / `posts-sync.sh` set the tab |
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
| `scripts/systemd/` | the timers actually installed on the fetch host |
| `scripts/test_gdl_sync.py` | its tests |
@@ -746,7 +784,7 @@ before something worse.
changelog cites "posts & reels extractors... content retrieval failures", and
upgrading (`pipx upgrade gallery-dl`) confirmed it -- a full `run-profiles.sh`
across all six profiles came back with **zero** `HTTP redirect` errors on
posts or reels, first time since 2026-08-26. `reels-scrape.py`/`reels-sync.sh`/
posts or reels, first time since 2026-08-26. `reels-scrape.py` (now `instagram-chrome-scrape.py`)/`reels-sync.sh`/
`run-reels-all.sh` are no longer needed for routine use; left in place as a
fallback (and because the whole-archive dedup fix they picked up along the
way, see below, is independently useful). If this regresses, check the
@@ -768,7 +806,8 @@ Posts, highlights and stories were unaffected at the time; only reels broke
(posts later broke too, some time before 2026-09-13, in the same way — same
fix resolved both).
`scripts/reels-scrape.py` works around it by never calling that endpoint:
`scripts/reels-scrape.py` (renamed `instagram-chrome-scrape.py` on 2026-10-22, when it gained
`--tab posts`) works around it by never calling that endpoint:
it drives the same signed-in Chrome via its loopback CDP port (`:9222`,
already exposed for MCP automation), scrolls the reels tab like a person
would, and scrapes `/reel/<code>/` links out of the rendered page. It only
@@ -841,7 +880,7 @@ PROFILE=someuser
# 1. Scrape by scrolling; dedupe against the archive; write new URLs to a file.
# Needs gallery-dl's own pipx venv python -- that's where websocket-client
# (the one extra dependency this needs) got injected.
~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/reels-scrape.py \
~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/instagram-chrome-scrape.py \
--profile "$PROFILE" \
--index https://instaarchive.ergosteur.com \
--out ~/gdl/"$PROFILE"-reels.txt
@@ -857,7 +896,7 @@ cd ~/gdl && PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
Notes:
- `reels-scrape.py` fails fast if Chrome/CDP isn't up
- `instagram-chrome-scrape.py` fails fast if Chrome/CDP isn't up
(`curl -s http://localhost:9222/json/version` to check first).
- If it finds nothing new, `gdl-sync.py --post-urls-file` refuses to run
("no usable URLs") rather than doing nothing quietly — expected when a
@@ -880,7 +919,7 @@ them, filed them correctly under `{username}` (the post's *true* owner, per
its own metadata) — where they already existed from that profile's own
regular sync — and `rsync --ignore-existing` silently skipped every one, so
nothing was lost or duplicated. But 9 Instagram requests were spent finding
that out. Since a shortcode is globally unique, `reels-scrape.py` now checks
that out. Since a shortcode is globally unique, the scraper now checks
every archived profile's listing, not just the one being scraped — all local
requests to the viewer's own API, never to `instagram.com`, so checking all
16 profiles costs nothing on the budget that actually matters. Confirmed