tooling: scheduled profiles/full-sweep list posts+reels by browser (chrome-sync per profile/tab, early stop for weekly); GDL_LISTING=api fallback; reels-tab sweep note in MANUAL-RUN

MANUAL-RUN.md

@@ -0,0 +1,109 @@
+# Manual run — scrape, archive, push to the NAS
+
+**`tooling` branch only. Never push this to GitHub.** Like `TOOLING.md`, it names the fetch host,
+the NAS hosts and their paths.
+
+How to do by hand everything the timers normally do, plus the second NAS hop that feeds
+`cosmo_web`. `TOOLING.md` has the details behind each step.
+
+## Where things go
+
+1. **mattellite → ana-nas.** Every scrape script ends by rsyncing its new files to
+   `ana-nas:/volume1/rslsync/sync/Instagram-archive/archives/` (10.20.28.200).
+   <https://instaarchive.ergosteur.com> reads straight from there. **There is no separate step
+   for this hop.**
+2. **ana-nas → DH2300.** The `archiver-instagram` container on `DH2300-D7CA` pulls that archive,
+   makes thumbnails, remuxes video, normalizes into `cosmo_web`'s data and translates captions.
+   It runs weekly (Wednesday 04:00). Step 6 below runs it by hand.
+
+## 1. Pre-flight (mattellite)
+
+```sh
+ssh matt@mattellite
+~/gdl/check-session.py                                   # want: "session ok ... expires <date>"
+curl -s http://localhost:9222/json/version | head -2     # Chrome DevTools port is up
+~/gdl/gdl-upgrade.sh                                     # gallery-dl + yt-dlp to latest
+```
+
+If the session check fails, log in to Instagram in the VNC Chrome (display `:1`), **reload the
+tab** (a dead session can still look logged in) and re-check.
+
+## 2. Stories — first; they expire in 24 h
+
+```sh
+~/gdl/run-stories.sh
+```
+
+About 2 minutes. `rsync ... (code 23)` / `failed to set permissions` is expected: the script
+says "files landed" when that is the only problem. `No results` for a profile just means it has
+no active story.
+
+## 3. Posts and reels
+
+**While gallery-dl's listing is broken** ([gh#9829](https://github.com/mikf/gallery-dl/issues/9829),
+from 2026-10-22), use the browser scraper, one profile at a time:
+
+```sh
+for p in heejin kimxxlip; do ~/gdl/posts-sync.sh $p; done
+for p in withaseul cher_ryppo zindoriyam; do ~/gdl/posts-sync.sh $p; done
+~/gdl/posts-sync.sh official_artms
+```
+
+- `posts-sync.sh` scrolls `/<user>/` in the signed-in Chrome, stops once it reaches posts the
+  archive already holds, and fetches only the new ones as single posts. Each profile takes a few
+  minutes, mostly the 12–20 s gap between fetches.
+- It prints `no new posts for <user>` when a profile is caught up.
+- Logs: `~/gdl/logs/posts-<user>-*.log`.
+- **Then sweep the Reels tab too:** `~/gdl/run-reels-all.sh` (~15 min). Most reels never
+  appear in the main grid. On 2026-10-23 only 3 of heejin's 114 reels did, so `posts-sync.sh`
+  alone misses them. `reels-sync.sh <user>` does a single profile.
+
+**Or the whole thing in one command**, the same run the weekly timer does: gallery-dl for
+stories and highlights, then the browser for every profile's posts grid and Reels tab, each
+stopping once it reaches archived content:
+
+```sh
+~/gdl/run-profiles.sh
+```
+
+Once a gallery-dl release fixes the listing, `GDL_LISTING=api ~/gdl/run-profiles.sh` does posts
+and reels through gallery-dl again (~5 min).
+
+For long runs, detach:
+`setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &`, then
+`tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`.
+
+**Stop at once** on a `429` from `scontent-*.cdninstagram.com` or a `400 Bad Request` on
+`/api/v1/feed/reels_media/`. See "When a run fails" in `TOOLING.md`.
+
+## 4. Highlights
+
+These only come through `run-profiles.sh`. Expect them to fail with the same
+`Requested user could not be found` until gh#9829 is fixed in a release, so skip them until then.
+
+## 5. Check it landed (any machine)
+
+```sh
+for u in heejin kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
+  n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
+  printf '%-16s %s\n' "$u" "$n"
+done
+```
+
+Or browse <https://instaarchive.ergosteur.com>. New files appear without a redeploy.
+
+## 6. Push to `cosmo_web` (the DH2300 hop)
+
+This runs the same command the container's cron line runs (from `docker/archiver-instagram-entrypoint.sh`
+in Cosmo-Live-Downloads):
+
+```sh
+ssh matt@DH2300-D7CA
+docker exec cosmo-web-archiver-instagram-1 sh -c \
+  'cd /work && . /app/instagram-archiver.env && python3 scripts/instagram_archive_run.py'
+docker logs --tail 50 cosmo-web-archiver-instagram-1
+```
+
+It pings that container's Healthchecks check, same as a scheduled run. The pipeline itself
+(mirror, thumbnail, remux, normalize, collaborators, translate) is described in the docstring of
+`scripts/instagram_archive_run.py` in Cosmo-Live-Downloads.

TOOLING.md

@@ -23,6 +23,9 @@
 
 ## Quick reference — getting new content onto the site
 
+For the whole thing by hand, step by step (including the DH2300 / `cosmo_web` hop), see
+[MANUAL-RUN.md](MANUAL-RUN.md).
+
 The three systemd timers (see "Scheduling" below) run these unattended; run
 them by hand only to get content sooner. Everything runs on `mattellite`; it
 fetches, publishes to the NAS itself, and the site picks it up with no deploy
@@ -283,6 +286,68 @@ loudly, verified with both a real captured rsync failure and unit tests.
 The other 22 release tags stay on gitea. Pushing them all to GitHub triggers
 one container build per tag, because each tag carries its own workflow file.
 
+## 2026-10-22 — gallery-dl auto-upgrade; posts listing went empty
+
+Scheduled and manual runs were failing quietly from early October: after the session was
+restored, `posts` listings for `heejin`/`kimxxlip` returned `No results` in ~3 s (a caught-up
+profile skips; it does not say that), while stories worked and the posts opened fine in the
+browser. The host was on gallery-dl 1.32.14; 1.32.17 (2026-10-17) lists "instagram: update
+`user` & `posts` query parameters". After upgrading, the profile lookup failed instead
+(`KeyError: 'data'`, `Requested user could not be found`). That is upstream
+[gh#9829](https://github.com/mikf/gallery-dl/issues/9829) -- Instagram changed the query again
+on ~10-21; a fix is merged upstream but **not in a release** as of 10-22. Single-post fetches
+(`/p/<code>/`) still work, so the gap is the *listing*, which the browser covers -- see
+"Posts grid by browser" below.
+
+`scripts/gdl-upgrade.sh` (`pipx upgrade --include-injected gallery-dl`, so yt-dlp too) now runs:
+
+- at the start of every `gdl-cron.sh` run, after the session preflight and before any request;
+  logs `upgrade: up to date` or `upgrade: CHANGED a -> b`, and never fails the run;
+- weekly from `gdl-upgrade.timer` (systemd user timer, installed on `mattellite` with the
+  others -- systemd rather than cron, to match the rest of the host).
+
+It costs no Instagram requests. The trade-off: a bad upstream release is picked up
+automatically; roll back with `pipx install --force gallery-dl==<old>` and re-inject yt-dlp.
+
+### Posts grid by browser: `posts-sync.sh`
+
+The reels workaround from August, generalised. `reels-scrape.py` is now
+**`instagram-chrome-scrape.py`** with `--tab reels|posts`, and `reels-sync.sh`'s body is now
+**`chrome-sync.sh`** (`GDL_TAB=reels|posts`); `reels-sync.sh` and the new `posts-sync.sh` are
+two-line wrappers that set the tab, so every older command still works.
+
+```sh
+ssh mattellite
+~/gdl/posts-sync.sh heejin          # scroll /heejin/, fetch new /p/ links as single posts
+```
+
+`--tab posts` scrolls the main grid (`/<user>/`), collects `/p/` and `/reel/` links, and dedupes
+against the whole archive like the reels tab does. Unlike reels it **stops early**: the grid is
+newest-first, so after 2 scroll rounds in a row that turned up nothing unarchived
+(`--max-known-rounds`, round 1 exempt because pinned posts sit on top) there is nothing newer to
+find. First run, `heejin`: 3 scrolls, 36 posts seen, 30 already held, 6 new.
+
+**Scheduled since 2026-10-23.** `gdl-cron.sh profiles` and `full-sweep` now list posts *and*
+reels this way: gallery-dl only does `--only stories,highlights`, then `chrome-sync.sh` runs per
+profile for the `posts` grid and the `reels` tab (both tabs needed: on 10-23 only 3 of heejin's
+114 reels were in the main grid). `profiles` uses the early stop (2 known rounds) on both tabs,
+and `full-sweep` scrolls every grid to the bottom. A failed browser step fails the run, which
+shows on Healthchecks. This reverses the earlier "by hand only" decision on purpose: the API
+listing is broken and the owner chose the browser. The trade-off is that unattended runs now
+need Chrome up on `:9222` with a live session, and they open and close their own tabs in it.
+`GDL_LISTING=api` restores the all-gallery-dl run once a release fixes gh#9829.
+
+**Deploy scripts atomically, never in place.** bash reads a script as it runs, so overwriting
+`gdl-cron.sh` under a running sync makes bash resume at the same byte offset in the *new* text.
+On 10-22 an `scp` over it mid-run made the wrapper die after the sync with
+`line 163: status: unbound variable`. The fetch and publish had finished, but the exit line and
+the Healthchecks ping were skipped. Copy to a temp name and `mv` it into place (a new inode; the
+running bash keeps the old one), or check `pgrep -af gdl-` first.
+
+**A `--min-interval` trap:** a run that returns `No results` still counts as a fetch, so the
+next run skips the source for 20 h. Re-run after a bad result with `--min-interval 0`
+(not `--force`, which also drops the probe cache).
+
 ## Guards — recreate these after a fresh clone
 
 Neither guard is versioned, so a new clone has **no protection at all**:
@@ -315,6 +380,9 @@ every deletion because of it.
 | path | what it is |
 |---|---|
 | `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
+| `scripts/gdl-upgrade.sh` | `pipx upgrade` gallery-dl + yt-dlp; run by `gdl-cron.sh` and `gdl-upgrade.timer` |
+| `scripts/instagram-chrome-scrape.py` | list a profile's reels tab or posts grid by scrolling the signed-in Chrome over CDP |
+| `scripts/chrome-sync.sh` | scrape + fetch new posts for one profile; `reels-sync.sh` / `posts-sync.sh` set the tab |
 | `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
 | `scripts/systemd/` | the timers actually installed on the fetch host |
 | `scripts/test_gdl_sync.py` | its tests |
@@ -722,7 +790,7 @@ before something worse.
 changelog cites "posts & reels extractors... content retrieval failures", and
 upgrading (`pipx upgrade gallery-dl`) confirmed it -- a full `run-profiles.sh`
 across all six profiles came back with **zero** `HTTP redirect` errors on
-posts or reels, first time since 2026-08-26. `reels-scrape.py`/`reels-sync.sh`/
+posts or reels, first time since 2026-08-26. `reels-scrape.py` (now `instagram-chrome-scrape.py`)/`reels-sync.sh`/
 `run-reels-all.sh` are no longer needed for routine use; left in place as a
 fallback (and because the whole-archive dedup fix they picked up along the
 way, see below, is independently useful). If this regresses, check the
@@ -744,7 +812,8 @@ Posts, highlights and stories were unaffected at the time; only reels broke
 (posts later broke too, some time before 2026-09-13, in the same way — same
 fix resolved both).
 
-`scripts/reels-scrape.py` works around it by never calling that endpoint:
+`scripts/reels-scrape.py` (renamed `instagram-chrome-scrape.py` on 2026-10-22, when it gained
+`--tab posts`) works around it by never calling that endpoint:
 it drives the same signed-in Chrome via its loopback CDP port (`:9222`,
 already exposed for MCP automation), scrolls the reels tab like a person
 would, and scrapes `/reel/<code>/` links out of the rendered page. It only
@@ -773,11 +842,11 @@ after — left behind for inspection, same as `gdl-cron.sh`'s `staging-*`.
 Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the
 page, 10 new ones fetched and published cleanly on the first run.
 
-**Not wired into `gdl-cron.sh` or the timers.** Both scripts drive your
+**Not wired into `gdl-cron.sh` or the timers** (as of 2026-08; **superseded 2026-10-23**: the
-actual browser session rather than firing a background request, and are
+weekly and quarterly runs now use the browser listing, see "Posts grid by browser" above). Both
-slower by design (real scrolling, not an API call) — both good reasons not
+scripts drive your actual browser session rather than firing a background request, and are
-to run this unattended without deciding that deliberately. Today it is a
+slower by design (real scrolling, not an API call), both good reasons not to run this
-per-profile, by-hand tool only.
+unattended without deciding that deliberately.
 
 For all six ARTMS profiles in one go, `~/gdl/run-reels-all.sh` just calls
 `reels-sync.sh` once per profile in `artms_account_links.txt`, same
@@ -817,7 +886,7 @@ PROFILE=someuser
 # 1. Scrape by scrolling; dedupe against the archive; write new URLs to a file.
 #    Needs gallery-dl's own pipx venv python -- that's where websocket-client
 #    (the one extra dependency this needs) got injected.
-~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/reels-scrape.py \
+~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/instagram-chrome-scrape.py \
   --profile "$PROFILE" \
   --index https://instaarchive.ergosteur.com \
   --out ~/gdl/"$PROFILE"-reels.txt
@@ -833,7 +902,7 @@ cd ~/gdl && PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
 
 Notes:
 
-- `reels-scrape.py` fails fast if Chrome/CDP isn't up
+- `instagram-chrome-scrape.py` fails fast if Chrome/CDP isn't up
   (`curl -s http://localhost:9222/json/version` to check first).
 - If it finds nothing new, `gdl-sync.py --post-urls-file` refuses to run
   ("no usable URLs") rather than doing nothing quietly — expected when a
@@ -856,7 +925,7 @@ them, filed them correctly under `{username}` (the post's *true* owner, per
 its own metadata) — where they already existed from that profile's own
 regular sync — and `rsync --ignore-existing` silently skipped every one, so
 nothing was lost or duplicated. But 9 Instagram requests were spent finding
-that out. Since a shortcode is globally unique, `reels-scrape.py` now checks
+that out. Since a shortcode is globally unique, the scraper now checks
 every archived profile's listing, not just the one being scraped — all local
 requests to the viewer's own API, never to `instagram.com`, so checking all
 16 profiles costs nothing on the budget that actually matters. Confirmed

scripts/chrome-sync.sh

@@ -0,0 +1,120 @@
+#!/bin/bash
+# The full scrape-then-fetch pipeline for one profile: scroll the real page in
+# the signed-in Chrome (instagram-chrome-scrape.py), then fetch and publish
+# whatever's new as single posts. See TOOLING.md ("Reels: the API is blocked,
+# scrape by scrolling instead", and 2026-10-22 for the posts grid) for why this
+# exists at all -- it drives the actual signed-in browser session instead of
+# the listing API, and is slower by design.
+#
+# Usage: GDL_TAB=reels|posts ./chrome-sync.sh <username-or-profile-url>
+# Normally called through reels-sync.sh / posts-sync.sh, which set GDL_TAB.
+#
+# Also run by gdl-cron.sh (profiles, full-sweep) once per profile and tab since
+# 2026-10-23, with GDL_MAX_KNOWN_ROUNDS set -- see TOOLING.md "Posts grid by browser".
+# Exits non-zero if either step does. Everything is logged.
+set -eu
+
+RAW="${1:?usage: GDL_TAB=reels|posts chrome-sync.sh <username-or-profile-url>}"
+TAB="${GDL_TAB:?set GDL_TAB=reels or posts (or use reels-sync.sh / posts-sync.sh)}"
+
+GDL_HOME="${GDL_HOME:-$HOME/gdl}"
+GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
+INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
+PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
+
+# Same hand-paced pacing gdl-cron.sh uses -- see its comment for why. Override
+# per-run with GDL_SLEEP_REQUEST etc. if you ever need to, but raise them
+# rather than lower them.
+SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
+SLEEP="${GDL_SLEEP:-5 10}"
+RATE="${GDL_RATE:-500K}"
+SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}"
+MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}"
+# Empty = instagram-chrome-scrape.py's own default (2 for posts, 0 = scroll to the end for reels).
+MAX_KNOWN_ROUNDS="${GDL_MAX_KNOWN_ROUNDS:-}"
+
+PATH="$HOME/.local/bin:$PATH"; export PATH
+
+if [ ! -x "$GDL_PYTHON" ]; then
+  echo "instagram-chrome-scrape.py needs gallery-dl's own pipx venv python (websocket-client" >&2
+  echo "was injected there, not into the system python): $GDL_PYTHON not found" >&2
+  exit 2
+fi
+
+# Same username-from-URL parsing gdl-sync.py already does for --urls-file,
+# reused rather than re-implemented so the two never drift apart.
+PROFILE=$("$GDL_PYTHON" -c "
+import re, sys, importlib.util
+from pathlib import Path
+spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
+gdl = importlib.util.module_from_spec(spec)
+sys.modules['gdl_sync'] = gdl
+spec.loader.exec_module(gdl)
+raw = '$RAW'
+m = gdl.RE_PROFILE_URL.match(raw)
+user = m.group('user') if m else raw.strip('/')
+if not user or '/' in user or ' ' in user:
+    print(f'cannot read a username from {raw!r}', file=sys.stderr)
+    sys.exit(1)
+print(user)
+")
+
+mkdir -p "$GDL_HOME/logs"
+LOG="$GDL_HOME/logs/$TAB-$PROFILE-$(date +%Y%m%d-%H%M%S)-$$.log"
+URLS_FILE="$GDL_HOME/$PROFILE-$TAB.txt"
+STAGING="$GDL_HOME/staging-$TAB-$PROFILE"
+
+echo "=== $TAB-sync $PROFILE $(date -Is) ===" | tee -a "$LOG"
+
+# The exit status has to survive the pipe into tee -- see gdl-cron.sh's
+# comment on PIPESTATUS for why this needs to be bash, not sh.
+set +e
+"$GDL_PYTHON" "$GDL_HOME/instagram-chrome-scrape.py" \
+    --profile "$PROFILE" \
+    --tab "$TAB" \
+    --index "$INDEX" \
+    --scroll-pause $SCROLL_PAUSE \
+    --max-idle-rounds "$MAX_IDLE_ROUNDS" \
+    ${MAX_KNOWN_ROUNDS:+--max-known-rounds "$MAX_KNOWN_ROUNDS"} \
+    --out "$URLS_FILE" 2>&1 | tee -a "$LOG"
+scrape_status=${PIPESTATUS[0]}
+set -e
+
+if [ "$scrape_status" -ne 0 ]; then
+  echo "=== exit $scrape_status (scrape failed) at $(date -Is) ===" | tee -a "$LOG"
+  exit "$scrape_status"
+fi
+
+if [ ! -s "$URLS_FILE" ]; then
+  echo "no new $TAB for $PROFILE; nothing to fetch" | tee -a "$LOG"
+  echo "=== exit 0 at $(date -Is) ===" | tee -a "$LOG"
+  exit 0
+fi
+
+# Staging is wiped every run on purpose -- same reasoning as gdl-cron.sh: what
+# we already hold is decided by the archive dedupe in instagram-chrome-scrape.py, not by
+# what happens to be sitting in staging.
+rm -rf "$STAGING"
+
+set +e
+# shellcheck disable=SC2086
+"$GDL_HOME/gdl-sync.py" \
+    --publish "$PUBLISH" \
+    --staging "$STAGING" \
+    --post-urls-file "$URLS_FILE" \
+    --sleep-request $SLEEP_REQUEST \
+    --sleep $SLEEP \
+    --rate "$RATE" \
+    --execute 2>&1 | tee -a "$LOG"
+status=${PIPESTATUS[0]}
+set -e
+
+echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
+
+# Keep the log directory from growing without bound -- scoped to this
+# script's own logs so it never touches gdl-cron.sh's rotation.
+ls -1t "$GDL_HOME/logs" | grep "^$TAB-[^0-9]" | tail -n +30 | while read -r old; do
+  rm -f "$GDL_HOME/logs/$old"
+done
+
+exit "$status"

scripts/gdl-cron.sh

@@ -10,6 +10,11 @@
 #                          EDITED after we archived them, and by far the most
 #                          expensive thing here -- see TOOLING.md before running.
 #
+# Posts and reels are LISTED by scrolling the signed-in Chrome (chrome-sync.sh,
+# per profile and tab) rather than by gallery-dl's listing API, which broke on
+# 2026-10-22 (gh#9829) while single-post fetches kept working. gallery-dl still
+# does stories and highlights. GDL_LISTING=api restores the old all-API run.
+#
 # Exits non-zero if the sync does, so cron mails you. Everything is logged.
 set -eu
 
@@ -41,6 +46,15 @@ SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
 SLEEP="${GDL_SLEEP:-5 10}"
 RATE="${GDL_RATE:-500K}"
 
+LISTING="${GDL_LISTING:-browser}"
+case "$LISTING" in
+  browser|api) ;;
+  *) echo "GDL_LISTING must be browser or api, not '$LISTING'" >&2; exit 2 ;;
+esac
+GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
+# Set per mode below when posts/reels go through the browser: empty = no browser step.
+BROWSER_KNOWN_ROUNDS=""
+
 case "$MODE" in
   # --min-interval 8, not the 20h default. The floor exists to stop an ABORTED
   # RESTART re-enumerating profiles -- a minutes-to-hours concern. At 20h the
@@ -50,10 +64,23 @@ case "$MODE" in
   # request per profile, so the worst case 8h permits is roughly twelve
   # requests in a day instead of six.
   stories)    ARGS="--only stories --min-interval 8" ;;
-  profiles)   ARGS="--only posts,reels,stories,highlights --abort 50" ;;
+  # Browser listing: stop scrolling a grid after 2 rounds of nothing unarchived,
+  # the browser's equivalent of --abort 50.
+  profiles)
+    if [ "$LISTING" = browser ]; then
+      ARGS="--only stories,highlights --abort 50"; BROWSER_KNOWN_ROUNDS=2
+    else
+      ARGS="--only posts,reels,stories,highlights --abort 50"
+    fi ;;
   # No --abort: the whole point of full-sweep is enumerating to the end, so it
-  # is the only run that notices carousels edited after we archived them.
+  # is the only run that notices carousels edited after we archived them. The
-  full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
+  # browser equivalent is scrolling every grid to the bottom (0 = no early stop).
+  full-sweep)
+    if [ "$LISTING" = browser ]; then
+      ARGS="--only stories,highlights"; BROWSER_KNOWN_ROUNDS=0
+    else
+      ARGS="--only posts,reels,stories,highlights"
+    fi ;;
   # Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
   # "sweep" did not say it was the exhaustive one. Catch the old names rather
   # than failing with a bare error, in case something still passes them.
@@ -84,6 +111,14 @@ if [ -x "$SESSION_CHECK" ]; then
   echo "preflight: $SESSION_MSG" | tee -a "$LOG"
 fi
 
+# Upgrade gallery-dl/yt-dlp first (PyPI only, no Instagram requests; never fatal). Instagram
+# endpoints change and gallery-dl follows within days -- on 1.32.14 the posts listing silently
+# returned "No results" after 1.32.17 had already fixed the query. See gdl-upgrade.sh.
+UPGRADE="$GDL_HOME/gdl-upgrade.sh"
+if [ -x "$UPGRADE" ]; then
+  "$UPGRADE" 2>&1 | tee -a "$LOG" || true
+fi
+
 # Staging is wiped every run ON PURPOSE. What we already hold is decided by the
 # skip-archive (--download-archive), never by which files happen to be sitting
 # in staging, so starting empty is correct -- and it keeps the publish rsync
@@ -117,6 +152,39 @@ set +e
 status=${PIPESTATUS[0]}
 set -e
 
+# Posts and reels by browser: scrape each profile's grid and Reels tab, fetch only
+# the new shortcodes as single posts. chrome-sync.sh logs and publishes on its own;
+# its output is copied into this run's log so the Healthchecks body shows it too.
+if [ -n "$BROWSER_KNOWN_ROUNDS" ]; then
+  mapfile -t PROFILES < <("$GDL_PYTHON" -c "
+import sys, importlib.util
+from pathlib import Path
+spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
+gdl = importlib.util.module_from_spec(spec)
+sys.modules['gdl_sync'] = gdl
+spec.loader.exec_module(gdl)
+for p in gdl.read_urls_file(Path('$GDL_HOME/artms_account_links.txt')):
+    print(p)
+")
+  browser_failed=0
+  for p in "${PROFILES[@]}"; do
+    for tab in posts reels; do
+      echo "==> browser: $p / $tab" | tee -a "$LOG"
+      set +e
+      GDL_TAB=$tab GDL_MAX_KNOWN_ROUNDS=$BROWSER_KNOWN_ROUNDS \
+        "$GDL_HOME/chrome-sync.sh" "$p" 2>&1 | tee -a "$LOG" >/dev/null
+      s=${PIPESTATUS[0]}
+      set -e
+      if [ "$s" -ne 0 ]; then
+        echo "    browser $p / $tab FAILED (exit $s)" | tee -a "$LOG" >&2
+        browser_failed=$((browser_failed + 1))
+      fi
+    done
+  done
+  echo "browser listing: ${#PROFILES[@]} profile(s), $browser_failed step(s) failed" | tee -a "$LOG"
+  [ "$browser_failed" -gt 0 ] && [ "$status" -eq 0 ] && status=1
+fi
+
 # A stories run that skipped every source is NOT a success. It means the
 # min-interval floor blocked the one surface that cannot be backfilled, and
 # without this it looks identical to a clean run: exit 0, "0 step(s) failed".

scripts/gdl-sync.py

@@ -225,7 +225,7 @@ def read_urls_file(path: Path) -> list[str]:
 def read_post_urls_file(path: Path) -> list[str]:
     """
     Read individual post/reel URLs from a file, one per line -- the output
-    format `reels-scrape.py` writes. Blank lines and `#` comments are skipped;
+    format `instagram-chrome-scrape.py` writes. Blank lines and `#` comments are skipped;
     order is kept and duplicates dropped, same conventions as read_urls_file.
     """
     urls: list[str] = []
@@ -929,7 +929,7 @@ def main() -> int:
                    help="file of post/reel URLs, one per line; # comments and "
                         "blank lines allowed. Same handling as --post-url, "
                         "paced with --sleep-request between items. This is "
-                        "the format reels-scrape.py writes")
+                        "the format instagram-chrome-scrape.py writes")
     ap.add_argument("--cookies", default="chrome:/home/matt/.config/google-chrome-devtools",
                     help="gallery-dl --cookies-from-browser value")
     ap.add_argument("--archive-db", type=Path, default=None,

scripts/gdl-upgrade.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+# Upgrade gallery-dl and its injected yt-dlp (both pipx). Instagram changes its
+# endpoints often and gallery-dl follows within days -- 1.32.17 (2026-10-17)
+# "update user & posts query parameters" landed while we were still on 1.32.14
+# and the posts listing silently returned "No results".
+#
+# Called by gdl-cron.sh before every scrape and weekly by gdl-upgrade.timer.
+# Never fatal: a failed upgrade (offline, PyPI down) must not stop a stories
+# run, which cannot be backfilled -- it just scrapes with what is installed.
+# Costs no Instagram requests (PyPI only).
+set -u
+
+PATH="$HOME/.local/bin:$PATH"; export PATH
+
+VENV_PY="${GDL_VENV_PY:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python}"
+
+versions() {
+  printf 'gallery-dl %s, yt-dlp %s' \
+    "$(gallery-dl --version 2>/dev/null || echo '?')" \
+    "$("$VENV_PY" -c 'import yt_dlp; print(yt_dlp.version.__version__)' 2>/dev/null || echo '?')"
+}
+
+before=$(versions)
+if ! timeout 180 pipx upgrade --quiet --include-injected gallery-dl >/dev/null 2>&1; then
+  echo "upgrade: WARNING pipx upgrade failed; continuing with $before" >&2
+  exit 0
+fi
+after=$(versions)
+
+if [ "$before" = "$after" ]; then
+  echo "upgrade: up to date ($after)"
+else
+  echo "upgrade: CHANGED $before -> $after"
+fi

scripts/instagram-chrome-scrape.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 """
-Find a profile's reels by scrolling the real page, not calling the API.
+Find a profile's reels (or posts) by scrolling the real page, not calling the API.
 
 gallery-dl's dedicated reels extractor POSTs to /api/v1/clips/user/, which
 Instagram now 302-redirects to the home page for this account -- confirmed on
@@ -11,11 +11,19 @@ on loopback CDP for MCP automation -- see TOOLING.md) via the DevTools
 protocol, scrolls it like a person would, and scrapes `/reel/<code>/` links
 out of the rendered page instead.
 
+`--tab posts` does the same for the main profile grid (`/<user>/`, both `/p/` and
+`/reel/` links), for when gallery-dl's posts listing breaks -- on 2026-10-22,
+1.32.14 returned "No results" and 1.32.17 "Requested user could not be found"
+(upstream gh#9829), while single-post fetches kept working. Since the grid
+runs newest-first, it stops scrolling once rounds turn up nothing but
+already-archived posts (`--max-known-rounds`) instead of walking a whole
+profile's history.
+
 This only finds shortcodes; it never downloads anything itself. What comes
 out the other end is deduped against the archive (via the same --index used
 elsewhere) and printed as plain post URLs -- feed them to gdl-sync.py:
 
-    ./scripts/reels-scrape.py --profile someuser \\
+    ./scripts/instagram-chrome-scrape.py --profile someuser \\
         --index https://instaarchive.ergosteur.com > /tmp/someuser-reels.txt
 
     ./scripts/gdl-sync.py --publish user@host:/path --staging /var/tmp/gdl \\
@@ -38,7 +46,13 @@ import time
 from pathlib import Path
 from urllib.request import urlopen
 
-RE_REEL_HREF = re.compile(r"/reel/([^/?#]+)")
+RE_POST_HREF = re.compile(r"/(p|reel)/([^/?#]+)")
+
+# tab -> (page path under the profile, link kinds to collect)
+TABS = {
+    "reels": ("reels/", ("reel",)),
+    "posts": ("", ("p", "reel")),
+}
 
 # The reels tab has finished loading once a scroll round finds no NEW
 # shortcodes this many times in a row -- lazy-loaded feeds sometimes stall
@@ -46,21 +60,29 @@ RE_REEL_HREF = re.compile(r"/reel/([^/?#]+)")
 DEFAULT_MAX_IDLE_ROUNDS = 3
 DEFAULT_MAX_SCROLLS = 200
 DEFAULT_SCROLL_PAUSE = (2.0, 3.5)
+# Posts tab only: stop after this many consecutive scroll rounds whose new
+# shortcodes were all already archived. The first round is exempt -- it holds
+# the pinned posts, which are old by nature and sit above the new ones.
+DEFAULT_MAX_KNOWN_ROUNDS = 2
 
-SCRAPE_JS = (
+
+def scrape_js(kinds: tuple[str, ...]) -> str:
+    selector = ", ".join(f'a[href*="/{k}/"]' for k in kinds)
+    return (
         "(() => {"
         "window.scrollTo(0, document.body.scrollHeight);"
-    "return Array.from(document.querySelectorAll('a[href*=\"/reel/\"]'))"
+        f"return Array.from(document.querySelectorAll({json.dumps(selector)}))"
         ".map(a => a.getAttribute('href'));"
         "})()"
     )
 
 
-def extract_shortcodes(hrefs: list[str]) -> list[str]:
+def extract_shortcodes(hrefs: list[str],
+                       kinds: tuple[str, ...] = ("reel",)) -> list[str]:
     codes = []
     for href in hrefs:
-        if m := RE_REEL_HREF.search(href):
+        if (m := RE_POST_HREF.search(href)) and m.group(1) in kinds:
-            codes.append(m.group(1))
+            codes.append(m.group(2))
     return codes
 
 
@@ -100,23 +122,45 @@ class CDP:
         self.ws.close()
 
 
-def scrape_reels(user: str, cdp_port: int = 9222,
+def should_stop_known(round_no: int, new: list[str], have: set[str],
+                      streak: int, max_known_rounds: int) -> tuple[bool, int]:
+    """
+    Early-stop bookkeeping for a newest-first grid: returns (stop, streak).
+    A round counts toward the streak when it found new shortcodes and every
+    one is already archived; a round with anything unarchived resets it. An
+    empty round leaves it alone (that is what max_idle_rounds is for), and
+    round 1 never counts -- pinned posts come first.
+    """
+    if max_known_rounds <= 0 or round_no == 1 or not new:
+        return False, streak
+    if all(c in have for c in new):
+        streak += 1
+    else:
+        streak = 0
+    return streak >= max_known_rounds, streak
+
+
+def scrape_grid(user: str, tab: str = "reels", cdp_port: int = 9222,
                 scroll_pause: tuple[float, float] = DEFAULT_SCROLL_PAUSE,
                 max_idle_rounds: int = DEFAULT_MAX_IDLE_ROUNDS,
                 max_scrolls: int = DEFAULT_MAX_SCROLLS,
+                have: set[str] | None = None,
+                max_known_rounds: int = 0,
                 log=lambda msg: None) -> list[str]:
     """
-    Open the profile's reels tab in a NEW tab of the already-signed-in Chrome,
+    Open the profile's reels tab (or main grid) in a NEW tab of the
-    scroll it to the bottom repeatedly, and collect every unique `/reel/`
+    already-signed-in Chrome, scroll it to the bottom repeatedly, and collect
-    shortcode that appears. Closes the tab when done either way.
+    every unique shortcode that appears. Closes the tab when done either way.
     """
+    path, kinds = TABS[tab]
+    js = scrape_js(kinds)
     version = json.loads(urlopen(f"http://localhost:{cdp_port}/json/version",
                                  timeout=10).read())
     browser = CDP(version["webSocketDebuggerUrl"])
     target_id = None
     try:
         target = browser.send("Target.createTarget", {
-            "url": f"https://www.instagram.com/{user}/reels/"})
+            "url": f"https://www.instagram.com/{user}/{path}"})
         target_id = target["targetId"]
         attach = browser.send("Target.attachToTarget", {
             "targetId": target_id, "flatten": True})
@@ -128,16 +172,24 @@ def scrape_reels(user: str, cdp_port: int = 9222,
 
         seen: set[str] = set()
         idle = 0
+        known_streak = 0
         for i in range(max_scrolls):
             result = browser.send(
                 "Runtime.evaluate",
-                {"expression": SCRAPE_JS, "returnByValue": True},
+                {"expression": js, "returnByValue": True},
                 session_id=session_id)
             hrefs = result.get("result", {}).get("value") or []
-            codes = extract_shortcodes(hrefs)
+            codes = extract_shortcodes(hrefs, kinds)
-            new = [c for c in codes if c not in seen]
+            new = list(dict.fromkeys(c for c in codes if c not in seen))
             seen.update(new)
-            log(f"  scroll {i + 1}: {len(seen)} unique reels so far (+{len(new)})")
+            log(f"  scroll {i + 1}: {len(seen)} unique {tab} so far (+{len(new)})")
+
+            stop, known_streak = should_stop_known(
+                i + 1, new, have or set(), known_streak, max_known_rounds)
+            if stop:
+                log(f"  {known_streak} round(s) of already-archived {tab} in a row; "
+                    f"stopping")
+                break
 
             if new:
                 idle = 0
@@ -162,7 +214,9 @@ def main() -> int:
     ap = argparse.ArgumentParser(description=__doc__,
                                  formatter_class=argparse.RawDescriptionHelpFormatter)
     ap.add_argument("--profile", required=True,
-                    help="Instagram username to scrape reels for")
+                    help="Instagram username to scrape")
+    ap.add_argument("--tab", choices=sorted(TABS), default="reels",
+                    help="reels tab, or the main posts grid (default: reels)")
     ap.add_argument("--index", required=True,
                     help="existing archive listing, same as gdl-sync.py's "
                          "--index: a local root, or the viewer's base URL. "
@@ -179,6 +233,11 @@ def main() -> int:
     ap.add_argument("--max-scrolls", type=int, default=DEFAULT_MAX_SCROLLS,
                     help="hard ceiling on scroll rounds, in case a page never "
                          "goes idle")
+    ap.add_argument("--max-known-rounds", type=int, default=None,
+                    help="stop after this many consecutive scroll rounds of "
+                         "only already-archived shortcodes; 0 disables "
+                         f"(default: {DEFAULT_MAX_KNOWN_ROUNDS} for posts, "
+                         "0 for reels)")
     ap.add_argument("--out", type=Path,
                     help="write new reel URLs here, one per line (default: "
                          "stdout)")
@@ -208,18 +267,28 @@ def main() -> int:
     print(f"archive already holds {len(have)} shortcode(s) across "
           f"{len(profiles)} profile(s)", file=sys.stderr)
 
-    print(f"scraping https://www.instagram.com/{args.profile}/reels/ ...",
+    max_known = args.max_known_rounds
+    if max_known is None:
+        max_known = DEFAULT_MAX_KNOWN_ROUNDS if args.tab == "posts" else 0
+
+    path, _ = TABS[args.tab]
+    print(f"scraping https://www.instagram.com/{args.profile}/{path} ...",
           file=sys.stderr)
-    codes = scrape_reels(args.profile, cdp_port=args.cdp_port,
+    codes = scrape_grid(args.profile, tab=args.tab, cdp_port=args.cdp_port,
                         scroll_pause=tuple(args.scroll_pause),
                         max_idle_rounds=args.max_idle_rounds,
                         max_scrolls=args.max_scrolls,
+                        have=have, max_known_rounds=max_known,
                         log=lambda msg: print(msg, file=sys.stderr))
 
     new_codes = [c for c in codes if c not in have]
-    print(f"found {len(codes)} reel(s) on the page, {len(codes) - len(new_codes)} "
+    print(f"found {len(codes)} {args.tab} on the page, {len(codes) - len(new_codes)} "
           f"already archived, {len(new_codes)} new", file=sys.stderr)
 
+    # /p/<code>/ resolves reels too, so the posts grid needs no per-kind URL.
+    if args.tab == "posts":
+        urls = [f"https://www.instagram.com/p/{c}/" for c in new_codes]
+    else:
         urls = [f"https://www.instagram.com/{args.profile}/reel/{c}/"
                 for c in new_codes]
     text = "\n".join(urls)

scripts/posts-sync.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+# chrome-sync.sh for the main profile grid: scroll /<user>/ in the signed-in
+# Chrome, collect /p/ and /reel/ links, fetch only the new ones as single
+# posts. A by-hand fallback for when gallery-dl's posts listing is broken
+# (2026-10-22: upstream gh#9829) while single-post fetches still work -- see
+# TOOLING.md. Stops scrolling once it reaches already-archived posts.
+#
+# Usage: ./posts-sync.sh <username-or-profile-url>
+set -eu
+export GDL_TAB=posts
+exec "$(dirname "$0")/chrome-sync.sh" "$@"

scripts/reels-sync.sh

@@ -1,114 +1,8 @@
 #!/bin/bash
-# The full reels pipeline for one profile: scrape by scrolling the real page,
+# chrome-sync.sh for a profile's reels tab. See TOOLING.md ("Reels: the API is
-# then fetch and publish whatever's new. See TOOLING.md ("Reels: the API is
+# blocked, scrape by scrolling instead").
-# blocked, scrape by scrolling instead") for why this exists at all -- the
-# dedicated reels API is blocked, this drives the actual signed-in browser
-# session instead, and is slower by design.
 #
 # Usage: ./reels-sync.sh <username-or-profile-url>
-#   ./reels-sync.sh zindoriyam
-#   ./reels-sync.sh https://www.instagram.com/zindoriyam/
-#
-# Deliberately NOT wired into gdl-cron.sh or the timers -- see TOOLING.md.
-# Exits non-zero if either step does. Everything is logged.
 set -eu
-
+export GDL_TAB=reels
-RAW="${1:?usage: reels-sync.sh <username-or-profile-url>}"
+exec "$(dirname "$0")/chrome-sync.sh" "$@"
-
-GDL_HOME="${GDL_HOME:-$HOME/gdl}"
-GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
-INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
-PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
-
-# Same hand-paced pacing gdl-cron.sh uses -- see its comment for why. Override
-# per-run with GDL_SLEEP_REQUEST etc. if you ever need to, but raise them
-# rather than lower them.
-SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
-SLEEP="${GDL_SLEEP:-5 10}"
-RATE="${GDL_RATE:-500K}"
-SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}"
-MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}"
-
-PATH="$HOME/.local/bin:$PATH"; export PATH
-
-if [ ! -x "$GDL_PYTHON" ]; then
-  echo "reels-scrape.py needs gallery-dl's own pipx venv python (websocket-client" >&2
-  echo "was injected there, not into the system python): $GDL_PYTHON not found" >&2
-  exit 2
-fi
-
-# Same username-from-URL parsing gdl-sync.py already does for --urls-file,
-# reused rather than re-implemented so the two never drift apart.
-PROFILE=$("$GDL_PYTHON" -c "
-import re, sys, importlib.util
-from pathlib import Path
-spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
-gdl = importlib.util.module_from_spec(spec)
-sys.modules['gdl_sync'] = gdl
-spec.loader.exec_module(gdl)
-raw = '$RAW'
-m = gdl.RE_PROFILE_URL.match(raw)
-user = m.group('user') if m else raw.strip('/')
-if not user or '/' in user or ' ' in user:
-    print(f'cannot read a username from {raw!r}', file=sys.stderr)
-    sys.exit(1)
-print(user)
-")
-
-mkdir -p "$GDL_HOME/logs"
-LOG="$GDL_HOME/logs/reels-$PROFILE-$(date +%Y%m%d-%H%M%S)-$$.log"
-URLS_FILE="$GDL_HOME/$PROFILE-reels.txt"
-STAGING="$GDL_HOME/staging-reels-$PROFILE"
-
-echo "=== reels-sync $PROFILE $(date -Is) ===" | tee -a "$LOG"
-
-# The exit status has to survive the pipe into tee -- see gdl-cron.sh's
-# comment on PIPESTATUS for why this needs to be bash, not sh.
-set +e
-"$GDL_PYTHON" "$GDL_HOME/reels-scrape.py" \
-    --profile "$PROFILE" \
-    --index "$INDEX" \
-    --scroll-pause $SCROLL_PAUSE \
-    --max-idle-rounds "$MAX_IDLE_ROUNDS" \
-    --out "$URLS_FILE" 2>&1 | tee -a "$LOG"
-scrape_status=${PIPESTATUS[0]}
-set -e
-
-if [ "$scrape_status" -ne 0 ]; then
-  echo "=== exit $scrape_status (scrape failed) at $(date -Is) ===" | tee -a "$LOG"
-  exit "$scrape_status"
-fi
-
-if [ ! -s "$URLS_FILE" ]; then
-  echo "no new reels for $PROFILE; nothing to fetch" | tee -a "$LOG"
-  echo "=== exit 0 at $(date -Is) ===" | tee -a "$LOG"
-  exit 0
-fi
-
-# Staging is wiped every run on purpose -- same reasoning as gdl-cron.sh: what
-# we already hold is decided by the archive dedupe in reels-scrape.py, not by
-# what happens to be sitting in staging.
-rm -rf "$STAGING"
-
-set +e
-# shellcheck disable=SC2086
-"$GDL_HOME/gdl-sync.py" \
-    --publish "$PUBLISH" \
-    --staging "$STAGING" \
-    --post-urls-file "$URLS_FILE" \
-    --sleep-request $SLEEP_REQUEST \
-    --sleep $SLEEP \
-    --rate "$RATE" \
-    --execute 2>&1 | tee -a "$LOG"
-status=${PIPESTATUS[0]}
-set -e
-
-echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
-
-# Keep the log directory from growing without bound -- scoped to this
-# script's own logs so it never touches gdl-cron.sh's rotation.
-ls -1t "$GDL_HOME/logs" | grep '^reels-' | tail -n +30 | while read -r old; do
-  rm -f "$GDL_HOME/logs/$old"
-done
-
-exit "$status"

scripts/run-reels-all.sh

@@ -5,9 +5,9 @@
 # version of running reels-sync.sh six times by hand, which is what the
 # 2026-09-03 run was before this script existed.
 #
-# Deliberately not wired into gdl-cron.sh or a timer -- same reasoning as
+# A by-hand full sweep of every Reels tab (no early stop). The weekly
-# reels-sync.sh itself: it drives your actual browser session and is slower
+# gdl-cron.sh profiles run lists reels through chrome-sync.sh itself, with an
-# by design, both good reasons to run it deliberately, not unattended.
+# early stop, since 2026-10-23 -- see TOOLING.md "Posts grid by browser".
 #
 # Usage: ./reels-all.sh
 #   GDL_ARTMS_LINKS=/path/to/list ./reels-all.sh   # override the profile list

scripts/systemd/gdl-upgrade.service

@@ -0,0 +1,9 @@
+[Unit]
+Description=Upgrade gallery-dl and yt-dlp (pipx)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=%h/gdl/gdl-upgrade.sh
+Nice=10

scripts/systemd/gdl-upgrade.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Weekly gallery-dl and yt-dlp upgrade
+# gdl-cron.sh also upgrades before every scrape; this keeps the tools current
+# on weeks when no run happens to call it.
+
+[Timer]
+OnCalendar=weekly
+RandomizedDelaySec=1d
+Persistent=true
+
+[Install]
+WantedBy=timers.target

scripts/test_gdl_sync.py

@@ -326,7 +326,7 @@ class UrlsFile(unittest.TestCase):
 
 
 class PostUrlsFile(unittest.TestCase):
-    """The format reels-scrape.py writes: whole URLs, not usernames."""
+    """The format instagram-chrome-scrape.py writes: whole URLs, not usernames."""
 
     def test_skips_comments_blanks_and_duplicates(self):
         with tempfile.TemporaryDirectory() as d:

scripts/test_instagram_chrome_scrape.py

@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+"""
+Tests for the pure parts of instagram-chrome-scrape.py -- everything except the actual
+CDP session, which needs a live signed-in Chrome and is exercised by hand.
+
+    python3 -m unittest discover -s scripts -p 'test_*.py'
+"""
+
+import importlib.util
+import sys
+import unittest
+from pathlib import Path
+
+_spec = importlib.util.spec_from_file_location(
+    "instagram_chrome_scrape", Path(__file__).with_name("instagram-chrome-scrape.py"))
+scrape = importlib.util.module_from_spec(_spec)
+sys.modules["instagram_chrome_scrape"] = scrape
+_spec.loader.exec_module(scrape)
+
+
+class ExtractShortcodes(unittest.TestCase):
+    def test_reads_relative_and_absolute_hrefs(self):
+        hrefs = [
+            "/someuser/reel/ABC123/",
+            "https://www.instagram.com/someuser/reel/DEF456/",
+            "/reel/GHI789/?img_index=1",
+        ]
+        self.assertEqual(scrape.extract_shortcodes(hrefs),
+                         ["ABC123", "DEF456", "GHI789"])
+
+    def test_ignores_non_reel_links(self):
+        hrefs = ["/someuser/", "/someuser/p/ABC123/", "/explore/tags/foo/"]
+        self.assertEqual(scrape.extract_shortcodes(hrefs), [])
+
+    def test_empty_input(self):
+        self.assertEqual(scrape.extract_shortcodes([]), [])
+
+    def test_posts_kinds_read_both_post_and_reel_links(self):
+        hrefs = ["/someuser/p/ABC123/", "/p/DEF456/?img_index=2",
+                 "/someuser/reel/GHI789/", "/explore/tags/foo/"]
+        self.assertEqual(scrape.extract_shortcodes(hrefs, ("p", "reel")),
+                         ["ABC123", "DEF456", "GHI789"])
+
+
+class ShouldStopKnown(unittest.TestCase):
+    have = {"OLD1", "OLD2", "OLD3", "OLD4"}
+
+    def step(self, round_no, new, streak=0, max_known=2):
+        return scrape.should_stop_known(round_no, new, self.have, streak, max_known)
+
+    def test_first_round_never_counts(self):
+        # pinned posts: old, but on top of the grid
+        self.assertEqual(self.step(1, ["OLD1", "OLD2"]), (False, 0))
+
+    def test_stops_after_consecutive_known_rounds(self):
+        stop, streak = self.step(2, ["OLD1"])
+        self.assertEqual((stop, streak), (False, 1))
+        self.assertEqual(self.step(3, ["OLD2"], streak), (True, 2))
+
+    def test_unarchived_resets_streak(self):
+        self.assertEqual(self.step(3, ["OLD3", "NEW1"], streak=1), (False, 0))
+
+    def test_empty_round_leaves_streak(self):
+        self.assertEqual(self.step(3, [], streak=1), (False, 1))
+
+    def test_disabled(self):
+        self.assertEqual(self.step(5, ["OLD1"], streak=9, max_known=0), (False, 9))
+
+
+if __name__ == "__main__":
+    unittest.main(verbosity=2)

scripts/test_reels_scrape.py

@@ -1,40 +0,0 @@
-#!/usr/bin/env python3
-"""
-Tests for the pure parts of reels-scrape.py -- everything except the actual
-CDP session, which needs a live signed-in Chrome and is exercised by hand.
-
-    python3 -m unittest discover -s scripts -p 'test_*.py'
-"""
-
-import importlib.util
-import sys
-import unittest
-from pathlib import Path
-
-_spec = importlib.util.spec_from_file_location(
-    "reels_scrape", Path(__file__).with_name("reels-scrape.py"))
-scrape = importlib.util.module_from_spec(_spec)
-sys.modules["reels_scrape"] = scrape
-_spec.loader.exec_module(scrape)
-
-
-class ExtractShortcodes(unittest.TestCase):
-    def test_reads_relative_and_absolute_hrefs(self):
-        hrefs = [
-            "/someuser/reel/ABC123/",
-            "https://www.instagram.com/someuser/reel/DEF456/",
-            "/reel/GHI789/?img_index=1",
-        ]
-        self.assertEqual(scrape.extract_shortcodes(hrefs),
-                         ["ABC123", "DEF456", "GHI789"])
-
-    def test_ignores_non_reel_links(self):
-        hrefs = ["/someuser/", "/someuser/p/ABC123/", "/explore/tags/foo/"]
-        self.assertEqual(scrape.extract_shortcodes(hrefs), [])
-
-    def test_empty_input(self):
-        self.assertEqual(scrape.extract_shortcodes([]), [])
-
-
-if __name__ == "__main__":
-    unittest.main(verbosity=2)