tooling: scheduled profiles/full-sweep list posts+reels by browser (chrome-sync per profile/tab, early stop for weekly); GDL_LISTING=api fallback; reels-tab sweep note in MANUAL-RUN

MANUAL-RUN.md

@@ -54,16 +54,21 @@ for p in withaseul cher_ryppo zindoriyam; do ~/gdl/posts-sync.sh $p; done
   minutes, mostly the 12–20 s gap between fetches.
 - It prints `no new posts for <user>` when a profile is caught up.
 - Logs: `~/gdl/logs/posts-<user>-*.log`.
-- Reels also show up in the main grid. `~/gdl/run-reels-all.sh` sweeps the separate Reels tab
+- **Then sweep the Reels tab too:** `~/gdl/run-reels-all.sh` (~15 min). Most reels never
-  too (~15 min) and is rarely needed.
+  appear in the main grid. On 2026-10-23 only 3 of heejin's 114 reels did, so `posts-sync.sh`
+  alone misses them. `reels-sync.sh <user>` does a single profile.
 
-**Once a gallery-dl release fixes the listing**, one command covers posts, reels, stories and
+**Or the whole thing in one command**, the same run the weekly timer does: gallery-dl for
-highlights for all six profiles:
+stories and highlights, then the browser for every profile's posts grid and Reels tab, each
+stopping once it reaches archived content:
 
 ```sh
-~/gdl/run-profiles.sh        # ~5 min
+~/gdl/run-profiles.sh
 ```
 
+Once a gallery-dl release fixes the listing, `GDL_LISTING=api ~/gdl/run-profiles.sh` does posts
+and reels through gallery-dl again (~5 min).
+
 For long runs, detach:
 `setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &`, then
 `tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`.

TOOLING.md

@@ -327,9 +327,15 @@ newest-first, so after 2 scroll rounds in a row that turned up nothing unarchive
 (`--max-known-rounds`, round 1 exempt because pinned posts sit on top) there is nothing newer to
 find. First run, `heejin`: 3 scrolls, 36 posts seen, 30 already held, 6 new.
 
-Same caveats as reels: it drives the signed-in Chrome on `:9222`, so it is by hand, not on a
+**Scheduled since 2026-10-23.** `gdl-cron.sh profiles` and `full-sweep` now list posts *and*
-timer. Once a gallery-dl release fixes the listing, `run-profiles.sh` covers posts again and
+reels this way: gallery-dl only does `--only stories,highlights`, then `chrome-sync.sh` runs per
-this goes back to being a fallback.
+profile for the `posts` grid and the `reels` tab (both tabs needed: on 10-23 only 3 of heejin's
+114 reels were in the main grid). `profiles` uses the early stop (2 known rounds) on both tabs,
+and `full-sweep` scrolls every grid to the bottom. A failed browser step fails the run, which
+shows on Healthchecks. This reverses the earlier "by hand only" decision on purpose: the API
+listing is broken and the owner chose the browser. The trade-off is that unattended runs now
+need Chrome up on `:9222` with a live session, and they open and close their own tabs in it.
+`GDL_LISTING=api` restores the all-gallery-dl run once a release fixes gh#9829.
 
 **Deploy scripts atomically, never in place.** bash reads a script as it runs, so overwriting
 `gdl-cron.sh` under a running sync makes bash resume at the same byte offset in the *new* text.
@@ -836,11 +842,11 @@ after — left behind for inspection, same as `gdl-cron.sh`'s `staging-*`.
 Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the
 page, 10 new ones fetched and published cleanly on the first run.
 
-**Not wired into `gdl-cron.sh` or the timers.** Both scripts drive your
+**Not wired into `gdl-cron.sh` or the timers** (as of 2026-08; **superseded 2026-10-23**: the
-actual browser session rather than firing a background request, and are
+weekly and quarterly runs now use the browser listing, see "Posts grid by browser" above). Both
-slower by design (real scrolling, not an API call) — both good reasons not
+scripts drive your actual browser session rather than firing a background request, and are
-to run this unattended without deciding that deliberately. Today it is a
+slower by design (real scrolling, not an API call), both good reasons not to run this
-per-profile, by-hand tool only.
+unattended without deciding that deliberately.
 
 For all six ARTMS profiles in one go, `~/gdl/run-reels-all.sh` just calls
 `reels-sync.sh` once per profile in `artms_account_links.txt`, same

scripts/chrome-sync.sh

@@ -9,7 +9,8 @@
 # Usage: GDL_TAB=reels|posts ./chrome-sync.sh <username-or-profile-url>
 # Normally called through reels-sync.sh / posts-sync.sh, which set GDL_TAB.
 #
-# Deliberately NOT wired into gdl-cron.sh or the timers -- see TOOLING.md.
+# Also run by gdl-cron.sh (profiles, full-sweep) once per profile and tab since
+# 2026-10-23, with GDL_MAX_KNOWN_ROUNDS set -- see TOOLING.md "Posts grid by browser".
 # Exits non-zero if either step does. Everything is logged.
 set -eu
 
@@ -29,6 +30,8 @@ SLEEP="${GDL_SLEEP:-5 10}"
 RATE="${GDL_RATE:-500K}"
 SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}"
 MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}"
+# Empty = instagram-chrome-scrape.py's own default (2 for posts, 0 = scroll to the end for reels).
+MAX_KNOWN_ROUNDS="${GDL_MAX_KNOWN_ROUNDS:-}"
 
 PATH="$HOME/.local/bin:$PATH"; export PATH
 
@@ -72,6 +75,7 @@ set +e
     --index "$INDEX" \
     --scroll-pause $SCROLL_PAUSE \
     --max-idle-rounds "$MAX_IDLE_ROUNDS" \
+    ${MAX_KNOWN_ROUNDS:+--max-known-rounds "$MAX_KNOWN_ROUNDS"} \
     --out "$URLS_FILE" 2>&1 | tee -a "$LOG"
 scrape_status=${PIPESTATUS[0]}
 set -e

scripts/gdl-cron.sh

@@ -10,6 +10,11 @@
 #                          EDITED after we archived them, and by far the most
 #                          expensive thing here -- see TOOLING.md before running.
 #
+# Posts and reels are LISTED by scrolling the signed-in Chrome (chrome-sync.sh,
+# per profile and tab) rather than by gallery-dl's listing API, which broke on
+# 2026-10-22 (gh#9829) while single-post fetches kept working. gallery-dl still
+# does stories and highlights. GDL_LISTING=api restores the old all-API run.
+#
 # Exits non-zero if the sync does, so cron mails you. Everything is logged.
 set -eu
 
@@ -41,6 +46,15 @@ SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
 SLEEP="${GDL_SLEEP:-5 10}"
 RATE="${GDL_RATE:-500K}"
 
+LISTING="${GDL_LISTING:-browser}"
+case "$LISTING" in
+  browser|api) ;;
+  *) echo "GDL_LISTING must be browser or api, not '$LISTING'" >&2; exit 2 ;;
+esac
+GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
+# Set per mode below when posts/reels go through the browser: empty = no browser step.
+BROWSER_KNOWN_ROUNDS=""
+
 case "$MODE" in
   # --min-interval 8, not the 20h default. The floor exists to stop an ABORTED
   # RESTART re-enumerating profiles -- a minutes-to-hours concern. At 20h the
@@ -50,10 +64,23 @@ case "$MODE" in
   # request per profile, so the worst case 8h permits is roughly twelve
   # requests in a day instead of six.
   stories)    ARGS="--only stories --min-interval 8" ;;
-  profiles)   ARGS="--only posts,reels,stories,highlights --abort 50" ;;
+  # Browser listing: stop scrolling a grid after 2 rounds of nothing unarchived,
+  # the browser's equivalent of --abort 50.
+  profiles)
+    if [ "$LISTING" = browser ]; then
+      ARGS="--only stories,highlights --abort 50"; BROWSER_KNOWN_ROUNDS=2
+    else
+      ARGS="--only posts,reels,stories,highlights --abort 50"
+    fi ;;
   # No --abort: the whole point of full-sweep is enumerating to the end, so it
-  # is the only run that notices carousels edited after we archived them.
+  # is the only run that notices carousels edited after we archived them. The
-  full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
+  # browser equivalent is scrolling every grid to the bottom (0 = no early stop).
+  full-sweep)
+    if [ "$LISTING" = browser ]; then
+      ARGS="--only stories,highlights"; BROWSER_KNOWN_ROUNDS=0
+    else
+      ARGS="--only posts,reels,stories,highlights"
+    fi ;;
   # Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
   # "sweep" did not say it was the exhaustive one. Catch the old names rather
   # than failing with a bare error, in case something still passes them.
@@ -125,6 +152,39 @@ set +e
 status=${PIPESTATUS[0]}
 set -e
 
+# Posts and reels by browser: scrape each profile's grid and Reels tab, fetch only
+# the new shortcodes as single posts. chrome-sync.sh logs and publishes on its own;
+# its output is copied into this run's log so the Healthchecks body shows it too.
+if [ -n "$BROWSER_KNOWN_ROUNDS" ]; then
+  mapfile -t PROFILES < <("$GDL_PYTHON" -c "
+import sys, importlib.util
+from pathlib import Path
+spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
+gdl = importlib.util.module_from_spec(spec)
+sys.modules['gdl_sync'] = gdl
+spec.loader.exec_module(gdl)
+for p in gdl.read_urls_file(Path('$GDL_HOME/artms_account_links.txt')):
+    print(p)
+")
+  browser_failed=0
+  for p in "${PROFILES[@]}"; do
+    for tab in posts reels; do
+      echo "==> browser: $p / $tab" | tee -a "$LOG"
+      set +e
+      GDL_TAB=$tab GDL_MAX_KNOWN_ROUNDS=$BROWSER_KNOWN_ROUNDS \
+        "$GDL_HOME/chrome-sync.sh" "$p" 2>&1 | tee -a "$LOG" >/dev/null
+      s=${PIPESTATUS[0]}
+      set -e
+      if [ "$s" -ne 0 ]; then
+        echo "    browser $p / $tab FAILED (exit $s)" | tee -a "$LOG" >&2
+        browser_failed=$((browser_failed + 1))
+      fi
+    done
+  done
+  echo "browser listing: ${#PROFILES[@]} profile(s), $browser_failed step(s) failed" | tee -a "$LOG"
+  [ "$browser_failed" -gt 0 ] && [ "$status" -eq 0 ] && status=1
+fi
+
 # A stories run that skipped every source is NOT a success. It means the
 # min-interval floor blocked the one surface that cannot be backfilled, and
 # without this it looks identical to a clean run: exit 0, "0 step(s) failed".

scripts/run-reels-all.sh

@@ -5,9 +5,9 @@
 # version of running reels-sync.sh six times by hand, which is what the
 # 2026-09-03 run was before this script existed.
 #
-# Deliberately not wired into gdl-cron.sh or a timer -- same reasoning as
+# A by-hand full sweep of every Reels tab (no early stop). The weekly
-# reels-sync.sh itself: it drives your actual browser session and is slower
+# gdl-cron.sh profiles run lists reels through chrome-sync.sh itself, with an
-# by design, both good reasons to run it deliberately, not unattended.
+# early stop, since 2026-10-23 -- see TOOLING.md "Posts grid by browser".
 #
 # Usage: ./reels-all.sh
 #   GDL_ARTMS_LINKS=/path/to/list ./reels-all.sh   # override the profile list