From fe6ebdd75b6d45976bbb103abcec6d5fe1506111 Mon Sep 17 00:00:00 2001 From: ergosteur Date: Fri, 23 Oct 2026 00:49:45 -0400 Subject: [PATCH] tooling: scheduled profiles/full-sweep list posts+reels by browser (chrome-sync per profile/tab, early stop for weekly); GDL_LISTING=api fallback; reels-tab sweep note in MANUAL-RUN --- MANUAL-RUN.md | 15 ++++++--- TOOLING.md | 22 +++++++++----- scripts/chrome-sync.sh | 6 +++- scripts/gdl-cron.sh | 66 ++++++++++++++++++++++++++++++++++++++-- scripts/run-reels-all.sh | 6 ++-- 5 files changed, 95 insertions(+), 20 deletions(-) diff --git a/MANUAL-RUN.md b/MANUAL-RUN.md index c7484b2..9f24cfd 100644 --- a/MANUAL-RUN.md +++ b/MANUAL-RUN.md @@ -54,16 +54,21 @@ for p in withaseul cher_ryppo zindoriyam; do ~/gdl/posts-sync.sh $p; done minutes, mostly the 12–20 s gap between fetches. - It prints `no new posts for ` when a profile is caught up. - Logs: `~/gdl/logs/posts--*.log`. -- Reels also show up in the main grid. `~/gdl/run-reels-all.sh` sweeps the separate Reels tab - too (~15 min) and is rarely needed. +- **Then sweep the Reels tab too:** `~/gdl/run-reels-all.sh` (~15 min). Most reels never + appear in the main grid. On 2026-10-23 only 3 of heejin's 114 reels did, so `posts-sync.sh` + alone misses them. `reels-sync.sh ` does a single profile. -**Once a gallery-dl release fixes the listing**, one command covers posts, reels, stories and -highlights for all six profiles: +**Or the whole thing in one command**, the same run the weekly timer does: gallery-dl for +stories and highlights, then the browser for every profile's posts grid and Reels tab, each +stopping once it reaches archived content: ```sh -~/gdl/run-profiles.sh # ~5 min +~/gdl/run-profiles.sh ``` +Once a gallery-dl release fixes the listing, `GDL_LISTING=api ~/gdl/run-profiles.sh` does posts +and reels through gallery-dl again (~5 min). + For long runs, detach: `setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &`, then `tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`. diff --git a/TOOLING.md b/TOOLING.md index 46de308..7f39eb0 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -327,9 +327,15 @@ newest-first, so after 2 scroll rounds in a row that turned up nothing unarchive (`--max-known-rounds`, round 1 exempt because pinned posts sit on top) there is nothing newer to find. First run, `heejin`: 3 scrolls, 36 posts seen, 30 already held, 6 new. -Same caveats as reels: it drives the signed-in Chrome on `:9222`, so it is by hand, not on a -timer. Once a gallery-dl release fixes the listing, `run-profiles.sh` covers posts again and -this goes back to being a fallback. +**Scheduled since 2026-10-23.** `gdl-cron.sh profiles` and `full-sweep` now list posts *and* +reels this way: gallery-dl only does `--only stories,highlights`, then `chrome-sync.sh` runs per +profile for the `posts` grid and the `reels` tab (both tabs needed: on 10-23 only 3 of heejin's +114 reels were in the main grid). `profiles` uses the early stop (2 known rounds) on both tabs, +and `full-sweep` scrolls every grid to the bottom. A failed browser step fails the run, which +shows on Healthchecks. This reverses the earlier "by hand only" decision on purpose: the API +listing is broken and the owner chose the browser. The trade-off is that unattended runs now +need Chrome up on `:9222` with a live session, and they open and close their own tabs in it. +`GDL_LISTING=api` restores the all-gallery-dl run once a release fixes gh#9829. **Deploy scripts atomically, never in place.** bash reads a script as it runs, so overwriting `gdl-cron.sh` under a running sync makes bash resume at the same byte offset in the *new* text. @@ -836,11 +842,11 @@ after — left behind for inspection, same as `gdl-cron.sh`'s `staging-*`. Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the page, 10 new ones fetched and published cleanly on the first run. -**Not wired into `gdl-cron.sh` or the timers.** Both scripts drive your -actual browser session rather than firing a background request, and are -slower by design (real scrolling, not an API call) — both good reasons not -to run this unattended without deciding that deliberately. Today it is a -per-profile, by-hand tool only. +**Not wired into `gdl-cron.sh` or the timers** (as of 2026-08; **superseded 2026-10-23**: the +weekly and quarterly runs now use the browser listing, see "Posts grid by browser" above). Both +scripts drive your actual browser session rather than firing a background request, and are +slower by design (real scrolling, not an API call), both good reasons not to run this +unattended without deciding that deliberately. For all six ARTMS profiles in one go, `~/gdl/run-reels-all.sh` just calls `reels-sync.sh` once per profile in `artms_account_links.txt`, same diff --git a/scripts/chrome-sync.sh b/scripts/chrome-sync.sh index 19c17a7..0af55a5 100755 --- a/scripts/chrome-sync.sh +++ b/scripts/chrome-sync.sh @@ -9,7 +9,8 @@ # Usage: GDL_TAB=reels|posts ./chrome-sync.sh # Normally called through reels-sync.sh / posts-sync.sh, which set GDL_TAB. # -# Deliberately NOT wired into gdl-cron.sh or the timers -- see TOOLING.md. +# Also run by gdl-cron.sh (profiles, full-sweep) once per profile and tab since +# 2026-10-23, with GDL_MAX_KNOWN_ROUNDS set -- see TOOLING.md "Posts grid by browser". # Exits non-zero if either step does. Everything is logged. set -eu @@ -29,6 +30,8 @@ SLEEP="${GDL_SLEEP:-5 10}" RATE="${GDL_RATE:-500K}" SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}" MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}" +# Empty = instagram-chrome-scrape.py's own default (2 for posts, 0 = scroll to the end for reels). +MAX_KNOWN_ROUNDS="${GDL_MAX_KNOWN_ROUNDS:-}" PATH="$HOME/.local/bin:$PATH"; export PATH @@ -72,6 +75,7 @@ set +e --index "$INDEX" \ --scroll-pause $SCROLL_PAUSE \ --max-idle-rounds "$MAX_IDLE_ROUNDS" \ + ${MAX_KNOWN_ROUNDS:+--max-known-rounds "$MAX_KNOWN_ROUNDS"} \ --out "$URLS_FILE" 2>&1 | tee -a "$LOG" scrape_status=${PIPESTATUS[0]} set -e diff --git a/scripts/gdl-cron.sh b/scripts/gdl-cron.sh index a22eb00..f0223a6 100755 --- a/scripts/gdl-cron.sh +++ b/scripts/gdl-cron.sh @@ -10,6 +10,11 @@ # EDITED after we archived them, and by far the most # expensive thing here -- see TOOLING.md before running. # +# Posts and reels are LISTED by scrolling the signed-in Chrome (chrome-sync.sh, +# per profile and tab) rather than by gallery-dl's listing API, which broke on +# 2026-10-22 (gh#9829) while single-post fetches kept working. gallery-dl still +# does stories and highlights. GDL_LISTING=api restores the old all-API run. +# # Exits non-zero if the sync does, so cron mails you. Everything is logged. set -eu @@ -41,6 +46,15 @@ SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}" SLEEP="${GDL_SLEEP:-5 10}" RATE="${GDL_RATE:-500K}" +LISTING="${GDL_LISTING:-browser}" +case "$LISTING" in + browser|api) ;; + *) echo "GDL_LISTING must be browser or api, not '$LISTING'" >&2; exit 2 ;; +esac +GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}" +# Set per mode below when posts/reels go through the browser: empty = no browser step. +BROWSER_KNOWN_ROUNDS="" + case "$MODE" in # --min-interval 8, not the 20h default. The floor exists to stop an ABORTED # RESTART re-enumerating profiles -- a minutes-to-hours concern. At 20h the @@ -50,10 +64,23 @@ case "$MODE" in # request per profile, so the worst case 8h permits is roughly twelve # requests in a day instead of six. stories) ARGS="--only stories --min-interval 8" ;; - profiles) ARGS="--only posts,reels,stories,highlights --abort 50" ;; + # Browser listing: stop scrolling a grid after 2 rounds of nothing unarchived, + # the browser's equivalent of --abort 50. + profiles) + if [ "$LISTING" = browser ]; then + ARGS="--only stories,highlights --abort 50"; BROWSER_KNOWN_ROUNDS=2 + else + ARGS="--only posts,reels,stories,highlights --abort 50" + fi ;; # No --abort: the whole point of full-sweep is enumerating to the end, so it - # is the only run that notices carousels edited after we archived them. - full-sweep) ARGS="--only posts,reels,stories,highlights" ;; + # is the only run that notices carousels edited after we archived them. The + # browser equivalent is scrolling every grid to the bottom (0 = no early stop). + full-sweep) + if [ "$LISTING" = browser ]; then + ARGS="--only stories,highlights"; BROWSER_KNOWN_ROUNDS=0 + else + ARGS="--only posts,reels,stories,highlights" + fi ;; # Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and # "sweep" did not say it was the exhaustive one. Catch the old names rather # than failing with a bare error, in case something still passes them. @@ -125,6 +152,39 @@ set +e status=${PIPESTATUS[0]} set -e +# Posts and reels by browser: scrape each profile's grid and Reels tab, fetch only +# the new shortcodes as single posts. chrome-sync.sh logs and publishes on its own; +# its output is copied into this run's log so the Healthchecks body shows it too. +if [ -n "$BROWSER_KNOWN_ROUNDS" ]; then + mapfile -t PROFILES < <("$GDL_PYTHON" -c " +import sys, importlib.util +from pathlib import Path +spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py')) +gdl = importlib.util.module_from_spec(spec) +sys.modules['gdl_sync'] = gdl +spec.loader.exec_module(gdl) +for p in gdl.read_urls_file(Path('$GDL_HOME/artms_account_links.txt')): + print(p) +") + browser_failed=0 + for p in "${PROFILES[@]}"; do + for tab in posts reels; do + echo "==> browser: $p / $tab" | tee -a "$LOG" + set +e + GDL_TAB=$tab GDL_MAX_KNOWN_ROUNDS=$BROWSER_KNOWN_ROUNDS \ + "$GDL_HOME/chrome-sync.sh" "$p" 2>&1 | tee -a "$LOG" >/dev/null + s=${PIPESTATUS[0]} + set -e + if [ "$s" -ne 0 ]; then + echo " browser $p / $tab FAILED (exit $s)" | tee -a "$LOG" >&2 + browser_failed=$((browser_failed + 1)) + fi + done + done + echo "browser listing: ${#PROFILES[@]} profile(s), $browser_failed step(s) failed" | tee -a "$LOG" + [ "$browser_failed" -gt 0 ] && [ "$status" -eq 0 ] && status=1 +fi + # A stories run that skipped every source is NOT a success. It means the # min-interval floor blocked the one surface that cannot be backfilled, and # without this it looks identical to a clean run: exit 0, "0 step(s) failed". diff --git a/scripts/run-reels-all.sh b/scripts/run-reels-all.sh index 662540e..8e43a1d 100755 --- a/scripts/run-reels-all.sh +++ b/scripts/run-reels-all.sh @@ -5,9 +5,9 @@ # version of running reels-sync.sh six times by hand, which is what the # 2026-09-03 run was before this script existed. # -# Deliberately not wired into gdl-cron.sh or a timer -- same reasoning as -# reels-sync.sh itself: it drives your actual browser session and is slower -# by design, both good reasons to run it deliberately, not unattended. +# A by-hand full sweep of every Reels tab (no early stop). The weekly +# gdl-cron.sh profiles run lists reels through chrome-sync.sh itself, with an +# early stop, since 2026-10-23 -- see TOOLING.md "Posts grid by browser". # # Usage: ./reels-all.sh # GDL_ARTMS_LINKS=/path/to/list ./reels-all.sh # override the profile list