diff --git a/TOOLING.md b/TOOLING.md index dd4ec40..2bc89a2 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -544,12 +544,40 @@ Posts, highlights and stories are unaffected; only reels breaks. it drives the same signed-in Chrome via its loopback CDP port (`:9222`, already exposed for MCP automation), scrolls the reels tab like a person would, and scrapes `/reel//` links out of the rendered page. It only -finds shortcodes — nothing is downloaded until step 2 — and dedupes against -the archive first, so re-running it costs nothing for reels already held. +finds shortcodes — nothing is downloaded until the fetch step — and dedupes +against the **whole archive**, not just the profile being scraped (see why +below), so re-running it costs nothing for reels already held anywhere. + +### The easy way: `reels-sync.sh` ```sh ssh mattellite +~/gdl/reels-sync.sh zindoriyam +# or a full URL: ~/gdl/reels-sync.sh https://www.instagram.com/zindoriyam/ +``` +Runs both steps (scrape, then fetch + publish whatever's new) with the same +hand-paced settings `gdl-cron.sh` uses, logs to `~/gdl/logs/reels--*`, +and exits 0 with "no new reels" printed when a profile is already caught up +— `gdl-sync.py` never even gets invoked in that case. Override pacing the +same way as `gdl-cron.sh` (`GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE`), plus +`GDL_SCROLL_PAUSE` and `GDL_MAX_IDLE_ROUNDS` for the scrape step. Staging +(`~/gdl/staging-reels-`) and the scraped URL list +(`~/gdl/-reels.txt`) are wiped at the START of the next run, not +after — left behind for inspection, same as `gdl-cron.sh`'s `staging-*`. + +Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the +page, 10 new ones fetched and published cleanly on the first run. + +**Not wired into `gdl-cron.sh` or the timers.** Both scripts drive your +actual browser session rather than firing a background request, and are +slower by design (real scrolling, not an API call) — both good reasons not +to run this unattended without deciding that deliberately. Today it is a +per-profile, by-hand tool only. + +### What it's doing, if you want to run the two steps separately + +```sh PROFILE=someuser # 1. Scrape by scrolling; dedupe against the archive; write new URLs to a file. @@ -567,18 +595,8 @@ cd ~/gdl && PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \ --sleep-request 12 20 --sleep 5 10 --rate 500K \ --post-urls-file ~/gdl/"$PROFILE"-reels.txt --dry-run # ...then swap --dry-run for --execute once the plan looks right. - -# 3. Clean up the scratch files -- reels-scrape.py's --out file and gdl-sync.py's -# own staging directory are both left behind on purpose (same reasoning as -# everywhere else here: nothing is silently deleted). -rm -rf ~/gdl/staging-reels-scraped ~/gdl/staging-reels-scraped.gdl-config.json \ - ~/gdl/"$PROFILE"-reels.txt ``` -Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the -page, 16 already archived (correctly skipped), 10 new ones fetched and -published cleanly. - Notes: - `reels-scrape.py` fails fast if Chrome/CDP isn't up @@ -589,11 +607,27 @@ Notes: - `--max-idle-rounds` (default 3) and `--scroll-pause MIN MAX` (default `2.0 3.5`) are worth raising for an unusually large or slow-loading reels tab; the default stops once 3 consecutive scrolls find nothing new. -- **Not wired into `gdl-cron.sh` or the timers.** It drives your actual - browser session rather than firing a background request, and it is slower - by design (real scrolling, not an API call) — both good reasons not to run - it unattended without deciding that deliberately. Today it is a per-profile, - by-hand tool only. + +### Why dedup checks the whole archive, not just the scraped profile + +First version deduped only against the scraped profile's own directories. +Re-running it on `zindoriyam` minutes after a successful fetch found "9 new" +reels again — all reposts/collabs originally by `0ct0ber19` and +`official_artms` (also tracked profiles). The *old*, broken direct-reels-tab +fetch had left orphaned sidecar-only remnants (`.json`/`.txt`, no media) for +them, misfiled under `zindoriyam - reels/` with the true owner's name baked +into the filename stem — `index_existing()` correctly does not count a +sidecar-only entry as "held", so they looked new. `gdl-sync.py` re-fetched +them, filed them correctly under `{username}` (the post's *true* owner, per +its own metadata) — where they already existed from that profile's own +regular sync — and `rsync --ignore-existing` silently skipped every one, so +nothing was lost or duplicated. But 9 Instagram requests were spent finding +that out. Since a shortcode is globally unique, `reels-scrape.py` now checks +every archived profile's listing, not just the one being scraped — all local +requests to the viewer's own API, never to `instagram.com`, so checking all +16 profiles costs nothing on the budget that actually matters. Confirmed +fixed the same day: re-running against `zindoriyam` immediately afterward +found "26 already archived, 0 new" and exited clean. ## What changed on 2026-08-20 diff --git a/scripts/reels-scrape.py b/scripts/reels-scrape.py index 170206b..dc2e1e1 100644 --- a/scripts/reels-scrape.py +++ b/scripts/reels-scrape.py @@ -192,9 +192,21 @@ def main() -> int: spec.loader.exec_module(gdl) index = gdl.ArchiveIndex(args.index) - have = {code for code, _ in gdl.index_existing(index.listing(args.profile))} - print(f"archive already holds {len(have)} shortcode(s) for {args.profile}", - file=sys.stderr) + # Deduped against the WHOLE archive, not just this profile's own + # directories: a shortcode is globally unique, and a reels tab commonly + # surfaces reposts/collabs by OTHER tracked accounts. Missing that let a + # 2026-08-27 run re-fetch 9 reels already held under their true owner's + # directory -- gallery-dl filed them correctly there (by the post's real + # `username`, not the scraped profile), so nothing was lost, but it spent + # 9 avoidable Instagram requests to find that out. All of this is local + # requests to our own viewer, never to instagram.com, so checking every + # profile costs nothing on the budget that actually matters. + profiles = index.profiles() + have: set[str] = set() + for profile in profiles: + have.update(code for code, _ in gdl.index_existing(index.listing(profile))) + print(f"archive already holds {len(have)} shortcode(s) across " + f"{len(profiles)} profile(s)", file=sys.stderr) print(f"scraping https://www.instagram.com/{args.profile}/reels/ ...", file=sys.stderr) diff --git a/scripts/reels-sync.sh b/scripts/reels-sync.sh new file mode 100755 index 0000000..17740d4 --- /dev/null +++ b/scripts/reels-sync.sh @@ -0,0 +1,114 @@ +#!/bin/bash +# The full reels pipeline for one profile: scrape by scrolling the real page, +# then fetch and publish whatever's new. See TOOLING.md ("Reels: the API is +# blocked, scrape by scrolling instead") for why this exists at all -- the +# dedicated reels API is blocked, this drives the actual signed-in browser +# session instead, and is slower by design. +# +# Usage: ./reels-sync.sh +# ./reels-sync.sh zindoriyam +# ./reels-sync.sh https://www.instagram.com/zindoriyam/ +# +# Deliberately NOT wired into gdl-cron.sh or the timers -- see TOOLING.md. +# Exits non-zero if either step does. Everything is logged. +set -eu + +RAW="${1:?usage: reels-sync.sh }" + +GDL_HOME="${GDL_HOME:-$HOME/gdl}" +GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}" +INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}" +PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}" + +# Same hand-paced pacing gdl-cron.sh uses -- see its comment for why. Override +# per-run with GDL_SLEEP_REQUEST etc. if you ever need to, but raise them +# rather than lower them. +SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}" +SLEEP="${GDL_SLEEP:-5 10}" +RATE="${GDL_RATE:-500K}" +SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}" +MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}" + +PATH="$HOME/.local/bin:$PATH"; export PATH + +if [ ! -x "$GDL_PYTHON" ]; then + echo "reels-scrape.py needs gallery-dl's own pipx venv python (websocket-client" >&2 + echo "was injected there, not into the system python): $GDL_PYTHON not found" >&2 + exit 2 +fi + +# Same username-from-URL parsing gdl-sync.py already does for --urls-file, +# reused rather than re-implemented so the two never drift apart. +PROFILE=$("$GDL_PYTHON" -c " +import re, sys, importlib.util +from pathlib import Path +spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py')) +gdl = importlib.util.module_from_spec(spec) +sys.modules['gdl_sync'] = gdl +spec.loader.exec_module(gdl) +raw = '$RAW' +m = gdl.RE_PROFILE_URL.match(raw) +user = m.group('user') if m else raw.strip('/') +if not user or '/' in user or ' ' in user: + print(f'cannot read a username from {raw!r}', file=sys.stderr) + sys.exit(1) +print(user) +") + +mkdir -p "$GDL_HOME/logs" +LOG="$GDL_HOME/logs/reels-$PROFILE-$(date +%Y%m%d-%H%M%S)-$$.log" +URLS_FILE="$GDL_HOME/$PROFILE-reels.txt" +STAGING="$GDL_HOME/staging-reels-$PROFILE" + +echo "=== reels-sync $PROFILE $(date -Is) ===" | tee -a "$LOG" + +# The exit status has to survive the pipe into tee -- see gdl-cron.sh's +# comment on PIPESTATUS for why this needs to be bash, not sh. +set +e +"$GDL_PYTHON" "$GDL_HOME/reels-scrape.py" \ + --profile "$PROFILE" \ + --index "$INDEX" \ + --scroll-pause $SCROLL_PAUSE \ + --max-idle-rounds "$MAX_IDLE_ROUNDS" \ + --out "$URLS_FILE" 2>&1 | tee -a "$LOG" +scrape_status=${PIPESTATUS[0]} +set -e + +if [ "$scrape_status" -ne 0 ]; then + echo "=== exit $scrape_status (scrape failed) at $(date -Is) ===" | tee -a "$LOG" + exit "$scrape_status" +fi + +if [ ! -s "$URLS_FILE" ]; then + echo "no new reels for $PROFILE; nothing to fetch" | tee -a "$LOG" + echo "=== exit 0 at $(date -Is) ===" | tee -a "$LOG" + exit 0 +fi + +# Staging is wiped every run on purpose -- same reasoning as gdl-cron.sh: what +# we already hold is decided by the archive dedupe in reels-scrape.py, not by +# what happens to be sitting in staging. +rm -rf "$STAGING" + +set +e +# shellcheck disable=SC2086 +"$GDL_HOME/gdl-sync.py" \ + --publish "$PUBLISH" \ + --staging "$STAGING" \ + --post-urls-file "$URLS_FILE" \ + --sleep-request $SLEEP_REQUEST \ + --sleep $SLEEP \ + --rate "$RATE" \ + --execute 2>&1 | tee -a "$LOG" +status=${PIPESTATUS[0]} +set -e + +echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG" + +# Keep the log directory from growing without bound -- scoped to this +# script's own logs so it never touches gdl-cron.sh's rotation. +ls -1t "$GDL_HOME/logs" | grep '^reels-' | tail -n +30 | while read -r old; do + rm -f "$GDL_HOME/logs/$old" +done + +exit "$status"