From 5148b677cef7e1266c071e02dac48d5a953fdfca Mon Sep 17 00:00:00 2001 From: ergosteur Date: Mon, 14 Sep 2026 02:03:37 -0400 Subject: [PATCH] docs: record that the reels/posts API block is fixed upstream gallery-dl 1.32.12 (we were on 1.32.9, three releases behind) resolved the "HTTP redirect to home page" failures on both posts and reels -- its changelog cites exactly this ("posts & reels extractors... content retrieval failures"). Verified with a full run-profiles.sh across all six profiles: zero redirect errors, first time since 2026-08-26. The browser-scrolling workaround (reels-scrape.py/reels-sync.sh/ run-reels-all.sh) is no longer needed for routine use as a result, but stays in place as a fallback. Updated the quick-reference commands to drop run-reels-all.sh from the routine list; kept the original diagnosis section intact as history, with a note at the top pointing to the fix. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk --- TOOLING.md | 52 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 17 deletions(-) diff --git a/TOOLING.md b/TOOLING.md index d975720..7e3c353 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -31,21 +31,21 @@ with no deploy or restart. ssh mattellite ~/gdl/run-stories.sh # ~1 min. DO THIS OFTEN - stories die in 24h -~/gdl/run-profiles.sh # ~5 min. posts, highlights, stories (not reels -- see below) -~/gdl/run-reels-all.sh # ~15-20 min. reels, all six profiles, by scrolling +~/gdl/run-profiles.sh # ~5 min. posts, reels, highlights, stories ~/gdl/run-full-sweep.sh # rarely, ~420 requests. Read its own comment first. ``` -Fixed-mode entry points around `gdl-cron.sh`/`reels-sync.sh`, so the mode -never has to be remembered or typed — each just runs its own thing with the -right flags already set. `run-profiles.sh` covers every surface gallery-dl's API -still reaches, so it is the one to run if you only run one, but it is no -substitute for `run-stories.sh` (a story posted and expired between two runs of -it is simply gone) or `run-reels-all.sh` (`/api/v1/clips/user/` is blocked -- -see "Reels: the API is blocked" below -- `run-profiles.sh` silently fails that -one surface every time, by design, until that changes). All four are safe to -run back-to-back; the `--min-interval` floor makes a repeat a no-op rather -than a re-fetch. +Fixed-mode entry points around `gdl-cron.sh`, so the mode never has to be +remembered or typed — each just runs its own thing with the right flags +already set. `run-profiles.sh` covers every surface, so it is the one to run +if you only run one, but it is no substitute for `run-stories.sh`: a story +posted and expired between two `run-profiles.sh` runs is simply gone. Both +are safe to run back-to-back; the `--min-interval` floor makes a repeat a +no-op rather than a re-fetch. + +**`run-reels-all.sh` (browser-scrolling) is no longer needed for routine use +as of 2026-09-14** — see "Reels: the API is blocked" below, now fixed +upstream. `run-profiles.sh`'s own `reels` surface covers it again. Then check — new posts appear without a redeploy. The archive index re-reads a directory when its mtime changes. @@ -540,14 +540,32 @@ before something worse. ## Reels: the API is blocked, scrape by scrolling instead -As of 2026-08-26/27, `gallery-dl`'s dedicated reels extractor fails on every +**Fixed upstream 2026-09-14 — read this note before the rest of the section.** +`gallery-dl` 1.32.12 (released 2026-09-12; we were on 1.32.9) resolved it: its +changelog cites "posts & reels extractors... content retrieval failures", and +upgrading (`pipx upgrade gallery-dl`) confirmed it -- a full `run-profiles.sh` +across all six profiles came back with **zero** `HTTP redirect` errors on +posts or reels, first time since 2026-08-26. `reels-scrape.py`/`reels-sync.sh`/ +`run-reels-all.sh` are no longer needed for routine use; left in place as a +fallback (and because the whole-archive dedup fix they picked up along the +way, see below, is independently useful). If this regresses, check the +gallery-dl changelog for a newer release before re-reaching for the scroll +workaround -- it may already be fixed there too. + +The rest of this section is kept as-is: the diagnosis, the workaround, and +what it taught along the way are all still accurate history, just no longer +the operative fix. + +As of 2026-08-26/27, `gallery-dl`'s dedicated reels extractor failed on every profile with `HTTP redirect to home page` — confirmed hours apart, with a freshly-warmed session and a correct `X-IG-WWW-Claim` header (that was the -first suspect; ruled out by tracing the raw HTTP exchange). It is not a -scraping-warning interstitial — the reels tab loads completely normally in a -real, already-signed-in browser — so this is Meta blocking the specific +first suspect; ruled out by tracing the raw HTTP exchange). It was not a +scraping-warning interstitial — the reels tab loaded completely normally in a +real, already-signed-in browser — so this was Meta blocking the specific `/api/v1/clips/user/` endpoint gallery-dl calls, not an account-health issue. -Posts, highlights and stories are unaffected; only reels breaks. +Posts, highlights and stories were unaffected at the time; only reels broke +(posts later broke too, some time before 2026-09-13, in the same way — same +fix resolved both). `scripts/reels-scrape.py` works around it by never calling that endpoint: it drives the same signed-in Chrome via its loopback CDP port (`:9222`,