feat: scrape reels by scrolling the real page, since the API is blocked

gallery-dl's dedicated reels extractor POSTs to /api/v1/clips/user/,
which now 302-redirects for this account -- confirmed across multiple
profiles, hours apart, with a freshly-warmed session and a correct
X-IG-WWW-Claim header (ruled out as the cause). The reels tab itself
loads fine in a real, already-signed-in browser, so reels-scrape.py
drives that same Chrome via its loopback CDP port, scrolls the reels
tab like a person would, and scrapes /reel/<code>/ links out of the
rendered page instead of calling the blocked endpoint at all.

It only finds shortcodes -- deduped against the archive via the same
--index gdl-sync.py already uses -- and prints new post URLs. Feeding
many of those into gdl-sync.py needed two small additions: a
--post-urls-file so the list doesn't have to become a giant argv, and
inter-item pacing in run_post_urls (each --post-url was its own
subprocess with nothing pacing the gap between them).

Verified end to end against zindoriyam: 26 reels found, 16 already
archived, 10 new ones fetched and published cleanly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
This commit is contained in:
2026-08-27 14:36:09 -04:00
co-authored by Claude Sonnet 5
parent eceee6ec02
commit 2f46123022
4 changed files with 327 additions and 3 deletions
+18
View File
@@ -250,5 +250,23 @@ class UrlsFile(unittest.TestCase):
self.assertEqual(gdl.read_urls_file(p), ["a", "b", "c", "d", "e"])
class PostUrlsFile(unittest.TestCase):
"""The format reels-scrape.py writes: whole URLs, not usernames."""
def test_skips_comments_blanks_and_duplicates(self):
with tempfile.TemporaryDirectory() as d:
p = Path(d) / "reels.txt"
p.write_text(
"# scraped 2026-08-27\n"
"https://www.instagram.com/u/reel/AAA/\n"
"\n"
"https://www.instagram.com/u/reel/BBB/\n"
"https://www.instagram.com/u/reel/AAA/\n") # duplicate
self.assertEqual(gdl.read_post_urls_file(p), [
"https://www.instagram.com/u/reel/AAA/",
"https://www.instagram.com/u/reel/BBB/",
])
if __name__ == "__main__":
unittest.main(verbosity=2)