feat: scrape reels by scrolling the real page, since the API is blocked

gallery-dl's dedicated reels extractor POSTs to /api/v1/clips/user/,
which now 302-redirects for this account -- confirmed across multiple
profiles, hours apart, with a freshly-warmed session and a correct
X-IG-WWW-Claim header (ruled out as the cause). The reels tab itself
loads fine in a real, already-signed-in browser, so reels-scrape.py
drives that same Chrome via its loopback CDP port, scrolls the reels
tab like a person would, and scrapes /reel/<code>/ links out of the
rendered page instead of calling the blocked endpoint at all.

It only finds shortcodes -- deduped against the archive via the same
--index gdl-sync.py already uses -- and prints new post URLs. Feeding
many of those into gdl-sync.py needed two small additions: a
--post-urls-file so the list doesn't have to become a giant argv, and
inter-item pacing in run_post_urls (each --post-url was its own
subprocess with nothing pacing the gap between them).

Verified end to end against zindoriyam: 26 reels found, 16 already
archived, 10 new ones fetched and published cleanly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
This commit is contained in:
2026-08-27 14:36:09 -04:00
co-authored by Claude Sonnet 5
parent eceee6ec02
commit 2f46123022
4 changed files with 327 additions and 3 deletions
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
"""
Tests for the pure parts of reels-scrape.py -- everything except the actual
CDP session, which needs a live signed-in Chrome and is exercised by hand.
python3 -m unittest discover -s scripts -p 'test_*.py'
"""
import importlib.util
import sys
import unittest
from pathlib import Path
_spec = importlib.util.spec_from_file_location(
"reels_scrape", Path(__file__).with_name("reels-scrape.py"))
scrape = importlib.util.module_from_spec(_spec)
sys.modules["reels_scrape"] = scrape
_spec.loader.exec_module(scrape)
class ExtractShortcodes(unittest.TestCase):
def test_reads_relative_and_absolute_hrefs(self):
hrefs = [
"/someuser/reel/ABC123/",
"https://www.instagram.com/someuser/reel/DEF456/",
"/reel/GHI789/?img_index=1",
]
self.assertEqual(scrape.extract_shortcodes(hrefs),
["ABC123", "DEF456", "GHI789"])
def test_ignores_non_reel_links(self):
hrefs = ["/someuser/", "/someuser/p/ABC123/", "/explore/tags/foo/"]
self.assertEqual(scrape.extract_shortcodes(hrefs), [])
def test_empty_input(self):
self.assertEqual(scrape.extract_shortcodes([]), [])
if __name__ == "__main__":
unittest.main(verbosity=2)