feat: scrape reels by scrolling the real page, since the API is blocked
gallery-dl's dedicated reels extractor POSTs to /api/v1/clips/user/, which now 302-redirects for this account -- confirmed across multiple profiles, hours apart, with a freshly-warmed session and a correct X-IG-WWW-Claim header (ruled out as the cause). The reels tab itself loads fine in a real, already-signed-in browser, so reels-scrape.py drives that same Chrome via its loopback CDP port, scrolls the reels tab like a person would, and scrapes /reel/<code>/ links out of the rendered page instead of calling the blocked endpoint at all. It only finds shortcodes -- deduped against the archive via the same --index gdl-sync.py already uses -- and prints new post URLs. Feeding many of those into gdl-sync.py needed two small additions: a --post-urls-file so the list doesn't have to become a giant argv, and inter-item pacing in run_post_urls (each --post-url was its own subprocess with nothing pacing the gap between them). Verified end to end against zindoriyam: 26 reels found, 16 already archived, 10 new ones fetched and published cleanly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
This commit is contained in:
@@ -250,5 +250,23 @@ class UrlsFile(unittest.TestCase):
|
||||
self.assertEqual(gdl.read_urls_file(p), ["a", "b", "c", "d", "e"])
|
||||
|
||||
|
||||
class PostUrlsFile(unittest.TestCase):
|
||||
"""The format reels-scrape.py writes: whole URLs, not usernames."""
|
||||
|
||||
def test_skips_comments_blanks_and_duplicates(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
p = Path(d) / "reels.txt"
|
||||
p.write_text(
|
||||
"# scraped 2026-08-27\n"
|
||||
"https://www.instagram.com/u/reel/AAA/\n"
|
||||
"\n"
|
||||
"https://www.instagram.com/u/reel/BBB/\n"
|
||||
"https://www.instagram.com/u/reel/AAA/\n") # duplicate
|
||||
self.assertEqual(gdl.read_post_urls_file(p), [
|
||||
"https://www.instagram.com/u/reel/AAA/",
|
||||
"https://www.instagram.com/u/reel/BBB/",
|
||||
])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
||||
Reference in New Issue
Block a user