feat: add reels-sync.sh, and dedupe reels-scrape.py against the whole archive
reels-sync.sh is the single-command version of the two-step pipeline: scrape a profile's reels tab, then fetch and publish whatever's new, with the same hand-paced settings gdl-cron.sh uses. Takes a bare username or a full profile URL. Exits clean without touching gdl-sync.py at all when a profile has nothing new. Also fixes a real inefficiency in reels-scrape.py's dedup, found by running the new script twice in a row: checking only the scraped profile's own directories missed that a shortcode already existed under its true owner elsewhere in the archive (reposts/collabs by other tracked accounts), so 9 already-held reels got re-fetched for no reason. Shortcodes are globally unique, so dedup now checks every archived profile's listing -- all local requests to the viewer's own API, never instagram.com, so this costs nothing on the budget that actually matters. See TOOLING.md for the full story. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
This commit is contained in:
+15
-3
@@ -192,9 +192,21 @@ def main() -> int:
|
||||
spec.loader.exec_module(gdl)
|
||||
|
||||
index = gdl.ArchiveIndex(args.index)
|
||||
have = {code for code, _ in gdl.index_existing(index.listing(args.profile))}
|
||||
print(f"archive already holds {len(have)} shortcode(s) for {args.profile}",
|
||||
file=sys.stderr)
|
||||
# Deduped against the WHOLE archive, not just this profile's own
|
||||
# directories: a shortcode is globally unique, and a reels tab commonly
|
||||
# surfaces reposts/collabs by OTHER tracked accounts. Missing that let a
|
||||
# 2026-08-27 run re-fetch 9 reels already held under their true owner's
|
||||
# directory -- gallery-dl filed them correctly there (by the post's real
|
||||
# `username`, not the scraped profile), so nothing was lost, but it spent
|
||||
# 9 avoidable Instagram requests to find that out. All of this is local
|
||||
# requests to our own viewer, never to instagram.com, so checking every
|
||||
# profile costs nothing on the budget that actually matters.
|
||||
profiles = index.profiles()
|
||||
have: set[str] = set()
|
||||
for profile in profiles:
|
||||
have.update(code for code, _ in gdl.index_existing(index.listing(profile)))
|
||||
print(f"archive already holds {len(have)} shortcode(s) across "
|
||||
f"{len(profiles)} profile(s)", file=sys.stderr)
|
||||
|
||||
print(f"scraping https://www.instagram.com/{args.profile}/reels/ ...",
|
||||
file=sys.stderr)
|
||||
|
||||
Reference in New Issue
Block a user