feat: add reels-sync.sh, and dedupe reels-scrape.py against the whole archive

reels-sync.sh is the single-command version of the two-step pipeline:
scrape a profile's reels tab, then fetch and publish whatever's new,
with the same hand-paced settings gdl-cron.sh uses. Takes a bare
username or a full profile URL. Exits clean without touching
gdl-sync.py at all when a profile has nothing new.

Also fixes a real inefficiency in reels-scrape.py's dedup, found by
running the new script twice in a row: checking only the scraped
profile's own directories missed that a shortcode already existed
under its true owner elsewhere in the archive (reposts/collabs by
other tracked accounts), so 9 already-held reels got re-fetched for no
reason. Shortcodes are globally unique, so dedup now checks every
archived profile's listing -- all local requests to the viewer's own
API, never instagram.com, so this costs nothing on the budget that
actually matters. See TOOLING.md for the full story.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
This commit is contained in:
2026-08-27 14:54:39 -04:00
co-authored by Claude Sonnet 5
parent 784ba43bbc
commit e0566f06ac
3 changed files with 180 additions and 20 deletions
+15 -3
View File
@@ -192,9 +192,21 @@ def main() -> int:
spec.loader.exec_module(gdl)
index = gdl.ArchiveIndex(args.index)
have = {code for code, _ in gdl.index_existing(index.listing(args.profile))}
print(f"archive already holds {len(have)} shortcode(s) for {args.profile}",
file=sys.stderr)
# Deduped against the WHOLE archive, not just this profile's own
# directories: a shortcode is globally unique, and a reels tab commonly
# surfaces reposts/collabs by OTHER tracked accounts. Missing that let a
# 2026-08-27 run re-fetch 9 reels already held under their true owner's
# directory -- gallery-dl filed them correctly there (by the post's real
# `username`, not the scraped profile), so nothing was lost, but it spent
# 9 avoidable Instagram requests to find that out. All of this is local
# requests to our own viewer, never to instagram.com, so checking every
# profile costs nothing on the budget that actually matters.
profiles = index.profiles()
have: set[str] = set()
for profile in profiles:
have.update(code for code, _ in gdl.index_existing(index.listing(profile)))
print(f"archive already holds {len(have)} shortcode(s) across "
f"{len(profiles)} profile(s)", file=sys.stderr)
print(f"scraping https://www.instagram.com/{args.profile}/reels/ ...",
file=sys.stderr)