feat: stop enumerating a profile once it reaches what we already hold
The skip-archive suppresses downloads, which spends the CDN. It does nothing about the listing pass, which spends `instagram.com` — the surface that actually bans accounts — and that cost scales with how BIG a profile is, not with how much of it is new. A 2275-post profile paid ~76 pages every run to discover three new posts. Seeding saved the second full pass, never the first. Measured from sidecar write times during today's run, free because the run was paying for the listing anyway: three new posts took ~100s each, and the other 2272 were written in a single second — enumeration with nothing to show for it. `--abort N` passes gallery-dl's `skip: abort:N`, stopping the extractor after N consecutive already-archived files. Resuming a stopped run with `--abort 50` enumerated 7 posts of cher_ryppo's 2151 and still caught every new one. Three things make this safe, and all of them are load-bearing: - N counts FILES, not posts, so it has to clear the largest already-held carousel — one post in this archive is 22 media. A reels tab needs 50 actual reels for the same threshold, since those are single-media. - It applies to posts and reels only. Stories are always new, and highlight items are not ordered in a way that makes early abort safe. - The REST listing is strictly reverse-chronological. Test case 16 claimed 0ct0ber19 returns its 3 pinned posts out of date order; that is true of the web grid but not of this endpoint, measured today. Front-loaded old posts are the one thing that would trip abort before it reached anything new, so the correction is what licenses the feature rather than a footnote to it. Default is 0 — walk everything — because aborting early stops noticing edited carousels (test case 15), which only a full enumeration finds. Routine runs want 50; a full sweep is still worth running occasionally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
+28
-2
@@ -219,7 +219,7 @@ class ArchiveIndex:
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
def build_config(rate: str, sleep_request: list[float],
|
||||
sleep: list[float]) -> dict:
|
||||
sleep: list[float], abort: int = 0) -> dict:
|
||||
"""
|
||||
The config is generated rather than checked in so the safety-critical
|
||||
options cannot drift out of sync with the docs.
|
||||
@@ -253,7 +253,26 @@ def build_config(rate: str, sleep_request: list[float],
|
||||
|
||||
def post_like(stem: str) -> dict:
|
||||
"""Naming for surfaces whose unit is a post (posts, reels)."""
|
||||
skip: dict = {}
|
||||
if abort:
|
||||
# Stop enumerating once `abort` consecutive files are already in
|
||||
# the skip-archive. The listing pass -- not the downloading -- is
|
||||
# what costs `instagram.com` requests, and it otherwise walks the
|
||||
# whole profile every run to find three new posts.
|
||||
#
|
||||
# Safe here only because the REST listing is strictly
|
||||
# reverse-chronological: the web grid hoists pinned posts to the
|
||||
# front, but this endpoint does not (measured 2026-08-20), so old
|
||||
# posts never appear before new ones.
|
||||
#
|
||||
# Counted in FILES, not posts, so it must clear the largest
|
||||
# already-held carousel -- 22 media for one real post in this
|
||||
# archive. It also means edited carousels (test case 15) stop
|
||||
# being noticed, so a full sweep is still worth running
|
||||
# occasionally.
|
||||
skip["skip"] = f"abort:{abort}"
|
||||
return {
|
||||
**skip,
|
||||
# `sidecar_shortcode` is set only for carousels, so it is the
|
||||
# carousel discriminator. First matching condition wins.
|
||||
"filename": {
|
||||
@@ -666,6 +685,12 @@ def main() -> int:
|
||||
ap.add_argument("--max-sources", type=int, default=0, metavar="N",
|
||||
help="hard ceiling on sources touched in one run "
|
||||
"(0 = no limit)")
|
||||
ap.add_argument("--abort", type=int, default=0, metavar="N",
|
||||
help="stop enumerating posts/reels after N consecutive "
|
||||
"already-archived FILES (0 = walk everything, the "
|
||||
"default). 50 is a safe routine value; it cuts the "
|
||||
"per-run listing cost by roughly 85%%, at the price "
|
||||
"of no longer noticing edited carousels")
|
||||
ap.add_argument("--probe-ttl", type=float, default=24.0, metavar="HOURS",
|
||||
help="reuse cached listing results younger than this")
|
||||
ap.add_argument("--force", action="store_true",
|
||||
@@ -704,7 +729,8 @@ def main() -> int:
|
||||
else:
|
||||
selected = [Profile(p) for p in sorted(names)]
|
||||
|
||||
config = build_config(args.rate, list(args.sleep_request), list(args.sleep))
|
||||
config = build_config(args.rate, list(args.sleep_request),
|
||||
list(args.sleep), args.abort)
|
||||
args.staging.mkdir(parents=True, exist_ok=True)
|
||||
# Deliberately a SIBLING of the staging directory, not inside it: staging is
|
||||
# rsynced wholesale into the archive, and a dry run caught this file being
|
||||
|
||||
Reference in New Issue
Block a user