ergosteur/instaarchive-viewer
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: stop enumerating a profile once it reaches what we already hold

Browse Source 
The skip-archive suppresses downloads, which spends the CDN. It does nothing
about the listing pass, which spends `instagram.com` — the surface that
actually bans accounts — and that cost scales with how BIG a profile is, not
with how much of it is new. A 2275-post profile paid ~76 pages every run to
discover three new posts. Seeding saved the second full pass, never the first.

Measured from sidecar write times during today's run, free because the run was
paying for the listing anyway: three new posts took ~100s each, and the other
2272 were written in a single second — enumeration with nothing to show for it.

`--abort N` passes gallery-dl's `skip: abort:N`, stopping the extractor after N
consecutive already-archived files. Resuming a stopped run with `--abort 50`
enumerated 7 posts of cher_ryppo's 2151 and still caught every new one.

Three things make this safe, and all of them are load-bearing:

- N counts FILES, not posts, so it has to clear the largest already-held
  carousel — one post in this archive is 22 media. A reels tab needs 50 actual
  reels for the same threshold, since those are single-media.
- It applies to posts and reels only. Stories are always new, and highlight
  items are not ordered in a way that makes early abort safe.
- The REST listing is strictly reverse-chronological. Test case 16 claimed
  0ct0ber19 returns its 3 pinned posts out of date order; that is true of the
  web grid but not of this endpoint, measured today. Front-loaded old posts are
  the one thing that would trip abort before it reached anything new, so the
  correction is what licenses the feature rather than a footnote to it.

Default is 0 — walk everything — because aborting early stops noticing edited
carousels (test case 15), which only a full enumeration finds. Routine runs
want 50; a full sweep is still worth running occasionally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
ergosteur
2026-08-20 14:31:33 -04:00
co-authored by Claude Opus 5
parent 01702ea24b
commit a83da461c1
2 changed files with 94 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gallery-dl.md
+66 -1
View File
@@ -359,6 +359,71 @@ The one trap, which silently seeds almost nothing if you get it backwards:
`live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
### The skip-archive saves the CDN, not `instagram.com`
Worth being exact about, because the two costs land on different surfaces and
only one of them bans accounts:
| what | which surface | scales with |
|---|---|---|
| downloading media | `scontent-*.cdninstagram.com` | how much is **new** |
| enumerating the profile to find it | `instagram.com` | how **big** the profile is |
The skip-archive suppresses the first. It does nothing about the second, so a
2275-post profile costs ~76 pages of pagination every run, forever, whether it
has three new posts or none. Seeding (above) saved a *second* full pass, not
the first.
Measured on the 2026-08-20 run, from sidecar write times in staging — free,
since the run was paying for the listing anyway:
```
1787248852 2026-08-19 … DcOeoVxkthi new, +0s
1787248944 2026-08-18 … DcLpfoJCZtp new, +92s
1787249058 2026-08-17 … DcIlGbxCUk0 new, +114s
1787249162 2026-07-24 … DbKr1TxlPSX ┐ all one second: nothing
1787249162 2026-08-15 … DcD-FdBCYGm ┘ downloaded, sidecars only
```
Three posts took ~100s each; the remaining 2272 were enumeration with nothing
to show for it.
**Pinned posts do not break early abort.** Test case 16 previously claimed
`0ct0ber19` returns its 3 pinned posts out of date order — that is true of the
*web grid*, but the REST `/posts/` listing came back strictly
reverse-chronological, newest first, no hoisting. That matters because
front-loaded old posts are the one thing that would make `skip: abort:N`
dangerous: it would trip on them and abort before reaching anything new.
So `skip: abort:N` is viable, and cuts ~420 requests per run to ~40-60:
| surface | live items | pages | with `abort:50` |
|---|---:|---:|---:|
| posts, 6 profiles | 11,248 | ~377 | ~12 |
| reels, 6 profiles | 1,080 | ~24 | ~8 |
| stories + highlights | — | ~20 | ~20 |
N counts consecutive skipped **files**, not posts, so it must clear the largest
already-held carousel — `DcD-FdBCYGm` alone is 22 media. 50 is comfortable; 5
would not be.
**The tradeoff is edited carousels.** Test case 15 is a post that gained items
after we archived it, and only a full enumeration finds those. Suggested
policy: `abort:50` for routine runs, a full sweep occasionally.
Measured the same day, resuming a stopped run with `--abort 50`:
| source | live items | enumerated |
|---|---:|---:|
| `cher_ryppo` posts | 2,151 | **7** |
| `cher_ryppo` reels | 92 | 53 |
One page instead of 72, and every new post was still caught. The 7 is roughly
3 new posts plus 4 already-held carousels making up the 50 skipped files.
Reels need 53 because they are single-media, so 50 consecutive skips really is
50 reels — another reminder that N counts files, and that the same N behaves
very differently on a carousel-heavy surface than on a reels tab.
## Publishing
The fetch host stages to local disk and rsyncs afterwards. `rsync
@@ -493,7 +558,7 @@ Real subjects, all present in the archive today. See
| 13 | empty caption | `CrdsY5CrSsO` | media written, `.txt` absent |
| 14 | deleted post | `C0TgI7sphfZ` | on disk, absent live — must not be removed |
| 15 | edited carousel | `C7zG7-jJMlq` | 18 on disk, 8 live — must not be removed |
| 16 | pinned posts | `0ct0ber19` | 3 pinned, returned out of date order |
| 16 | pinned posts | `0ct0ber19` | REST listing is strictly reverse-chronological; see below |
| 17 | profile avatar | `0ct0ber19.jpg` | base dir, undated |
Cases 14–16 are reconciliation, not naming: **a sync must never delete**, since