feat: seed gallery-dl's skip-archive so the fetcher needs no archive copy

gallery-dl skips already-held media either by file existence -- which
requires the archive mounted where it writes -- or by a sqlite
skip-archive, which requires nothing on disk. Using the latter lets the
fetch host write to local disk and rsync afterwards, avoiding tens of
thousands of small writes over CIFS and keeping a mid-sync failure from
leaving partial files on the live Resilio share.

The key is archive_prefix + archive_fmt: the literal "instagram" plus the
per-media numeric pk. Verified against a real run -- a 3-image carousel
produced 3 rows and a re-run skipped every media file.

media_id is absent from our filenames, so the DB cannot be built from
names alone, but the listing pass we already make maps every live item to
its media_id, and a file listing says which we hold. Seeding therefore
costs no extra Instagram requests and no archive content -- the listing
GET /api/archives/:name/files already serves is enough.

Measured on 0ct0ber19: 2275 live items, 2248 seeded, 27 left to fetch --
exactly the media of the two posts added since the last crawl.

The trap worth the comment it carries: posts and reels are filed under
post_shortcode, while stories and highlights use the per-item shortcode
(post_shortcode there is the containing reel's id, shared by every item).
Matching on the wrong field seeded 5 of 2275 rather than failing loudly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-16 21:10:37 -04:00
co-authored by Claude Opus 5
parent 7b63ba6a76
commit 4fee8b1dfe
3 changed files with 146 additions and 11 deletions
+47
View File
@@ -204,6 +204,53 @@ viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
only matters when testing by hand.
## Incremental sync — why the fetch host needs no copy of the archive
gallery-dl can skip already-held media two ways, and the difference decides
whether the fetcher needs the archive mounted:
- **By file existence** (default). Needs the destination to already contain the
files, so it only works if the archive is mounted where gallery-dl writes.
- **By skip-archive** (`--download-archive`). A sqlite DB of ids. Needs nothing
on disk.
We use the second, so the fetch host can write to **local disk and rsync
afterwards**. That avoids writing tens of thousands of small files over CIFS,
and keeps a mid-sync failure from leaving partial files on the live Resilio
share.
The key is `archive_prefix + archive_fmt`, which for this extractor is the
literal `instagram` plus the per-media numeric pk (`instagram.py:25`,
`job.py:713-719`). Verified: a 3-image carousel produced
```
instagram3079387627521318672
instagram3079387627521429433
instagram3079387627529716672
```
and a second run skipped every media file, rewriting only the idempotent
`.txt`/`.json` sidecars.
**Seeding.** `media_id` is not in our filenames, so the DB cannot be built from
names alone — but one listing pass (the pass we make anyway) maps every live
item to its `media_id`, and the archive's *file listing* says which we already
hold. No extra Instagram requests, and no archive content — a listing is
enough, which `GET /api/archives/:name/files` already serves.
Measured on `0ct0ber19`: 2275 live media items, 2248 seeded from the existing
listing, **27 left to download** — precisely the media of the two posts added
since the last crawl.
The one trap, which silently seeds almost nothing if you get it backwards:
| surface | filed under | why |
|---|---|---|
| posts, reels | `post_shortcode` | carousel children each have their own `shortcode`, which never appears in a filename |
| stories, highlights | `shortcode` (per item) | `post_shortcode` is the containing reel's id, shared by every item |
`live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
## Known quirks
- **`count` is not the emitted file count.** For 135 of 214 posts it was exactly