feat: seed gallery-dl's skip-archive so the fetcher needs no archive copy
gallery-dl skips already-held media either by file existence -- which requires the archive mounted where it writes -- or by a sqlite skip-archive, which requires nothing on disk. Using the latter lets the fetch host write to local disk and rsync afterwards, avoiding tens of thousands of small writes over CIFS and keeping a mid-sync failure from leaving partial files on the live Resilio share. The key is archive_prefix + archive_fmt: the literal "instagram" plus the per-media numeric pk. Verified against a real run -- a 3-image carousel produced 3 rows and a re-run skipped every media file. media_id is absent from our filenames, so the DB cannot be built from names alone, but the listing pass we already make maps every live item to its media_id, and a file listing says which we hold. Seeding therefore costs no extra Instagram requests and no archive content -- the listing GET /api/archives/:name/files already serves is enough. Measured on 0ct0ber19: 2275 live items, 2248 seeded, 27 left to fetch -- exactly the media of the two posts added since the last crawl. The trap worth the comment it carries: posts and reels are filed under post_shortcode, while stories and highlights use the per-item shortcode (post_shortcode there is the containing reel's id, shared by every item). Matching on the wrong field seeded 5 of 2275 rather than failing loudly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -204,6 +204,53 @@ viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
|
||||
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
|
||||
only matters when testing by hand.
|
||||
|
||||
## Incremental sync — why the fetch host needs no copy of the archive
|
||||
|
||||
gallery-dl can skip already-held media two ways, and the difference decides
|
||||
whether the fetcher needs the archive mounted:
|
||||
|
||||
- **By file existence** (default). Needs the destination to already contain the
|
||||
files, so it only works if the archive is mounted where gallery-dl writes.
|
||||
- **By skip-archive** (`--download-archive`). A sqlite DB of ids. Needs nothing
|
||||
on disk.
|
||||
|
||||
We use the second, so the fetch host can write to **local disk and rsync
|
||||
afterwards**. That avoids writing tens of thousands of small files over CIFS,
|
||||
and keeps a mid-sync failure from leaving partial files on the live Resilio
|
||||
share.
|
||||
|
||||
The key is `archive_prefix + archive_fmt`, which for this extractor is the
|
||||
literal `instagram` plus the per-media numeric pk (`instagram.py:25`,
|
||||
`job.py:713-719`). Verified: a 3-image carousel produced
|
||||
|
||||
```
|
||||
instagram3079387627521318672
|
||||
instagram3079387627521429433
|
||||
instagram3079387627529716672
|
||||
```
|
||||
|
||||
and a second run skipped every media file, rewriting only the idempotent
|
||||
`.txt`/`.json` sidecars.
|
||||
|
||||
**Seeding.** `media_id` is not in our filenames, so the DB cannot be built from
|
||||
names alone — but one listing pass (the pass we make anyway) maps every live
|
||||
item to its `media_id`, and the archive's *file listing* says which we already
|
||||
hold. No extra Instagram requests, and no archive content — a listing is
|
||||
enough, which `GET /api/archives/:name/files` already serves.
|
||||
|
||||
Measured on `0ct0ber19`: 2275 live media items, 2248 seeded from the existing
|
||||
listing, **27 left to download** — precisely the media of the two posts added
|
||||
since the last crawl.
|
||||
|
||||
The one trap, which silently seeds almost nothing if you get it backwards:
|
||||
|
||||
| surface | filed under | why |
|
||||
|---|---|---|
|
||||
| posts, reels | `post_shortcode` | carousel children each have their own `shortcode`, which never appears in a filename |
|
||||
| stories, highlights | `shortcode` (per item) | `post_shortcode` is the containing reel's id, shared by every item |
|
||||
|
||||
`live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
|
||||
|
||||
## Known quirks
|
||||
|
||||
- **`count` is not the emitted file count.** For 135 of 214 posts it was exactly
|
||||
|
||||
Reference in New Issue
Block a user