ergosteurandClaude Opus 5 4fee8b1dfe feat: seed gallery-dl's skip-archive so the fetcher needs no archive copy ...

gallery-dl skips already-held media either by file existence -- which
requires the archive mounted where it writes -- or by a sqlite
skip-archive, which requires nothing on disk. Using the latter lets the
fetch host write to local disk and rsync afterwards, avoiding tens of
thousands of small writes over CIFS and keeping a mid-sync failure from
leaving partial files on the live Resilio share.

The key is archive_prefix + archive_fmt: the literal "instagram" plus the
per-media numeric pk. Verified against a real run -- a 3-image carousel
produced 3 rows and a re-run skipped every media file.

media_id is absent from our filenames, so the DB cannot be built from
names alone, but the listing pass we already make maps every live item to
its media_id, and a file listing says which we hold. Seeding therefore
costs no extra Instagram requests and no archive content -- the listing
GET /api/archives/:name/files already serves is enough.

Measured on 0ct0ber19: 2275 live items, 2248 seeded, 27 left to fetch --
exactly the media of the two posts added since the last crawl.

The trap worth the comment it carries: posts and reels are filed under
post_shortcode, while stories and highlights use the per-item shortcode
(post_shortcode there is the containing reel's id, shared by every item).
Matching on the wrong field seeded 5 of 2275 rather than failing loudly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-16 21:10:37 -04:00

20 KiB

Raw History

View Raw