gallery-dl skips already-held media either by file existence -- which
requires the archive mounted where it writes -- or by a sqlite
skip-archive, which requires nothing on disk. Using the latter lets the
fetch host write to local disk and rsync afterwards, avoiding tens of
thousands of small writes over CIFS and keeping a mid-sync failure from
leaving partial files on the live Resilio share.
The key is archive_prefix + archive_fmt: the literal "instagram" plus the
per-media numeric pk. Verified against a real run -- a 3-image carousel
produced 3 rows and a re-run skipped every media file.
media_id is absent from our filenames, so the DB cannot be built from
names alone, but the listing pass we already make maps every live item to
its media_id, and a file listing says which we hold. Seeding therefore
costs no extra Instagram requests and no archive content -- the listing
GET /api/archives/:name/files already serves is enough.
Measured on 0ct0ber19: 2275 live items, 2248 seeded, 27 left to fetch --
exactly the media of the two posts added since the last crawl.
The trap worth the comment it carries: posts and reels are filed under
post_shortcode, while stories and highlights use the per-item shortcode
(post_shortcode there is the containing reel's id, shared by every item).
Matching on the wrong field seeded 5 of 2275 rather than failing loudly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every claim in docs/gallery-dl.md was measured against the live site and
the archive rather than taken from documentation, because two of the
assumptions turned out to be wrong.
The safety model is the reason the config looks the way it does.
gallery-dl has two API backends: the graphql one issues a request PER
POST for every video and carousel -- the pattern that got this account
banned via Instaloader -- while the default rest one paginates listings
at 30-50 items and carries carousel_media, video_versions and
product_type inline. A 300-post profile costs ~10 requests.
Findings worth recording:
- JD2 stamped filenames in desktop LOCAL time (US Eastern), not UTC.
Across 212 comparable posts: UTC 19 mismatches, UTC-5 10, UTC-4 zero.
{date:Olocal/%Y-%m-%d} reproduces it; the trailing separator must be
omitted or it lands in the strftime format.
- A profile's reels tab returns collab reels owned by OTHER accounts, so
the directory must be forced with -D. JD2 did the same: chuuo3o and
official_artms filenames sit inside "0ct0ber19 - reels".
- Stories and highlights need per-item {shortcode}; {post_shortcode} is
the reel's id and is shared by every item. {date} is per-item, verified
on a 154-item highlight with distinct times.
- gallery-dl reproduces JD2's caption .txt exactly, including writing
nothing for an empty caption and omitting the trailing newline.
- The json sidecar needs `include`, not `fields`; `fields` silently does
nothing in mode:json and leaks audio_user blobs. It yields `type`
(post/reel) -- Instagram's own flag, which can retire the lone-video
heuristic once the scanner reads it.
Naming differences between the two tools are cosmetic: EXPORT_RE already
makes the index optional and parseInt normalises zero-padding, so a mixed
archive parses identically. Tests pin that down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ergosteurandClaude Opus 5