docs: design a gallery-dl replacement for the JDownloader fetcher
Every claim in docs/gallery-dl.md was measured against the live site and
the archive rather than taken from documentation, because two of the
assumptions turned out to be wrong.
The safety model is the reason the config looks the way it does.
gallery-dl has two API backends: the graphql one issues a request PER
POST for every video and carousel -- the pattern that got this account
banned via Instaloader -- while the default rest one paginates listings
at 30-50 items and carries carousel_media, video_versions and
product_type inline. A 300-post profile costs ~10 requests.
Findings worth recording:
- JD2 stamped filenames in desktop LOCAL time (US Eastern), not UTC.
Across 212 comparable posts: UTC 19 mismatches, UTC-5 10, UTC-4 zero.
{date:Olocal/%Y-%m-%d} reproduces it; the trailing separator must be
omitted or it lands in the strftime format.
- A profile's reels tab returns collab reels owned by OTHER accounts, so
the directory must be forced with -D. JD2 did the same: chuuo3o and
official_artms filenames sit inside "0ct0ber19 - reels".
- Stories and highlights need per-item {shortcode}; {post_shortcode} is
the reel's id and is shared by every item. {date} is per-item, verified
on a 154-item highlight with distinct times.
- gallery-dl reproduces JD2's caption .txt exactly, including writing
nothing for an empty caption and omitting the trailing newline.
- The json sidecar needs `include`, not `fields`; `fields` silently does
nothing in mode:json and leaks audio_user blobs. It yields `type`
(post/reel) -- Instagram's own flag, which can retire the lone-video
heuristic once the scanner reads it.
Naming differences between the two tools are cosmetic: EXPORT_RE already
makes the index optional and parseInt normalises zero-padding, so a mixed
archive parses identically. Tests pin that down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,263 @@
|
||||
# gallery-dl — a CLI replacement for JDownloader2
|
||||
|
||||
Status: **design + verified config.** `scripts/gdl-sync.py` is a skeleton; no
|
||||
profile has been migrated yet.
|
||||
|
||||
Everything below was measured against the live site and the real archive on
|
||||
2026-08-16, not inferred from documentation.
|
||||
|
||||
## Why gallery-dl and not a hand-rolled script
|
||||
|
||||
The hard parts of fetching Instagram are pagination, cookie handling, CDN URL
|
||||
expiry and resumption. gallery-dl already has all of them, plus extractors that
|
||||
map 1:1 onto our sidecar directory layout (`posts`, `reels`, `stories`,
|
||||
`highlights`). Rolling our own would mean reimplementing the ban-sensitive part
|
||||
by hand.
|
||||
|
||||
## The safety model — read this before changing any option
|
||||
|
||||
The ban vector is **requests to `instagram.com`**, not bandwidth. See
|
||||
`docs/jdownloader.md` for the history; Instaloader got this account banned by
|
||||
asking `instagram.com` a question *per post*.
|
||||
|
||||
gallery-dl has two API backends and the difference is exactly that vector:
|
||||
|
||||
```python
|
||||
if self.config("api") == "graphql":
|
||||
self.api = InstagramGraphqlAPI(self) # per-post api.media() for every
|
||||
else: # video and every carousel
|
||||
self.api = InstagramRestAPI(self) # <- default, listing-only
|
||||
```
|
||||
|
||||
The REST backend paginates at `count: 30` (feed) / `page_size: 50` (clips), and
|
||||
those responses already carry `carousel_media`, `image_versions2`,
|
||||
`video_versions` and `product_type`. **No per-post request.** A 300-post
|
||||
profile costs roughly 10 requests to `instagram.com`.
|
||||
|
||||
Rules, in order of importance:
|
||||
|
||||
1. **`"api": "rest"` always.** Never `graphql`. This is the whole ballgame.
|
||||
2. **Never enable `metadata`-style options that trigger extra calls.** If a
|
||||
field is not already in the listing response, it is not worth a request.
|
||||
3. **Pace it.** `"sleep-request": [4.0, 7.0]` — a randomised gap, not a fixed
|
||||
one. Also `"sleep": [1.0, 3.0]` between downloads.
|
||||
4. **Cap the download rate** (`downloader.http.rate`) so the CDN side looks like
|
||||
a person, not a mirror.
|
||||
5. **Run from the same public IP as the browser the cookie came from.** At time
|
||||
of writing that is `mattellite` (`66.23.52.196`); the dev workstation is a
|
||||
*different* public IP and using the cookie from there is precisely what
|
||||
session-hijack detection looks for.
|
||||
6. **No programmatic login, ever.** gallery-dl's username/password path is
|
||||
disabled upstream anyway; use `--cookies-from-browser`.
|
||||
|
||||
Do not add proxy rotation, fingerprint spoofing or account rotation. Throttling
|
||||
and request-avoidance are welcome; evasion is not.
|
||||
|
||||
### Cookies
|
||||
|
||||
The logged-in Chrome on `mattellite` runs with a non-default profile:
|
||||
|
||||
```
|
||||
--user-data-dir=/home/matt/.config/google-chrome-devtools
|
||||
```
|
||||
|
||||
so the cookie flag is:
|
||||
|
||||
```
|
||||
--cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools"
|
||||
```
|
||||
|
||||
Plain `--cookies-from-browser chrome` fails with "Unable to find chrome cookies
|
||||
database" because it looks in `~/.config/google-chrome/`.
|
||||
|
||||
Anonymous access is **not** a viable fallback: it serves lower-resolution media,
|
||||
caps profile pagination at 12 posts, and returns `AuthRequired` for stories and
|
||||
highlights.
|
||||
|
||||
## Output format
|
||||
|
||||
The viewer's parser is the contract, not JD2's exact bytes. `EXPORT_RE` in
|
||||
`src/lib/archive-patterns.ts` accepts all of these, and normalises the index
|
||||
with `parseInt`, so **JD2 and gallery-dl naming interoperate**:
|
||||
|
||||
```
|
||||
"… - CrORBIcJJbM.mp4" -> postId=CrORBIcJJbM index=1
|
||||
"… - CrORBIcJJbM - 1.mp4" -> postId=CrORBIcJJbM index=1
|
||||
"… - C53YPQzp7Wj - 09.jpg" -> postId=C53YPQzp7Wj index=9
|
||||
```
|
||||
|
||||
That means zero-padding and the presence/absence of ` - N` on single-media posts
|
||||
are cosmetic. Don't spend effort forcing them.
|
||||
|
||||
### Directory layout
|
||||
|
||||
| kind | directory | note |
|
||||
|---|---|---|
|
||||
| posts | `<user>` | |
|
||||
| reels | `<user> - reels` | |
|
||||
| stories | `story - <user>` | |
|
||||
| highlights | `story highlights - <user> - <title>` | |
|
||||
|
||||
**Force the directory with `-D`; never use `{username}` for it.** A profile's
|
||||
reels tab returns *collab reels owned by other accounts* — `/0ct0ber19/reels/`
|
||||
served 6 reels owned by `official_artms` and 1 by `chuuo3o`. With
|
||||
`{username}` those would scatter into `official_artms - reels/`. JD2 got this
|
||||
right and the archive proves it: `chuuo3o` and `official_artms` filenames sit
|
||||
inside `0ct0ber19 - reels/`.
|
||||
|
||||
So: **owner in the filename, crawl scope in the directory.**
|
||||
|
||||
### Filenames
|
||||
|
||||
```jsonc
|
||||
"filename": {
|
||||
"sidecar_shortcode and count >= 10":
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num:02}.{extension}",
|
||||
"sidecar_shortcode":
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num}.{extension}",
|
||||
"":
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.{extension}"
|
||||
}
|
||||
```
|
||||
|
||||
`sidecar_shortcode` is set only when the post is a carousel, so it is the
|
||||
carousel discriminator. Conditions are evaluated in order, first match wins
|
||||
(`path.py:265`).
|
||||
|
||||
Stories and highlights use the per-item `{shortcode}`, not `{post_shortcode}`
|
||||
(which is the *reel's* id, shared by every item in it):
|
||||
|
||||
```
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {shortcode}.{extension}"
|
||||
```
|
||||
|
||||
`{date}` on a story/highlight file is the **per-item** `taken_at`
|
||||
(`instagram.py:337` prefers `item["taken_at"]`), verified on a 154-item
|
||||
highlight whose items carried distinct times while `post_date` stayed pinned to
|
||||
the reel. Highlights therefore gain real dates — today they fall back to
|
||||
directory mtime.
|
||||
|
||||
### The timezone is not UTC
|
||||
|
||||
JD2 stamped filenames in **desktop local time (US Eastern)**. Measured across
|
||||
212 comparable posts:
|
||||
|
||||
| model | mismatches |
|
||||
|---|---:|
|
||||
| UTC | 19 |
|
||||
| UTC−5 (EST) | 10 |
|
||||
| UTC−4 (EDT) | **0** |
|
||||
| America/New_York (DST-aware) | **0** |
|
||||
|
||||
`{date:Olocal/%Y-%m-%d}` uses the machine's local zone with per-timestamp DST
|
||||
awareness, which reproduces it — `mattellite` is `America/Toronto`, the same
|
||||
offsets. Note the **trailing `/` must be omitted**: `Olocal/%Y-%m-%d/` puts the
|
||||
separator into the strftime format and it sanitises to an underscore, giving
|
||||
`2026-08-15__0ct0ber19`.
|
||||
|
||||
If the sync ever moves to a host in another timezone, set an explicit
|
||||
`{date:O-4/…}` or the dates will silently shift for ~9% of posts.
|
||||
|
||||
### Caption sidecars
|
||||
|
||||
JD2 writes one `.txt` per post, named without the index, containing the caption
|
||||
with **no trailing newline**, and writes nothing when the caption is empty
|
||||
(measured: 197 of 217 posts, 86 of 86 reels, 0 of 10 stories, 0 of 16
|
||||
highlights). gallery-dl reproduces this exactly with the default
|
||||
`"empty": false`:
|
||||
|
||||
```jsonc
|
||||
{ "name": "metadata", "event": "post", "mode": "custom",
|
||||
"content-format": "{description}", "extension": "txt",
|
||||
"filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.txt" }
|
||||
```
|
||||
|
||||
`"event": "post"` is what makes it one file per post rather than per media file.
|
||||
|
||||
### Metadata sidecar (new — JD2 had no equivalent)
|
||||
|
||||
```jsonc
|
||||
{ "name": "metadata", "event": "post", "mode": "json",
|
||||
"filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.json",
|
||||
"include": ["post_shortcode","post_id","type","date","post_date","username",
|
||||
"fullname","owner_id","description","count","likes","post_url",
|
||||
"sidecar_shortcode"] }
|
||||
```
|
||||
|
||||
Use **`include`**, not `fields` — `fields` is for `mode: custom` and silently
|
||||
does nothing here, leaving `audio_user` blobs (including another user's profile
|
||||
picture URL) in the output.
|
||||
|
||||
The payoff is `type`, which is Instagram's own classification:
|
||||
|
||||
```json
|
||||
{ "post_shortcode": "DbdG9L9jU4m", "type": "post", "count": 2 } // feed video
|
||||
{ "post_shortcode": "Db-lNCoib9m", "type": "reel", "count": 1 } // real reel
|
||||
```
|
||||
|
||||
This is the `product_type: "clips"` signal, delivered free in the listing
|
||||
response. It is the authoritative answer to "is this a reel", and would let the
|
||||
viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
|
||||
"Scanner work" below.
|
||||
|
||||
**`type` is only populated by listing extractors.** Extracting a single
|
||||
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
|
||||
only matters when testing by hand.
|
||||
|
||||
## Known quirks
|
||||
|
||||
- **`count` is not the emitted file count.** For 135 of 214 posts it was exactly
|
||||
one higher than the number of files written. This makes the `count >= 10`
|
||||
padding condition mis-pad a handful of 9-item posts (10 of 214 measured). Since
|
||||
the parser normalises the index, this is cosmetic — but it means a re-fetch
|
||||
over an existing JD2 tree writes `- 01.jpg` beside an existing `- 1.jpg`.
|
||||
- **Carousels get edited.** Two posts had a different media count live than on
|
||||
disk. Padding width follows the count *at download time*, so a grown carousel
|
||||
produces mixed widths — the archive already contains one such post from JD2.
|
||||
- **Highlights already have two naming styles on disk**, and every undated file
|
||||
has a dated twin. The scanner dedupes by index so they render once; it is
|
||||
wasted disk, not a display bug.
|
||||
|
||||
## Scanner work (not done yet)
|
||||
|
||||
`useArchiveScanner` currently treats any `.json` in the tree as a possible
|
||||
manifest. Adding gallery-dl sidecars needs it to distinguish three things:
|
||||
|
||||
1. Instagram export manifests (`posts_1.json`) — existing path.
|
||||
2. Instaloader `.json.xz` — existing path, GraphQL node shape.
|
||||
3. gallery-dl `.json` — new, flat shape, identified by having
|
||||
`post_shortcode` + `type` at the top level.
|
||||
|
||||
Once (3) is read, `source`/`isStory` and the reel flag should come from `type`
|
||||
rather than from the directory and the lone-video heuristic.
|
||||
|
||||
## Test cases
|
||||
|
||||
Real subjects, all present in the archive today. See
|
||||
`scripts/gdl-sync.py --selftest` for the harness.
|
||||
|
||||
| # | case | shortcode | expected |
|
||||
|---|---|---|---|
|
||||
| 1 | single image | `CwcXnQhOqFG` | one `.jpg`, no index |
|
||||
| 2 | single feed video | `DbdG9L9jU4m` | one `.mp4`, `type: post` |
|
||||
| 3 | carousel, images only | `Cq8LrxSJAJE` | `- 1 … - 3` |
|
||||
| 4 | carousel, image + video | `CtohvHxLnWO` | `- 1.jpg … - 4.mp4`, **no `.txt`** |
|
||||
| 5 | carousel of exactly 9 | `Cv2Hb_brx_N` | 1-digit index |
|
||||
| 6 | carousel of 10+ | `CzM8Uf6B6H_` | 2-digit index `- 01 … - 10` |
|
||||
| 7 | reel shown on the posts grid | `C8FHM6EJl15` | in `<user>`, `type: reel` |
|
||||
| 8 | reel on the reels tab | `Db-lNCoib9m` | in `<user> - reels`, `type: reel` |
|
||||
| 9 | collab reel (other owner) | `DYcZOb0h6Sv` | dir `0ct0ber19 - reels`, filename `chuuo3o` |
|
||||
| 10 | story | live only | `story - <user>`, per-item shortcode + date |
|
||||
| 11 | story highlight | `C-IImhvpFuk` | `story highlights - <user> - <title>` |
|
||||
| 12 | highlight, unicode title | `Drawheeing⠀` | trailing U+2800 preserved in dirname |
|
||||
| 13 | empty caption | `CrdsY5CrSsO` | media written, `.txt` absent |
|
||||
| 14 | deleted post | `C0TgI7sphfZ` | on disk, absent live — must not be removed |
|
||||
| 15 | edited carousel | `C7zG7-jJMlq` | 18 on disk, 8 live — must not be removed |
|
||||
| 16 | pinned posts | `0ct0ber19` | 3 pinned, returned out of date order |
|
||||
| 17 | profile avatar | `0ct0ber19.jpg` | base dir, undated |
|
||||
|
||||
Cases 14–16 are reconciliation, not naming: **a sync must never delete**, since
|
||||
the archive deliberately outlives Instagram.
|
||||
|
||||
Not covered, decide before relying on them: the `/reposts/` tab (`0ct0ber19`
|
||||
has one) and `/tagged/`. Neither is fetched today.
|
||||
Reference in New Issue
Block a user