docs: design a gallery-dl replacement for the JDownloader fetcher
Every claim in docs/gallery-dl.md was measured against the live site and
the archive rather than taken from documentation, because two of the
assumptions turned out to be wrong.
The safety model is the reason the config looks the way it does.
gallery-dl has two API backends: the graphql one issues a request PER
POST for every video and carousel -- the pattern that got this account
banned via Instaloader -- while the default rest one paginates listings
at 30-50 items and carries carousel_media, video_versions and
product_type inline. A 300-post profile costs ~10 requests.
Findings worth recording:
- JD2 stamped filenames in desktop LOCAL time (US Eastern), not UTC.
Across 212 comparable posts: UTC 19 mismatches, UTC-5 10, UTC-4 zero.
{date:Olocal/%Y-%m-%d} reproduces it; the trailing separator must be
omitted or it lands in the strftime format.
- A profile's reels tab returns collab reels owned by OTHER accounts, so
the directory must be forced with -D. JD2 did the same: chuuo3o and
official_artms filenames sit inside "0ct0ber19 - reels".
- Stories and highlights need per-item {shortcode}; {post_shortcode} is
the reel's id and is shared by every item. {date} is per-item, verified
on a 154-item highlight with distinct times.
- gallery-dl reproduces JD2's caption .txt exactly, including writing
nothing for an empty caption and omitting the trailing newline.
- The json sidecar needs `include`, not `fields`; `fields` silently does
nothing in mode:json and leaks audio_user blobs. It yields `type`
(post/reel) -- Instagram's own flag, which can retire the lone-video
heuristic once the scanner reads it.
Naming differences between the two tools are cosmetic: EXPORT_RE already
makes the index optional and parseInt normalises zero-padding, so a mixed
archive parses identically. Tests pin that down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,263 @@
|
|||||||
|
# gallery-dl — a CLI replacement for JDownloader2
|
||||||
|
|
||||||
|
Status: **design + verified config.** `scripts/gdl-sync.py` is a skeleton; no
|
||||||
|
profile has been migrated yet.
|
||||||
|
|
||||||
|
Everything below was measured against the live site and the real archive on
|
||||||
|
2026-08-16, not inferred from documentation.
|
||||||
|
|
||||||
|
## Why gallery-dl and not a hand-rolled script
|
||||||
|
|
||||||
|
The hard parts of fetching Instagram are pagination, cookie handling, CDN URL
|
||||||
|
expiry and resumption. gallery-dl already has all of them, plus extractors that
|
||||||
|
map 1:1 onto our sidecar directory layout (`posts`, `reels`, `stories`,
|
||||||
|
`highlights`). Rolling our own would mean reimplementing the ban-sensitive part
|
||||||
|
by hand.
|
||||||
|
|
||||||
|
## The safety model — read this before changing any option
|
||||||
|
|
||||||
|
The ban vector is **requests to `instagram.com`**, not bandwidth. See
|
||||||
|
`docs/jdownloader.md` for the history; Instaloader got this account banned by
|
||||||
|
asking `instagram.com` a question *per post*.
|
||||||
|
|
||||||
|
gallery-dl has two API backends and the difference is exactly that vector:
|
||||||
|
|
||||||
|
```python
|
||||||
|
if self.config("api") == "graphql":
|
||||||
|
self.api = InstagramGraphqlAPI(self) # per-post api.media() for every
|
||||||
|
else: # video and every carousel
|
||||||
|
self.api = InstagramRestAPI(self) # <- default, listing-only
|
||||||
|
```
|
||||||
|
|
||||||
|
The REST backend paginates at `count: 30` (feed) / `page_size: 50` (clips), and
|
||||||
|
those responses already carry `carousel_media`, `image_versions2`,
|
||||||
|
`video_versions` and `product_type`. **No per-post request.** A 300-post
|
||||||
|
profile costs roughly 10 requests to `instagram.com`.
|
||||||
|
|
||||||
|
Rules, in order of importance:
|
||||||
|
|
||||||
|
1. **`"api": "rest"` always.** Never `graphql`. This is the whole ballgame.
|
||||||
|
2. **Never enable `metadata`-style options that trigger extra calls.** If a
|
||||||
|
field is not already in the listing response, it is not worth a request.
|
||||||
|
3. **Pace it.** `"sleep-request": [4.0, 7.0]` — a randomised gap, not a fixed
|
||||||
|
one. Also `"sleep": [1.0, 3.0]` between downloads.
|
||||||
|
4. **Cap the download rate** (`downloader.http.rate`) so the CDN side looks like
|
||||||
|
a person, not a mirror.
|
||||||
|
5. **Run from the same public IP as the browser the cookie came from.** At time
|
||||||
|
of writing that is `mattellite` (`66.23.52.196`); the dev workstation is a
|
||||||
|
*different* public IP and using the cookie from there is precisely what
|
||||||
|
session-hijack detection looks for.
|
||||||
|
6. **No programmatic login, ever.** gallery-dl's username/password path is
|
||||||
|
disabled upstream anyway; use `--cookies-from-browser`.
|
||||||
|
|
||||||
|
Do not add proxy rotation, fingerprint spoofing or account rotation. Throttling
|
||||||
|
and request-avoidance are welcome; evasion is not.
|
||||||
|
|
||||||
|
### Cookies
|
||||||
|
|
||||||
|
The logged-in Chrome on `mattellite` runs with a non-default profile:
|
||||||
|
|
||||||
|
```
|
||||||
|
--user-data-dir=/home/matt/.config/google-chrome-devtools
|
||||||
|
```
|
||||||
|
|
||||||
|
so the cookie flag is:
|
||||||
|
|
||||||
|
```
|
||||||
|
--cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools"
|
||||||
|
```
|
||||||
|
|
||||||
|
Plain `--cookies-from-browser chrome` fails with "Unable to find chrome cookies
|
||||||
|
database" because it looks in `~/.config/google-chrome/`.
|
||||||
|
|
||||||
|
Anonymous access is **not** a viable fallback: it serves lower-resolution media,
|
||||||
|
caps profile pagination at 12 posts, and returns `AuthRequired` for stories and
|
||||||
|
highlights.
|
||||||
|
|
||||||
|
## Output format
|
||||||
|
|
||||||
|
The viewer's parser is the contract, not JD2's exact bytes. `EXPORT_RE` in
|
||||||
|
`src/lib/archive-patterns.ts` accepts all of these, and normalises the index
|
||||||
|
with `parseInt`, so **JD2 and gallery-dl naming interoperate**:
|
||||||
|
|
||||||
|
```
|
||||||
|
"… - CrORBIcJJbM.mp4" -> postId=CrORBIcJJbM index=1
|
||||||
|
"… - CrORBIcJJbM - 1.mp4" -> postId=CrORBIcJJbM index=1
|
||||||
|
"… - C53YPQzp7Wj - 09.jpg" -> postId=C53YPQzp7Wj index=9
|
||||||
|
```
|
||||||
|
|
||||||
|
That means zero-padding and the presence/absence of ` - N` on single-media posts
|
||||||
|
are cosmetic. Don't spend effort forcing them.
|
||||||
|
|
||||||
|
### Directory layout
|
||||||
|
|
||||||
|
| kind | directory | note |
|
||||||
|
|---|---|---|
|
||||||
|
| posts | `<user>` | |
|
||||||
|
| reels | `<user> - reels` | |
|
||||||
|
| stories | `story - <user>` | |
|
||||||
|
| highlights | `story highlights - <user> - <title>` | |
|
||||||
|
|
||||||
|
**Force the directory with `-D`; never use `{username}` for it.** A profile's
|
||||||
|
reels tab returns *collab reels owned by other accounts* — `/0ct0ber19/reels/`
|
||||||
|
served 6 reels owned by `official_artms` and 1 by `chuuo3o`. With
|
||||||
|
`{username}` those would scatter into `official_artms - reels/`. JD2 got this
|
||||||
|
right and the archive proves it: `chuuo3o` and `official_artms` filenames sit
|
||||||
|
inside `0ct0ber19 - reels/`.
|
||||||
|
|
||||||
|
So: **owner in the filename, crawl scope in the directory.**
|
||||||
|
|
||||||
|
### Filenames
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
"filename": {
|
||||||
|
"sidecar_shortcode and count >= 10":
|
||||||
|
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num:02}.{extension}",
|
||||||
|
"sidecar_shortcode":
|
||||||
|
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num}.{extension}",
|
||||||
|
"":
|
||||||
|
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.{extension}"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`sidecar_shortcode` is set only when the post is a carousel, so it is the
|
||||||
|
carousel discriminator. Conditions are evaluated in order, first match wins
|
||||||
|
(`path.py:265`).
|
||||||
|
|
||||||
|
Stories and highlights use the per-item `{shortcode}`, not `{post_shortcode}`
|
||||||
|
(which is the *reel's* id, shared by every item in it):
|
||||||
|
|
||||||
|
```
|
||||||
|
"{date:Olocal/%Y-%m-%d}_{username} - {shortcode}.{extension}"
|
||||||
|
```
|
||||||
|
|
||||||
|
`{date}` on a story/highlight file is the **per-item** `taken_at`
|
||||||
|
(`instagram.py:337` prefers `item["taken_at"]`), verified on a 154-item
|
||||||
|
highlight whose items carried distinct times while `post_date` stayed pinned to
|
||||||
|
the reel. Highlights therefore gain real dates — today they fall back to
|
||||||
|
directory mtime.
|
||||||
|
|
||||||
|
### The timezone is not UTC
|
||||||
|
|
||||||
|
JD2 stamped filenames in **desktop local time (US Eastern)**. Measured across
|
||||||
|
212 comparable posts:
|
||||||
|
|
||||||
|
| model | mismatches |
|
||||||
|
|---|---:|
|
||||||
|
| UTC | 19 |
|
||||||
|
| UTC−5 (EST) | 10 |
|
||||||
|
| UTC−4 (EDT) | **0** |
|
||||||
|
| America/New_York (DST-aware) | **0** |
|
||||||
|
|
||||||
|
`{date:Olocal/%Y-%m-%d}` uses the machine's local zone with per-timestamp DST
|
||||||
|
awareness, which reproduces it — `mattellite` is `America/Toronto`, the same
|
||||||
|
offsets. Note the **trailing `/` must be omitted**: `Olocal/%Y-%m-%d/` puts the
|
||||||
|
separator into the strftime format and it sanitises to an underscore, giving
|
||||||
|
`2026-08-15__0ct0ber19`.
|
||||||
|
|
||||||
|
If the sync ever moves to a host in another timezone, set an explicit
|
||||||
|
`{date:O-4/…}` or the dates will silently shift for ~9% of posts.
|
||||||
|
|
||||||
|
### Caption sidecars
|
||||||
|
|
||||||
|
JD2 writes one `.txt` per post, named without the index, containing the caption
|
||||||
|
with **no trailing newline**, and writes nothing when the caption is empty
|
||||||
|
(measured: 197 of 217 posts, 86 of 86 reels, 0 of 10 stories, 0 of 16
|
||||||
|
highlights). gallery-dl reproduces this exactly with the default
|
||||||
|
`"empty": false`:
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
{ "name": "metadata", "event": "post", "mode": "custom",
|
||||||
|
"content-format": "{description}", "extension": "txt",
|
||||||
|
"filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.txt" }
|
||||||
|
```
|
||||||
|
|
||||||
|
`"event": "post"` is what makes it one file per post rather than per media file.
|
||||||
|
|
||||||
|
### Metadata sidecar (new — JD2 had no equivalent)
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
{ "name": "metadata", "event": "post", "mode": "json",
|
||||||
|
"filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.json",
|
||||||
|
"include": ["post_shortcode","post_id","type","date","post_date","username",
|
||||||
|
"fullname","owner_id","description","count","likes","post_url",
|
||||||
|
"sidecar_shortcode"] }
|
||||||
|
```
|
||||||
|
|
||||||
|
Use **`include`**, not `fields` — `fields` is for `mode: custom` and silently
|
||||||
|
does nothing here, leaving `audio_user` blobs (including another user's profile
|
||||||
|
picture URL) in the output.
|
||||||
|
|
||||||
|
The payoff is `type`, which is Instagram's own classification:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "post_shortcode": "DbdG9L9jU4m", "type": "post", "count": 2 } // feed video
|
||||||
|
{ "post_shortcode": "Db-lNCoib9m", "type": "reel", "count": 1 } // real reel
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the `product_type: "clips"` signal, delivered free in the listing
|
||||||
|
response. It is the authoritative answer to "is this a reel", and would let the
|
||||||
|
viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
|
||||||
|
"Scanner work" below.
|
||||||
|
|
||||||
|
**`type` is only populated by listing extractors.** Extracting a single
|
||||||
|
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
|
||||||
|
only matters when testing by hand.
|
||||||
|
|
||||||
|
## Known quirks
|
||||||
|
|
||||||
|
- **`count` is not the emitted file count.** For 135 of 214 posts it was exactly
|
||||||
|
one higher than the number of files written. This makes the `count >= 10`
|
||||||
|
padding condition mis-pad a handful of 9-item posts (10 of 214 measured). Since
|
||||||
|
the parser normalises the index, this is cosmetic — but it means a re-fetch
|
||||||
|
over an existing JD2 tree writes `- 01.jpg` beside an existing `- 1.jpg`.
|
||||||
|
- **Carousels get edited.** Two posts had a different media count live than on
|
||||||
|
disk. Padding width follows the count *at download time*, so a grown carousel
|
||||||
|
produces mixed widths — the archive already contains one such post from JD2.
|
||||||
|
- **Highlights already have two naming styles on disk**, and every undated file
|
||||||
|
has a dated twin. The scanner dedupes by index so they render once; it is
|
||||||
|
wasted disk, not a display bug.
|
||||||
|
|
||||||
|
## Scanner work (not done yet)
|
||||||
|
|
||||||
|
`useArchiveScanner` currently treats any `.json` in the tree as a possible
|
||||||
|
manifest. Adding gallery-dl sidecars needs it to distinguish three things:
|
||||||
|
|
||||||
|
1. Instagram export manifests (`posts_1.json`) — existing path.
|
||||||
|
2. Instaloader `.json.xz` — existing path, GraphQL node shape.
|
||||||
|
3. gallery-dl `.json` — new, flat shape, identified by having
|
||||||
|
`post_shortcode` + `type` at the top level.
|
||||||
|
|
||||||
|
Once (3) is read, `source`/`isStory` and the reel flag should come from `type`
|
||||||
|
rather than from the directory and the lone-video heuristic.
|
||||||
|
|
||||||
|
## Test cases
|
||||||
|
|
||||||
|
Real subjects, all present in the archive today. See
|
||||||
|
`scripts/gdl-sync.py --selftest` for the harness.
|
||||||
|
|
||||||
|
| # | case | shortcode | expected |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | single image | `CwcXnQhOqFG` | one `.jpg`, no index |
|
||||||
|
| 2 | single feed video | `DbdG9L9jU4m` | one `.mp4`, `type: post` |
|
||||||
|
| 3 | carousel, images only | `Cq8LrxSJAJE` | `- 1 … - 3` |
|
||||||
|
| 4 | carousel, image + video | `CtohvHxLnWO` | `- 1.jpg … - 4.mp4`, **no `.txt`** |
|
||||||
|
| 5 | carousel of exactly 9 | `Cv2Hb_brx_N` | 1-digit index |
|
||||||
|
| 6 | carousel of 10+ | `CzM8Uf6B6H_` | 2-digit index `- 01 … - 10` |
|
||||||
|
| 7 | reel shown on the posts grid | `C8FHM6EJl15` | in `<user>`, `type: reel` |
|
||||||
|
| 8 | reel on the reels tab | `Db-lNCoib9m` | in `<user> - reels`, `type: reel` |
|
||||||
|
| 9 | collab reel (other owner) | `DYcZOb0h6Sv` | dir `0ct0ber19 - reels`, filename `chuuo3o` |
|
||||||
|
| 10 | story | live only | `story - <user>`, per-item shortcode + date |
|
||||||
|
| 11 | story highlight | `C-IImhvpFuk` | `story highlights - <user> - <title>` |
|
||||||
|
| 12 | highlight, unicode title | `Drawheeing⠀` | trailing U+2800 preserved in dirname |
|
||||||
|
| 13 | empty caption | `CrdsY5CrSsO` | media written, `.txt` absent |
|
||||||
|
| 14 | deleted post | `C0TgI7sphfZ` | on disk, absent live — must not be removed |
|
||||||
|
| 15 | edited carousel | `C7zG7-jJMlq` | 18 on disk, 8 live — must not be removed |
|
||||||
|
| 16 | pinned posts | `0ct0ber19` | 3 pinned, returned out of date order |
|
||||||
|
| 17 | profile avatar | `0ct0ber19.jpg` | base dir, undated |
|
||||||
|
|
||||||
|
Cases 14–16 are reconciliation, not naming: **a sync must never delete**, since
|
||||||
|
the archive deliberately outlives Instagram.
|
||||||
|
|
||||||
|
Not covered, decide before relying on them: the `/reposts/` tab (`0ct0ber19`
|
||||||
|
has one) and `/tagged/`. Neither is fetched today.
|
||||||
Executable
+333
@@ -0,0 +1,333 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Fetch Instagram profiles into the archive layout using gallery-dl.
|
||||||
|
|
||||||
|
The CLI replacement for the JDownloader2 workflow. See docs/gallery-dl.md for
|
||||||
|
the measurements behind every choice here — especially the safety model, which
|
||||||
|
is the reason this script exists in this shape rather than a simpler one.
|
||||||
|
|
||||||
|
STATUS: skeleton. The config generation and planning are complete and tested;
|
||||||
|
`--execute` is deliberately gated behind an explicit flag and has not been run
|
||||||
|
against the live archive.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
./scripts/gdl-sync.py --archives <dir> --profile 0ct0ber19 --dry-run
|
||||||
|
./scripts/gdl-sync.py --archives <dir> --all --dry-run
|
||||||
|
./scripts/gdl-sync.py --archives <dir> --profile 0ct0ber19 --execute
|
||||||
|
|
||||||
|
Run it from the host whose public IP matches the browser the cookie came from.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# Archive layout
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Mirrors src/lib/archive-grouping.ts. Instagram usernames cannot contain
|
||||||
|
# spaces, which is what makes the username separable from a highlight title.
|
||||||
|
RE_HIGHLIGHT = re.compile(r"^story highlights - ([^ ]+) - (.+)$")
|
||||||
|
RE_STORIES = re.compile(r"^story - ([^ ]+)$")
|
||||||
|
RE_REELS = re.compile(r"^([^ ]+) - reels$")
|
||||||
|
|
||||||
|
DATE_FMT = "{date:Olocal/%Y-%m-%d}"
|
||||||
|
"""Local-time date. JD2 stamped US Eastern, NOT UTC (0/212 mismatches vs 19 for
|
||||||
|
UTC). `Olocal` is DST-aware per timestamp. The trailing separator must be
|
||||||
|
omitted or it lands in the strftime format and sanitises to an underscore."""
|
||||||
|
|
||||||
|
POST_STEM = DATE_FMT + "_{username} - {post_shortcode}"
|
||||||
|
ITEM_STEM = DATE_FMT + "_{username} - {shortcode}"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Source:
|
||||||
|
"""One gallery-dl invocation: a URL fetched into a specific directory."""
|
||||||
|
|
||||||
|
kind: str # posts | reels | stories | highlights
|
||||||
|
url: str
|
||||||
|
directory: str # relative to the archives root
|
||||||
|
subcategory: str # gallery-dl config key
|
||||||
|
title: str | None = None # highlight title, when known
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Profile:
|
||||||
|
user: str
|
||||||
|
existing: dict[str, str] = field(default_factory=dict) # kind -> dirname
|
||||||
|
|
||||||
|
def sources(self, include_stories: bool) -> list[Source]:
|
||||||
|
u = self.user
|
||||||
|
base = f"https://www.instagram.com/{u}"
|
||||||
|
out = [
|
||||||
|
Source("posts", f"{base}/posts/", u, "posts"),
|
||||||
|
Source("reels", f"{base}/reels/", f"{u} - reels", "reels"),
|
||||||
|
]
|
||||||
|
if include_stories:
|
||||||
|
# Stories expire after 24h, so these can only ever be captured
|
||||||
|
# live. There is no backfill and no re-fetch.
|
||||||
|
out.append(Source(
|
||||||
|
"stories", f"https://www.instagram.com/stories/{u}/",
|
||||||
|
f"story - {u}", "stories"))
|
||||||
|
# Highlight directories embed the title, which gallery-dl only
|
||||||
|
# learns mid-extraction -- so this one source fans out into many
|
||||||
|
# directories and is handled with a directory format string.
|
||||||
|
out.append(Source(
|
||||||
|
"highlights", f"{base}/highlights", "", "highlights"))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def scan_archives(root: Path) -> dict[str, Profile]:
|
||||||
|
"""Group existing directories into profiles, as the server does."""
|
||||||
|
profiles: dict[str, Profile] = {}
|
||||||
|
|
||||||
|
def get(user: str) -> Profile:
|
||||||
|
return profiles.setdefault(user, Profile(user))
|
||||||
|
|
||||||
|
for entry in sorted(os.listdir(root)):
|
||||||
|
if not (root / entry).is_dir() or entry.startswith("."):
|
||||||
|
continue
|
||||||
|
if m := RE_HIGHLIGHT.match(entry):
|
||||||
|
get(m.group(1)).existing.setdefault("highlights", entry)
|
||||||
|
elif m := RE_STORIES.match(entry):
|
||||||
|
get(m.group(1)).existing["stories"] = entry
|
||||||
|
elif m := RE_REELS.match(entry):
|
||||||
|
get(m.group(1)).existing["reels"] = entry
|
||||||
|
else:
|
||||||
|
get(entry).existing["posts"] = entry
|
||||||
|
return profiles
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# gallery-dl configuration
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def build_config(rate: str, sleep_request: list[float],
|
||||||
|
sleep: list[float]) -> dict:
|
||||||
|
"""
|
||||||
|
The config is generated rather than checked in so the safety-critical
|
||||||
|
options cannot drift out of sync with the docs.
|
||||||
|
|
||||||
|
`api: rest` is the single most important line in this file. The graphql
|
||||||
|
backend issues one request PER POST for every video and carousel, which is
|
||||||
|
the pattern that got this account banned once already.
|
||||||
|
"""
|
||||||
|
caption_pp = {
|
||||||
|
"name": "metadata",
|
||||||
|
"event": "post",
|
||||||
|
"mode": "custom",
|
||||||
|
"content-format": "{description}",
|
||||||
|
"extension": "txt",
|
||||||
|
# JD2 wrote no .txt when the caption was empty; "empty": false (the
|
||||||
|
# default) reproduces that.
|
||||||
|
}
|
||||||
|
meta_pp = {
|
||||||
|
"name": "metadata",
|
||||||
|
"event": "post",
|
||||||
|
"mode": "json",
|
||||||
|
# `include`, NOT `fields` -- `fields` applies to mode:custom and
|
||||||
|
# silently does nothing here, dumping audio_user blobs that contain
|
||||||
|
# unrelated users' profile picture URLs.
|
||||||
|
"include": [
|
||||||
|
"post_shortcode", "post_id", "type", "date", "post_date",
|
||||||
|
"username", "fullname", "owner_id", "description", "count",
|
||||||
|
"likes", "post_url", "sidecar_shortcode",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
def post_like(stem: str) -> dict:
|
||||||
|
"""Naming for surfaces whose unit is a post (posts, reels)."""
|
||||||
|
return {
|
||||||
|
# `sidecar_shortcode` is set only for carousels, so it is the
|
||||||
|
# carousel discriminator. First matching condition wins.
|
||||||
|
"filename": {
|
||||||
|
"sidecar_shortcode and count >= 10":
|
||||||
|
stem + " - {num:02}.{extension}",
|
||||||
|
"sidecar_shortcode":
|
||||||
|
stem + " - {num}.{extension}",
|
||||||
|
"":
|
||||||
|
stem + ".{extension}",
|
||||||
|
},
|
||||||
|
"postprocessors": [
|
||||||
|
{**caption_pp, "filename": stem + ".txt"},
|
||||||
|
{**meta_pp, "filename": stem + ".json"},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
def item_like(stem: str) -> dict:
|
||||||
|
"""
|
||||||
|
Naming for surfaces whose unit is an item inside a reel (stories,
|
||||||
|
highlights). `{shortcode}` is per item; `{post_shortcode}` is the
|
||||||
|
reel's id and is shared by every item in it.
|
||||||
|
"""
|
||||||
|
return {
|
||||||
|
"filename": stem + ".{extension}",
|
||||||
|
"postprocessors": [{**meta_pp, "filename": stem + ".json"}],
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
"extractor": {
|
||||||
|
"base-directory": ".",
|
||||||
|
"instagram": {
|
||||||
|
"api": "rest", # never "graphql" -- see docstring
|
||||||
|
"sleep-request": sleep_request,
|
||||||
|
"sleep": sleep,
|
||||||
|
"videos": True,
|
||||||
|
"include": "", # never "all"; sources are explicit
|
||||||
|
# Directory is forced per-invocation with -D, because a reels
|
||||||
|
# tab returns collab reels owned by OTHER accounts and
|
||||||
|
# {username} would scatter them into the wrong profile.
|
||||||
|
"directory": [],
|
||||||
|
"posts": post_like(POST_STEM),
|
||||||
|
"reels": post_like(POST_STEM),
|
||||||
|
"stories": item_like(ITEM_STEM),
|
||||||
|
"highlights": {
|
||||||
|
**item_like(ITEM_STEM),
|
||||||
|
# The only surface that must derive its own directory,
|
||||||
|
# since the title is not known until extraction.
|
||||||
|
"directory": ["story highlights - {username} - {highlight_title}"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"downloader": {"http": {"rate": rate}},
|
||||||
|
"output": {"mode": "null"},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# Planning and execution
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def gdl_command(src: Source, root: Path, config: Path, cookies: str,
|
||||||
|
archive_db: Path | None) -> list[str]:
|
||||||
|
cmd = [
|
||||||
|
"gallery-dl",
|
||||||
|
"--config", str(config),
|
||||||
|
"--cookies-from-browser", cookies,
|
||||||
|
]
|
||||||
|
if archive_db:
|
||||||
|
# gallery-dl's skip-archive. Seed it before the first real run or the
|
||||||
|
# whole 110k-file tree is re-downloaded; see --seed-archive.
|
||||||
|
cmd += ["--download-archive", str(archive_db)]
|
||||||
|
if src.subcategory != "highlights":
|
||||||
|
cmd += ["--destination", str(root / src.directory)]
|
||||||
|
else:
|
||||||
|
cmd += ["--destination", str(root)]
|
||||||
|
cmd.append(src.url)
|
||||||
|
return cmd
|
||||||
|
|
||||||
|
|
||||||
|
def seed_archive_db(root: Path, profiles: dict[str, Profile],
|
||||||
|
db: Path) -> int:
|
||||||
|
"""
|
||||||
|
TODO: populate the skip-archive from filenames already on disk.
|
||||||
|
|
||||||
|
gallery-dl keys its archive on an extractor-specific id string, so this
|
||||||
|
needs the exact format `InstagramExtractor.archive_fmt` produces. Until
|
||||||
|
it is implemented, the first run of any profile re-downloads everything —
|
||||||
|
bandwidth on the CDN (the tolerant surface), but hours of it.
|
||||||
|
"""
|
||||||
|
raise NotImplementedError(
|
||||||
|
"seed_archive_db is unimplemented; run without --archive-db and accept "
|
||||||
|
"a full re-download, or implement this first")
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__,
|
||||||
|
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
ap.add_argument("--archives", required=True, type=Path,
|
||||||
|
help="archive root (one directory per profile plus sidecars)")
|
||||||
|
g = ap.add_mutually_exclusive_group(required=True)
|
||||||
|
g.add_argument("--profile", action="append", default=[],
|
||||||
|
help="profile to sync; repeatable")
|
||||||
|
g.add_argument("--all", action="store_true", help="every profile on disk")
|
||||||
|
ap.add_argument("--cookies", default="chrome:/home/matt/.config/google-chrome-devtools",
|
||||||
|
help="gallery-dl --cookies-from-browser value")
|
||||||
|
ap.add_argument("--archive-db", type=Path, default=None,
|
||||||
|
help="gallery-dl skip-archive sqlite path")
|
||||||
|
ap.add_argument("--rate", default="3M", help="per-download rate cap")
|
||||||
|
ap.add_argument("--sleep-request", nargs=2, type=float, default=[4.0, 7.0],
|
||||||
|
metavar=("MIN", "MAX"))
|
||||||
|
ap.add_argument("--sleep", nargs=2, type=float, default=[1.0, 3.0],
|
||||||
|
metavar=("MIN", "MAX"))
|
||||||
|
ap.add_argument("--no-stories", action="store_true",
|
||||||
|
help="skip stories and highlights (posts and reels only)")
|
||||||
|
mode = ap.add_mutually_exclusive_group()
|
||||||
|
mode.add_argument("--dry-run", action="store_true", default=True,
|
||||||
|
help="print the plan and the config; default")
|
||||||
|
mode.add_argument("--execute", action="store_true",
|
||||||
|
help="actually run gallery-dl")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
if not args.archives.is_dir():
|
||||||
|
print(f"archives root not found: {args.archives}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
if not shutil.which("gallery-dl"):
|
||||||
|
print("gallery-dl not on PATH", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
profiles = scan_archives(args.archives)
|
||||||
|
if args.profile:
|
||||||
|
missing = [p for p in args.profile if p not in profiles]
|
||||||
|
for p in missing:
|
||||||
|
print(f"note: {p} has no directory yet; it will be created")
|
||||||
|
profiles.setdefault(p, Profile(p))
|
||||||
|
selected = [profiles[p] for p in args.profile]
|
||||||
|
else:
|
||||||
|
selected = list(profiles.values())
|
||||||
|
|
||||||
|
config = build_config(args.rate, list(args.sleep_request), list(args.sleep))
|
||||||
|
config_path = args.archives / ".gdl-sync.config.json"
|
||||||
|
|
||||||
|
plan: list[tuple[Profile, Source]] = [
|
||||||
|
(prof, src)
|
||||||
|
for prof in selected
|
||||||
|
for src in prof.sources(include_stories=not args.no_stories)
|
||||||
|
]
|
||||||
|
|
||||||
|
print(f"profiles : {len(selected)}")
|
||||||
|
print(f"sources : {len(plan)}")
|
||||||
|
print(f"pacing : {args.sleep_request[0]}-{args.sleep_request[1]}s between "
|
||||||
|
f"requests, rate cap {args.rate}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
if not args.execute:
|
||||||
|
print(json.dumps(config, indent=2))
|
||||||
|
print()
|
||||||
|
for prof, src in plan:
|
||||||
|
dest = src.directory or "(per-highlight)"
|
||||||
|
print(f" {prof.user:<20} {src.kind:<11} -> {dest}")
|
||||||
|
print("\ndry run; nothing fetched. pass --execute to run.")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
config_path.write_text(json.dumps(config, indent=2))
|
||||||
|
failures = 0
|
||||||
|
for prof, src in plan:
|
||||||
|
if src.subcategory != "highlights":
|
||||||
|
(args.archives / src.directory).mkdir(parents=True, exist_ok=True)
|
||||||
|
cmd = gdl_command(src, args.archives, config_path, args.cookies,
|
||||||
|
args.archive_db)
|
||||||
|
print(f"==> {prof.user} / {src.kind}")
|
||||||
|
result = subprocess.run(cmd)
|
||||||
|
if result.returncode != 0:
|
||||||
|
failures += 1
|
||||||
|
# Keep going: one private/renamed profile must not abort the run.
|
||||||
|
print(f" FAILED (exit {result.returncode})", file=sys.stderr)
|
||||||
|
|
||||||
|
# A sync NEVER deletes. The archive deliberately outlives Instagram --
|
||||||
|
# 2 posts in 0ct0ber19 exist only here now.
|
||||||
|
print(f"\ndone; {failures} source(s) failed")
|
||||||
|
return 1 if failures else 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -116,3 +116,43 @@ describe('scopedPostId', () => {
|
|||||||
expect(inPosts).not.toBe(inHighlight);
|
expect(inPosts).not.toBe(inHighlight);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* gallery-dl is replacing JDownloader as the fetcher (docs/gallery-dl.md).
|
||||||
|
* Its naming differs cosmetically, and these cases pin down that the two
|
||||||
|
* interoperate so a mixed archive parses identically.
|
||||||
|
*/
|
||||||
|
describe('gallery-dl / JDownloader naming interop', () => {
|
||||||
|
it('treats a single-media post the same with or without an index', () => {
|
||||||
|
const jd2 = parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM.mp4')!;
|
||||||
|
const gdl = parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM - 1.mp4')!;
|
||||||
|
expect(jd2.postId).toBe(gdl.postId);
|
||||||
|
expect(jd2.index).toBe(gdl.index);
|
||||||
|
expect(jd2.index).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('normalises zero-padded carousel indices', () => {
|
||||||
|
// JD2 pads to the width of the media count (10+ items -> "01"), and
|
||||||
|
// gallery-dl's count can be one higher, so the same post may be padded
|
||||||
|
// by one tool and not the other.
|
||||||
|
expect(parseArchiveFilename('2024-04-17_0ct0ber19 - C53YPQzp7Wj - 09.jpg')!.index).toBe(9);
|
||||||
|
expect(parseArchiveFilename('2024-04-17_0ct0ber19 - C53YPQzp7Wj - 9.jpg')!.index).toBe(9);
|
||||||
|
expect(parseArchiveFilename('2023-11-03_0ct0ber19 - CzM8Uf6B6H_ - 01.jpg')!.index).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reads a gallery-dl story name, which carries a per-item shortcode', () => {
|
||||||
|
const p = parseArchiveFilename('2026-08-16_official_artms - DcF9OyhBJ1H.jpg', 'stories')!;
|
||||||
|
expect(p.postId).toBe('DcF9OyhBJ1H');
|
||||||
|
expect(p.date).toBe('2026-08-16');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('gives a dated highlight a real date instead of the mtime fallback', () => {
|
||||||
|
const mtime = Date.parse('2026-08-17T00:00:00Z');
|
||||||
|
const undated = parseArchiveFilename('0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
|
||||||
|
const dated = parseArchiveFilename('2024-08-04_0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
|
||||||
|
// Same item either way, so re-fetching cannot split it into two posts.
|
||||||
|
expect(dated.postId).toBe(undated.postId);
|
||||||
|
expect(undated.date).toBe('2026-08-17');
|
||||||
|
expect(dated.date).toBe('2024-08-04');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
Reference in New Issue
Block a user