ergosteurandClaude Opus 5
acab376af5
The scripts and docs for fetching from Instagram now live here rather than on main, which is the branch published to GitHub. They carry things that do not belong in a public repo: the fetch host's public IP, the browser profile path the cookie is read from, the NAS archive path, and the list of accounts being archived. This branch is a superset of main — the viewer plus the tooling — so it can take main's changes by merging, and the npm script and CLAUDE.md entries that reference the tooling live here where the files actually exist. Restored with the sync work from the 2026-08-20 run already in place: the --abort flag, the corrected yt-dlp install advice, and the measurements behind both. Note that main's history was rewritten to strip these paths, so the tooling's own per-file history does not exist on this branch. It is preserved on gitea as pre-rewrite-20260820 and pre-rewrite-tooling-20260820. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
182 lines
7.4 KiB
Markdown
182 lines
7.4 KiB
Markdown
# JDownloader2 — archive fetching quick reference
|
||
|
||
How content gets into this archive, and why the setup is shaped the way it is.
|
||
|
||
## Why JDownloader and not Instaloader
|
||
|
||
There are two surfaces, and they're treated very differently:
|
||
|
||
| Surface | What hits it | Risk |
|
||
|---|---|---|
|
||
| `instagram.com` | profile pages, GraphQL/API metadata | Tied to your session, heavily rate-limited. **This is where bans come from.** |
|
||
| `scontent*.cdninstagram.com` | the actual media | Signed URLs, CDN-served, tolerant. Mostly a bandwidth question. |
|
||
|
||
JDownloader does nearly all its work on the CDN. Instaloader's value — the rich
|
||
`.json.xz` metadata — comes from asking `instagram.com` a question *per post*.
|
||
|
||
Concretely, from this archive: `rivvsofficial` has 188 post-metadata files, so
|
||
backfilling it cost 188 API requests for one 605-file profile. That's the ban
|
||
vector. Downloading the 238 photos was never the problem.
|
||
|
||
Instaloader got this account banned once. JDownloader with throttling did not.
|
||
|
||
> **The account was suspended anyway, on 2026-08-17, for "spam".** Not by
|
||
> JDownloader, and not by downloading. It was suspended during a day of
|
||
> *building and verifying* the gallery-dl replacement — automated browser
|
||
> scrolling to enumerate profile grids, repeated `--simulate` and `-j` metadata
|
||
> passes, and one aborted sync that re-ran every listing pass before dying.
|
||
>
|
||
> The framing above is right about which surface is dangerous and wrong about
|
||
> what reaches it. **Every read of `instagram.com` counts, including the ones
|
||
> that download nothing** — and read-only work is easy not to count precisely
|
||
> because it leaves no files behind. See the post-mortem at the top of
|
||
> `docs/gallery-dl.md`.
|
||
>
|
||
> The rule that would have prevented it: *verify against the archive, never
|
||
> against the live site*, and treat the first CDN `429` as the end of the
|
||
> session rather than a pacing knob.
|
||
|
||
### What the metadata gap actually costs
|
||
|
||
Comparing a JDownloader profile against an Instaloader one:
|
||
|
||
| | JDownloader | Instaloader |
|
||
|---|---|---|
|
||
| Media | ✅ | ✅ |
|
||
| Captions (`.txt`) | ✅ | ✅ |
|
||
| Dates (from filenames) | ✅ | ✅ |
|
||
| Bio / full name | ❌ | ✅ |
|
||
| Follower counts | ❌ | ✅ |
|
||
| External URL | ❌ | ✅ |
|
||
|
||
Captions already work — the viewer reads the `.txt` sidecars. Everything missing
|
||
lives in a *single* profile-level record, not the per-post ones. That's why
|
||
JDownloader-sourced profiles show "0 followers" and a placeholder bio.
|
||
|
||
Not worth extra requests. If you ever want it, the zero-request option is a
|
||
hand-written `profile.json` sidecar (not implemented yet — ask).
|
||
|
||
## Settings that matter
|
||
|
||
**Chunks per download → 1.** The single most important one. JDownloader splits
|
||
each file into multiple ranged requests by default; that `Range` pattern looks
|
||
nothing like a browser or the app. One chunk = one sequential GET per file.
|
||
`jd2-sync` sets `chunks=1` per job, so no global change is needed — but set it
|
||
globally too if you ever add links by hand.
|
||
|
||
**Max simultaneous downloads → 2–3**, connections-per-host low. Concurrency is
|
||
what turns "a user" into a statistic.
|
||
|
||
**Leave reconnect / IP-change features off.** A mid-session IP change on a live
|
||
cookie is a *stronger* anomaly signal than the request rate you'd be avoiding.
|
||
|
||
## The cookie
|
||
|
||
Exported manually from a real browser session. This is the right approach — no
|
||
programmatic login anywhere, which is the thing that actually gets flagged.
|
||
|
||
- Use it from the **same public IP** as the browser it came from. A cookie used
|
||
from a different network is what session-hijack detection looks for.
|
||
- When it expires, **re-export from the browser**. Never add a login step to a tool.
|
||
- It's a full account credential. Keep it off the NAS share and out of the repo.
|
||
|
||
## Workflow
|
||
|
||
Two URLs per profile, because the profile grid misses some reels:
|
||
|
||
```
|
||
https://www.instagram.com/<user>/
|
||
https://www.instagram.com/<user>/reels/
|
||
```
|
||
|
||
They overlap slightly — a reel caught by both lands in each directory and shows
|
||
up twice in the viewer. That's correct and matches Instagram, which also shows
|
||
reels in the profile grid *and* the Reels tab.
|
||
|
||
## Generating jobs
|
||
|
||
Instead of pasting URLs and setting output folders by hand:
|
||
|
||
```bash
|
||
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives --dry-run
|
||
```
|
||
|
||
Review, then write it into JDownloader's folder-watch directory:
|
||
|
||
```bash
|
||
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives \
|
||
--out ~/.jd2/folderwatch
|
||
```
|
||
|
||
JDownloader runs on the desktop while the archive lives on the NAS, so tell it
|
||
the path *it* sees:
|
||
|
||
```bash
|
||
npm run jd2 -- --archives /mnt/nas/Instagram-archive/archives \
|
||
--download-base 'Z:\Instagram-archive\archives' \
|
||
--out ~/.jd2/folderwatch
|
||
```
|
||
|
||
| Flag | Purpose |
|
||
|---|---|
|
||
| `--archives <dir>` | Archive root to scan (or `$ARCHIVES_DIR`) |
|
||
| `--out <dir>` | JDownloader folder-watch directory |
|
||
| `--download-base <dir>` | Root path as JDownloader sees it (Windows paths fine) |
|
||
| `--user <name>` | Just this profile (repeatable) |
|
||
| `--skip <name>` | Never emit jobs for this directory (repeatable) |
|
||
| `--chunks <n>` | Connections per file (default 1) |
|
||
| `--auto-start` | Start immediately instead of parking in LinkGrabber |
|
||
| `--all-reels` | Emit a reels job even where no reels directory exists |
|
||
| `--dry-run` | Print instead of writing |
|
||
|
||
Defaults are deliberately conservative: `chunks=1`, and links park in the
|
||
LinkGrabber for review rather than auto-starting.
|
||
|
||
Only posts and reels are emitted. Highlight URLs need a numeric id and story
|
||
URLs expire, so those stay manual.
|
||
|
||
Directories that aren't Instagram profiles are skipped by username shape
|
||
(letters, digits, dots, underscores, ≤30 chars) — pointing a crawl at those
|
||
spends `instagram.com` requests to be told the profile doesn't exist. For names
|
||
that *look* like usernames but aren't, use `--skip` or a `.jd2ignore` file in
|
||
the archive root, one name per line.
|
||
|
||
Format reference: `src/org/jdownloader/extensions/folderwatchV2/explain.txt`.
|
||
JDownloader develops on SVN — read it via the daily mirror at
|
||
<https://github.com/mycodedoesnotcompile2/jdownloader_mirror> (`svn_trunk/`),
|
||
not one of the abandoned GitHub copies.
|
||
|
||
## Expected layout
|
||
|
||
Everything downloads into `<archives>/`, one directory per source:
|
||
|
||
```
|
||
archives/
|
||
0ct0ber19/ posts
|
||
0ct0ber19 - reels/ reels
|
||
story - 0ct0ber19/ stories
|
||
story highlights - 0ct0ber19 - Heestory/ a highlight
|
||
```
|
||
|
||
Non-archive directories (tool output, exports from elsewhere) live *outside*
|
||
`archives/` so they never reach the viewer.
|
||
|
||
The server picks up changes automatically — its index is keyed on directory
|
||
mtime, so a new file invalidates only that directory.
|
||
|
||
## If something goes wrong
|
||
|
||
**429 / rate limited** — stop for hours, not seconds. Retrying into a limit is
|
||
what converts a soft throttle into something worse.
|
||
|
||
**Cookie stops working** — re-export from the browser. Don't add a login step.
|
||
|
||
**Files land in the wrong folder** — a Packagizer rule is overriding the job.
|
||
Generated jobs set `overwritePackagizerEnabled=TRUE` to prevent this; check that
|
||
rules aren't set to run after it.
|
||
|
||
**Viewer doesn't show new posts** — check the file is in the right directory and
|
||
matches the naming pattern (`YYYY-MM-DD_<user> - <shortcode>[ - NN].<ext>`).
|
||
The index refreshes on directory mtime, so a genuinely new file is picked up on
|
||
the next request.
|