ergosteurandClaude Opus 5
7a085d2272
The account was suspended on 2026-08-17 for "spam", during the session that built this tooling. Both fetching docs were confidently wrong about what the risk was, so both now carry the correction. docs/jdownloader.md said the ban vector is instagram.com requests, which is right, and implied that meant per-post metadata fetching, which is only part of it. The suspension came from read-only verification: automated browser scrolling to enumerate profile grids (~18 paginated loads per profile, done twice on one after a selector bug), repeated --simulate and -j passes over the same profiles, per-post /p/ fetches while testing filename formats, and an aborted sync that re-ran every listing pass before dying. None of that produced a file, and together it rivalled the real sync for request count. The rules that follow are in docs/gallery-dl.md: verify against the archive rather than the live site, count read-only work against the same budget, treat the first CDN 429 as the end of the session rather than a pacing knob, and cache probe_live so a restart does not re-enumerate everything. The warning order was CDN 429, then 400 on the highlights tray, then suspension. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
182 lines
7.4 KiB
Markdown
182 lines
7.4 KiB
Markdown
# JDownloader2 — archive fetching quick reference
|
||
|
||
How content gets into this archive, and why the setup is shaped the way it is.
|
||
|
||
## Why JDownloader and not Instaloader
|
||
|
||
There are two surfaces, and they're treated very differently:
|
||
|
||
| Surface | What hits it | Risk |
|
||
|---|---|---|
|
||
| `instagram.com` | profile pages, GraphQL/API metadata | Tied to your session, heavily rate-limited. **This is where bans come from.** |
|
||
| `scontent*.cdninstagram.com` | the actual media | Signed URLs, CDN-served, tolerant. Mostly a bandwidth question. |
|
||
|
||
JDownloader does nearly all its work on the CDN. Instaloader's value — the rich
|
||
`.json.xz` metadata — comes from asking `instagram.com` a question *per post*.
|
||
|
||
Concretely, from this archive: `rivvsofficial` has 188 post-metadata files, so
|
||
backfilling it cost 188 API requests for one 605-file profile. That's the ban
|
||
vector. Downloading the 238 photos was never the problem.
|
||
|
||
Instaloader got this account banned once. JDownloader with throttling did not.
|
||
|
||
> **The account was suspended anyway, on 2026-08-17, for "spam".** Not by
|
||
> JDownloader, and not by downloading. It was suspended during a day of
|
||
> *building and verifying* the gallery-dl replacement — automated browser
|
||
> scrolling to enumerate profile grids, repeated `--simulate` and `-j` metadata
|
||
> passes, and one aborted sync that re-ran every listing pass before dying.
|
||
>
|
||
> The framing above is right about which surface is dangerous and wrong about
|
||
> what reaches it. **Every read of `instagram.com` counts, including the ones
|
||
> that download nothing** — and read-only work is easy not to count precisely
|
||
> because it leaves no files behind. See the post-mortem at the top of
|
||
> `docs/gallery-dl.md`.
|
||
>
|
||
> The rule that would have prevented it: *verify against the archive, never
|
||
> against the live site*, and treat the first CDN `429` as the end of the
|
||
> session rather than a pacing knob.
|
||
|
||
### What the metadata gap actually costs
|
||
|
||
Comparing a JDownloader profile against an Instaloader one:
|
||
|
||
| | JDownloader | Instaloader |
|
||
|---|---|---|
|
||
| Media | ✅ | ✅ |
|
||
| Captions (`.txt`) | ✅ | ✅ |
|
||
| Dates (from filenames) | ✅ | ✅ |
|
||
| Bio / full name | ❌ | ✅ |
|
||
| Follower counts | ❌ | ✅ |
|
||
| External URL | ❌ | ✅ |
|
||
|
||
Captions already work — the viewer reads the `.txt` sidecars. Everything missing
|
||
lives in a *single* profile-level record, not the per-post ones. That's why
|
||
JDownloader-sourced profiles show "0 followers" and a placeholder bio.
|
||
|
||
Not worth extra requests. If you ever want it, the zero-request option is a
|
||
hand-written `profile.json` sidecar (not implemented yet — ask).
|
||
|
||
## Settings that matter
|
||
|
||
**Chunks per download → 1.** The single most important one. JDownloader splits
|
||
each file into multiple ranged requests by default; that `Range` pattern looks
|
||
nothing like a browser or the app. One chunk = one sequential GET per file.
|
||
`jd2-sync` sets `chunks=1` per job, so no global change is needed — but set it
|
||
globally too if you ever add links by hand.
|
||
|
||
**Max simultaneous downloads → 2–3**, connections-per-host low. Concurrency is
|
||
what turns "a user" into a statistic.
|
||
|
||
**Leave reconnect / IP-change features off.** A mid-session IP change on a live
|
||
cookie is a *stronger* anomaly signal than the request rate you'd be avoiding.
|
||
|
||
## The cookie
|
||
|
||
Exported manually from a real browser session. This is the right approach — no
|
||
programmatic login anywhere, which is the thing that actually gets flagged.
|
||
|
||
- Use it from the **same public IP** as the browser it came from. A cookie used
|
||
from a different network is what session-hijack detection looks for.
|
||
- When it expires, **re-export from the browser**. Never add a login step to a tool.
|
||
- It's a full account credential. Keep it off the NAS share and out of the repo.
|
||
|
||
## Workflow
|
||
|
||
Two URLs per profile, because the profile grid misses some reels:
|
||
|
||
```
|
||
https://www.instagram.com/<user>/
|
||
https://www.instagram.com/<user>/reels/
|
||
```
|
||
|
||
They overlap slightly — a reel caught by both lands in each directory and shows
|
||
up twice in the viewer. That's correct and matches Instagram, which also shows
|
||
reels in the profile grid *and* the Reels tab.
|
||
|
||
## Generating jobs
|
||
|
||
Instead of pasting URLs and setting output folders by hand:
|
||
|
||
```bash
|
||
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives --dry-run
|
||
```
|
||
|
||
Review, then write it into JDownloader's folder-watch directory:
|
||
|
||
```bash
|
||
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives \
|
||
--out ~/.jd2/folderwatch
|
||
```
|
||
|
||
JDownloader runs on the desktop while the archive lives on the NAS, so tell it
|
||
the path *it* sees:
|
||
|
||
```bash
|
||
npm run jd2 -- --archives /mnt/nas/Instagram-archive/archives \
|
||
--download-base 'Z:\Instagram-archive\archives' \
|
||
--out ~/.jd2/folderwatch
|
||
```
|
||
|
||
| Flag | Purpose |
|
||
|---|---|
|
||
| `--archives <dir>` | Archive root to scan (or `$ARCHIVES_DIR`) |
|
||
| `--out <dir>` | JDownloader folder-watch directory |
|
||
| `--download-base <dir>` | Root path as JDownloader sees it (Windows paths fine) |
|
||
| `--user <name>` | Just this profile (repeatable) |
|
||
| `--skip <name>` | Never emit jobs for this directory (repeatable) |
|
||
| `--chunks <n>` | Connections per file (default 1) |
|
||
| `--auto-start` | Start immediately instead of parking in LinkGrabber |
|
||
| `--all-reels` | Emit a reels job even where no reels directory exists |
|
||
| `--dry-run` | Print instead of writing |
|
||
|
||
Defaults are deliberately conservative: `chunks=1`, and links park in the
|
||
LinkGrabber for review rather than auto-starting.
|
||
|
||
Only posts and reels are emitted. Highlight URLs need a numeric id and story
|
||
URLs expire, so those stay manual.
|
||
|
||
Directories that aren't Instagram profiles are skipped by username shape
|
||
(letters, digits, dots, underscores, ≤30 chars) — pointing a crawl at those
|
||
spends `instagram.com` requests to be told the profile doesn't exist. For names
|
||
that *look* like usernames but aren't, use `--skip` or a `.jd2ignore` file in
|
||
the archive root, one name per line.
|
||
|
||
Format reference: `src/org/jdownloader/extensions/folderwatchV2/explain.txt`.
|
||
JDownloader develops on SVN — read it via the daily mirror at
|
||
<https://github.com/mycodedoesnotcompile2/jdownloader_mirror> (`svn_trunk/`),
|
||
not one of the abandoned GitHub copies.
|
||
|
||
## Expected layout
|
||
|
||
Everything downloads into `<archives>/`, one directory per source:
|
||
|
||
```
|
||
archives/
|
||
0ct0ber19/ posts
|
||
0ct0ber19 - reels/ reels
|
||
story - 0ct0ber19/ stories
|
||
story highlights - 0ct0ber19 - Heestory/ a highlight
|
||
```
|
||
|
||
Non-archive directories (tool output, exports from elsewhere) live *outside*
|
||
`archives/` so they never reach the viewer.
|
||
|
||
The server picks up changes automatically — its index is keyed on directory
|
||
mtime, so a new file invalidates only that directory.
|
||
|
||
## If something goes wrong
|
||
|
||
**429 / rate limited** — stop for hours, not seconds. Retrying into a limit is
|
||
what converts a soft throttle into something worse.
|
||
|
||
**Cookie stops working** — re-export from the browser. Don't add a login step.
|
||
|
||
**Files land in the wrong folder** — a Packagizer rule is overriding the job.
|
||
Generated jobs set `overwritePackagizerEnabled=TRUE` to prevent this; check that
|
||
rules aren't set to run after it.
|
||
|
||
**Viewer doesn't show new posts** — check the file is in the right directory and
|
||
matches the naming pattern (`YYYY-MM-DD_<user> - <shortcode>[ - NN].<ext>`).
|
||
The index refreshes on directory mtime, so a genuinely new file is picked up on
|
||
the next request.
|