ergosteurandClaude Opus 5
f0c054946f
Covers why fetching goes through JDownloader rather than Instaloader (the instagram.com vs CDN split, and what the metadata gap actually costs), the settings that matter, cookie handling, the two-URL workflow, jd2-sync usage, the expected on-disk layout, and what to do when something breaks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
166 lines
6.5 KiB
Markdown
166 lines
6.5 KiB
Markdown
# JDownloader2 — archive fetching quick reference
|
||
|
||
How content gets into this archive, and why the setup is shaped the way it is.
|
||
|
||
## Why JDownloader and not Instaloader
|
||
|
||
There are two surfaces, and they're treated very differently:
|
||
|
||
| Surface | What hits it | Risk |
|
||
|---|---|---|
|
||
| `instagram.com` | profile pages, GraphQL/API metadata | Tied to your session, heavily rate-limited. **This is where bans come from.** |
|
||
| `scontent*.cdninstagram.com` | the actual media | Signed URLs, CDN-served, tolerant. Mostly a bandwidth question. |
|
||
|
||
JDownloader does nearly all its work on the CDN. Instaloader's value — the rich
|
||
`.json.xz` metadata — comes from asking `instagram.com` a question *per post*.
|
||
|
||
Concretely, from this archive: `rivvsofficial` has 188 post-metadata files, so
|
||
backfilling it cost 188 API requests for one 605-file profile. That's the ban
|
||
vector. Downloading the 238 photos was never the problem.
|
||
|
||
Instaloader got this account banned once. JDownloader with throttling has not.
|
||
|
||
### What the metadata gap actually costs
|
||
|
||
Comparing a JDownloader profile against an Instaloader one:
|
||
|
||
| | JDownloader | Instaloader |
|
||
|---|---|---|
|
||
| Media | ✅ | ✅ |
|
||
| Captions (`.txt`) | ✅ | ✅ |
|
||
| Dates (from filenames) | ✅ | ✅ |
|
||
| Bio / full name | ❌ | ✅ |
|
||
| Follower counts | ❌ | ✅ |
|
||
| External URL | ❌ | ✅ |
|
||
|
||
Captions already work — the viewer reads the `.txt` sidecars. Everything missing
|
||
lives in a *single* profile-level record, not the per-post ones. That's why
|
||
JDownloader-sourced profiles show "0 followers" and a placeholder bio.
|
||
|
||
Not worth extra requests. If you ever want it, the zero-request option is a
|
||
hand-written `profile.json` sidecar (not implemented yet — ask).
|
||
|
||
## Settings that matter
|
||
|
||
**Chunks per download → 1.** The single most important one. JDownloader splits
|
||
each file into multiple ranged requests by default; that `Range` pattern looks
|
||
nothing like a browser or the app. One chunk = one sequential GET per file.
|
||
`jd2-sync` sets `chunks=1` per job, so no global change is needed — but set it
|
||
globally too if you ever add links by hand.
|
||
|
||
**Max simultaneous downloads → 2–3**, connections-per-host low. Concurrency is
|
||
what turns "a user" into a statistic.
|
||
|
||
**Leave reconnect / IP-change features off.** A mid-session IP change on a live
|
||
cookie is a *stronger* anomaly signal than the request rate you'd be avoiding.
|
||
|
||
## The cookie
|
||
|
||
Exported manually from a real browser session. This is the right approach — no
|
||
programmatic login anywhere, which is the thing that actually gets flagged.
|
||
|
||
- Use it from the **same public IP** as the browser it came from. A cookie used
|
||
from a different network is what session-hijack detection looks for.
|
||
- When it expires, **re-export from the browser**. Never add a login step to a tool.
|
||
- It's a full account credential. Keep it off the NAS share and out of the repo.
|
||
|
||
## Workflow
|
||
|
||
Two URLs per profile, because the profile grid misses some reels:
|
||
|
||
```
|
||
https://www.instagram.com/<user>/
|
||
https://www.instagram.com/<user>/reels/
|
||
```
|
||
|
||
They overlap slightly — a reel caught by both lands in each directory and shows
|
||
up twice in the viewer. That's correct and matches Instagram, which also shows
|
||
reels in the profile grid *and* the Reels tab.
|
||
|
||
## Generating jobs
|
||
|
||
Instead of pasting URLs and setting output folders by hand:
|
||
|
||
```bash
|
||
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives --dry-run
|
||
```
|
||
|
||
Review, then write it into JDownloader's folder-watch directory:
|
||
|
||
```bash
|
||
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives \
|
||
--out ~/.jd2/folderwatch
|
||
```
|
||
|
||
JDownloader runs on the desktop while the archive lives on the NAS, so tell it
|
||
the path *it* sees:
|
||
|
||
```bash
|
||
npm run jd2 -- --archives /mnt/nas/Instagram-archive/archives \
|
||
--download-base 'Z:\Instagram-archive\archives' \
|
||
--out ~/.jd2/folderwatch
|
||
```
|
||
|
||
| Flag | Purpose |
|
||
|---|---|
|
||
| `--archives <dir>` | Archive root to scan (or `$ARCHIVES_DIR`) |
|
||
| `--out <dir>` | JDownloader folder-watch directory |
|
||
| `--download-base <dir>` | Root path as JDownloader sees it (Windows paths fine) |
|
||
| `--user <name>` | Just this profile (repeatable) |
|
||
| `--skip <name>` | Never emit jobs for this directory (repeatable) |
|
||
| `--chunks <n>` | Connections per file (default 1) |
|
||
| `--auto-start` | Start immediately instead of parking in LinkGrabber |
|
||
| `--all-reels` | Emit a reels job even where no reels directory exists |
|
||
| `--dry-run` | Print instead of writing |
|
||
|
||
Defaults are deliberately conservative: `chunks=1`, and links park in the
|
||
LinkGrabber for review rather than auto-starting.
|
||
|
||
Only posts and reels are emitted. Highlight URLs need a numeric id and story
|
||
URLs expire, so those stay manual.
|
||
|
||
Directories that aren't Instagram profiles are skipped by username shape
|
||
(letters, digits, dots, underscores, ≤30 chars) — pointing a crawl at those
|
||
spends `instagram.com` requests to be told the profile doesn't exist. For names
|
||
that *look* like usernames but aren't, use `--skip` or a `.jd2ignore` file in
|
||
the archive root, one name per line.
|
||
|
||
Format reference: `src/org/jdownloader/extensions/folderwatchV2/explain.txt`.
|
||
JDownloader develops on SVN — read it via the daily mirror at
|
||
<https://github.com/mycodedoesnotcompile2/jdownloader_mirror> (`svn_trunk/`),
|
||
not one of the abandoned GitHub copies.
|
||
|
||
## Expected layout
|
||
|
||
Everything downloads into `<archives>/`, one directory per source:
|
||
|
||
```
|
||
archives/
|
||
0ct0ber19/ posts
|
||
0ct0ber19 - reels/ reels
|
||
story - 0ct0ber19/ stories
|
||
story highlights - 0ct0ber19 - Heestory/ a highlight
|
||
```
|
||
|
||
Non-archive directories (tool output, exports from elsewhere) live *outside*
|
||
`archives/` so they never reach the viewer.
|
||
|
||
The server picks up changes automatically — its index is keyed on directory
|
||
mtime, so a new file invalidates only that directory.
|
||
|
||
## If something goes wrong
|
||
|
||
**429 / rate limited** — stop for hours, not seconds. Retrying into a limit is
|
||
what converts a soft throttle into something worse.
|
||
|
||
**Cookie stops working** — re-export from the browser. Don't add a login step.
|
||
|
||
**Files land in the wrong folder** — a Packagizer rule is overriding the job.
|
||
Generated jobs set `overwritePackagizerEnabled=TRUE` to prevent this; check that
|
||
rules aren't set to run after it.
|
||
|
||
**Viewer doesn't show new posts** — check the file is in the right directory and
|
||
matches the naming pattern (`YYYY-MM-DD_<user> - <shortcode>[ - NN].<ext>`).
|
||
The index refreshes on directory mtime, so a genuinely new file is picked up on
|
||
the next request.
|