# JDownloader2 — archive fetching > **This file lives only on the `tooling` branch.** `main` is published to > GitHub and deliberately carries none of this — not the host details, not the > IPs, and not the account names. `main`'s history was redacted on 2026-08-20; > real names exist only here. > > There is no `npm run jd2` script — `package.json` and `CLAUDE.md` are kept > byte-identical to `main` so that merging `main` into `tooling` never > conflicts. Run the crawljob generator directly: > > ```sh > npx tsx scripts/jd2-sync.ts --archives --dry-run > ``` How content gets into this archive, and why the setup is shaped the way it is. ## Why JDownloader and not Instaloader There are two surfaces, and they're treated very differently: | Surface | What hits it | Risk | |---|---|---| | `instagram.com` | profile pages, GraphQL/API metadata | Tied to your session, heavily rate-limited. **This is where bans come from.** | | `scontent*.cdninstagram.com` | the actual media | Signed URLs, CDN-served, tolerant. Mostly a bandwidth question. | JDownloader does nearly all its work on the CDN. Instaloader's value — the rich `.json.xz` metadata — comes from asking `instagram.com` a question *per post*. Concretely, from this archive: `rivvsofficial` has 188 post-metadata files, so backfilling it cost 188 API requests for one 605-file profile. That's the ban vector. Downloading the 238 photos was never the problem. Instaloader got this account banned once. JDownloader with throttling did not. > **The account was suspended anyway, on 2026-08-17, for "spam".** Not by > JDownloader, and not by downloading. It was suspended during a day of > *building and verifying* the gallery-dl replacement — automated browser > scrolling to enumerate profile grids, repeated `--simulate` and `-j` metadata > passes, and one aborted sync that re-ran every listing pass before dying. > > The framing above is right about which surface is dangerous and wrong about > what reaches it. **Every read of `instagram.com` counts, including the ones > that download nothing** — and read-only work is easy not to count precisely > because it leaves no files behind. See the post-mortem at the top of > `docs/gallery-dl.md`. > > The rule that would have prevented it: *verify against the archive, never > against the live site*, and treat the first CDN `429` as the end of the > session rather than a pacing knob. ### What the metadata gap actually costs Comparing a JDownloader profile against an Instaloader one: | | JDownloader | Instaloader | |---|---|---| | Media | ✅ | ✅ | | Captions (`.txt`) | ✅ | ✅ | | Dates (from filenames) | ✅ | ✅ | | Bio / full name | ❌ | ✅ | | Follower counts | ❌ | ✅ | | External URL | ❌ | ✅ | Captions already work — the viewer reads the `.txt` sidecars. Everything missing lives in a *single* profile-level record, not the per-post ones. That's why JDownloader-sourced profiles show "0 followers" and a placeholder bio. Not worth extra requests. If you ever want it, the zero-request option is a hand-written `profile.json` sidecar (not implemented yet — ask). ## Settings that matter **Chunks per download → 1.** The single most important one. JDownloader splits each file into multiple ranged requests by default; that `Range` pattern looks nothing like a browser or the app. One chunk = one sequential GET per file. `jd2-sync` sets `chunks=1` per job, so no global change is needed — but set it globally too if you ever add links by hand. **Max simultaneous downloads → 2–3**, connections-per-host low. Concurrency is what turns "a user" into a statistic. **Leave reconnect / IP-change features off.** A mid-session IP change on a live cookie is a *stronger* anomaly signal than the request rate you'd be avoiding. ## The cookie Exported manually from a real browser session. This is the right approach — no programmatic login anywhere, which is the thing that actually gets flagged. - Use it from the **same public IP** as the browser it came from. A cookie used from a different network is what session-hijack detection looks for. - When it expires, **re-export from the browser**. Never add a login step to a tool. - It's a full account credential. Keep it off the NAS share and out of the repo. ## Workflow Two URLs per profile, because the profile grid misses some reels: ``` https://www.instagram.com// https://www.instagram.com//reels/ ``` They overlap slightly — a reel caught by both lands in each directory and shows up twice in the viewer. That's correct and matches Instagram, which also shows reels in the profile grid *and* the Reels tab. ## Generating jobs Instead of pasting URLs and setting output folders by hand: ```bash npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives --dry-run ``` Review, then write it into JDownloader's folder-watch directory: ```bash npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives \ --out ~/.jd2/folderwatch ``` JDownloader runs on the desktop while the archive lives on the NAS, so tell it the path *it* sees: ```bash npm run jd2 -- --archives /mnt/nas/Instagram-archive/archives \ --download-base 'Z:\Instagram-archive\archives' \ --out ~/.jd2/folderwatch ``` | Flag | Purpose | |---|---| | `--archives ` | Archive root to scan (or `$ARCHIVES_DIR`) | | `--out ` | JDownloader folder-watch directory | | `--download-base ` | Root path as JDownloader sees it (Windows paths fine) | | `--user ` | Just this profile (repeatable) | | `--skip ` | Never emit jobs for this directory (repeatable) | | `--chunks ` | Connections per file (default 1) | | `--auto-start` | Start immediately instead of parking in LinkGrabber | | `--all-reels` | Emit a reels job even where no reels directory exists | | `--dry-run` | Print instead of writing | Defaults are deliberately conservative: `chunks=1`, and links park in the LinkGrabber for review rather than auto-starting. Only posts and reels are emitted. Highlight URLs need a numeric id and story URLs expire, so those stay manual. Directories that aren't Instagram profiles are skipped by username shape (letters, digits, dots, underscores, ≤30 chars) — pointing a crawl at those spends `instagram.com` requests to be told the profile doesn't exist. For names that *look* like usernames but aren't, use `--skip` or a `.jd2ignore` file in the archive root, one name per line. Format reference: `src/org/jdownloader/extensions/folderwatchV2/explain.txt`. JDownloader develops on SVN — read it via the daily mirror at (`svn_trunk/`), not one of the abandoned GitHub copies. ## Expected layout Everything downloads into `/`, one directory per source: ``` archives/ 0ct0ber19/ posts 0ct0ber19 - reels/ reels story - 0ct0ber19/ stories story highlights - 0ct0ber19 - Heestory/ a highlight ``` Non-archive directories (tool output, exports from elsewhere) live *outside* `archives/` so they never reach the viewer. The server picks up changes automatically — its index is keyed on directory mtime, so a new file invalidates only that directory. ## If something goes wrong **429 / rate limited** — stop for hours, not seconds. Retrying into a limit is what converts a soft throttle into something worse. **Cookie stops working** — re-export from the browser. Don't add a login step. **Files land in the wrong folder** — a Packagizer rule is overriding the job. Generated jobs set `overwritePackagizerEnabled=TRUE` to prevent this; check that rules aren't set to run after it. **Viewer doesn't show new posts** — check the file is in the right directory and matches the naming pattern (`YYYY-MM-DD_ - [ - NN].`). The index refreshes on directory mtime, so a genuinely new file is picked up on the next request.