instaarchive-viewer/docs/jdownloader.md

JDownloader2 — archive fetching

This file lives only on the tooling branch. main is published to GitHub and deliberately carries none of this — not the host details, not the IPs, and not the account names. main's history was redacted on 2026-08-20; real names exist only here.

There is no npm run jd2 script — package.json and CLAUDE.md are kept byte-identical to main so that merging main into tooling never conflicts. Run the crawljob generator directly:

npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run

How content gets into this archive, and why the setup is shaped the way it is.

Why JDownloader and not Instaloader

There are two surfaces, and they're treated very differently:

Surface	What hits it	Risk
instagram.com	profile pages, GraphQL/API metadata	Tied to your session, heavily rate-limited. This is where bans come from.
scontent*.cdninstagram.com	the actual media	Signed URLs, CDN-served, tolerant. Mostly a bandwidth question.

JDownloader does nearly all its work on the CDN. Instaloader's value — the rich .json.xz metadata — comes from asking instagram.com a question per post.

Concretely, from this archive: rivvsofficial has 188 post-metadata files, so backfilling it cost 188 API requests for one 605-file profile. That's the ban vector. Downloading the 238 photos was never the problem.

Instaloader got this account banned once. JDownloader with throttling did not.

The account was suspended anyway, on 2026-08-17, for "spam". Not by JDownloader, and not by downloading. It was suspended during a day of building and verifying the gallery-dl replacement — automated browser scrolling to enumerate profile grids, repeated --simulate and -j metadata passes, and one aborted sync that re-ran every listing pass before dying.

The framing above is right about which surface is dangerous and wrong about what reaches it. Every read of instagram.com counts, including the ones that download nothing — and read-only work is easy not to count precisely because it leaves no files behind. See the post-mortem at the top of docs/gallery-dl.md.

The rule that would have prevented it: verify against the archive, never against the live site, and treat the first CDN 429 as the end of the session rather than a pacing knob.

What the metadata gap actually costs

Comparing a JDownloader profile against an Instaloader one:

	JDownloader	Instaloader
Media	✅	✅
Captions (.txt)	✅	✅
Dates (from filenames)	✅	✅
Bio / full name	❌	✅
Follower counts	❌	✅
External URL	❌	✅

Captions already work — the viewer reads the .txt sidecars. Everything missing lives in a single profile-level record, not the per-post ones. That's why JDownloader-sourced profiles show "0 followers" and a placeholder bio.

Not worth extra requests. If you ever want it, the zero-request option is a hand-written profile.json sidecar (not implemented yet — ask).

Settings that matter

Chunks per download → 1. The single most important one. JDownloader splits each file into multiple ranged requests by default; that Range pattern looks nothing like a browser or the app. One chunk = one sequential GET per file. jd2-sync sets chunks=1 per job, so no global change is needed — but set it globally too if you ever add links by hand.

Max simultaneous downloads → 2–3, connections-per-host low. Concurrency is what turns "a user" into a statistic.

Leave reconnect / IP-change features off. A mid-session IP change on a live cookie is a stronger anomaly signal than the request rate you'd be avoiding.

The cookie

Exported manually from a real browser session. This is the right approach — no programmatic login anywhere, which is the thing that actually gets flagged.

• Use it from the same public IP as the browser it came from. A cookie used from a different network is what session-hijack detection looks for.
• When it expires, re-export from the browser. Never add a login step to a tool.
• It's a full account credential. Keep it off the NAS share and out of the repo.

Workflow

Two URLs per profile, because the profile grid misses some reels:

https://www.instagram.com/<user>/
https://www.instagram.com/<user>/reels/

They overlap slightly — a reel caught by both lands in each directory and shows up twice in the viewer. That's correct and matches Instagram, which also shows reels in the profile grid and the Reels tab.

Generating jobs

Instead of pasting URLs and setting output folders by hand:

npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives --dry-run

Review, then write it into JDownloader's folder-watch directory:

npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives \
               --out ~/.jd2/folderwatch

JDownloader runs on the desktop while the archive lives on the NAS, so tell it the path it sees:

npm run jd2 -- --archives /mnt/nas/Instagram-archive/archives \
               --download-base 'Z:\Instagram-archive\archives' \
               --out ~/.jd2/folderwatch

Flag	Purpose
--archives <dir>	Archive root to scan (or $ARCHIVES_DIR)
--out <dir>	JDownloader folder-watch directory
--download-base <dir>	Root path as JDownloader sees it (Windows paths fine)
--user <name>	Just this profile (repeatable)
--skip <name>	Never emit jobs for this directory (repeatable)
--chunks <n>	Connections per file (default 1)
--auto-start	Start immediately instead of parking in LinkGrabber
--all-reels	Emit a reels job even where no reels directory exists
--dry-run	Print instead of writing

Defaults are deliberately conservative: chunks=1, and links park in the LinkGrabber for review rather than auto-starting.

Only posts and reels are emitted. Highlight URLs need a numeric id and story URLs expire, so those stay manual.

Directories that aren't Instagram profiles are skipped by username shape (letters, digits, dots, underscores, ≤30 chars) — pointing a crawl at those spends instagram.com requests to be told the profile doesn't exist. For names that look like usernames but aren't, use --skip or a .jd2ignore file in the archive root, one name per line.

Format reference: src/org/jdownloader/extensions/folderwatchV2/explain.txt. JDownloader develops on SVN — read it via the daily mirror at https://github.com/mycodedoesnotcompile2/jdownloader_mirror (svn_trunk/), not one of the abandoned GitHub copies.

Expected layout

Everything downloads into <archives>/, one directory per source:

archives/
  0ct0ber19/                                 posts
  0ct0ber19 - reels/                         reels
  story - 0ct0ber19/                         stories
  story highlights - 0ct0ber19 - Heestory/   a highlight

Non-archive directories (tool output, exports from elsewhere) live outside archives/ so they never reach the viewer.

The server picks up changes automatically — its index is keyed on directory mtime, so a new file invalidates only that directory.

If something goes wrong

429 / rate limited — stop for hours, not seconds. Retrying into a limit is what converts a soft throttle into something worse.

Cookie stops working — re-export from the browser. Don't add a login step.

Files land in the wrong folder — a Packagizer rule is overriding the job. Generated jobs set overwritePackagizerEnabled=TRUE to prevent this; check that rules aren't set to run after it.

Viewer doesn't show new posts — check the file is in the right directory and matches the naming pattern (YYYY-MM-DD_<user> - <shortcode>[ - NN].<ext>). The index refreshes on directory mtime, so a genuinely new file is picked up on the next request.