diff --git a/CLAUDE.md b/CLAUDE.md index 20c5bf2..bfc7d72 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,7 +16,8 @@ InstaArchive Viewer is a React 19 + Vite 6 PWA for browsing archived Instagram d - `npm test` / `npm run test:watch` — vitest - `npx vitest run src/lib/archive-patterns.test.ts` — a single test file - `npm run jd2 -- --archives --dry-run` — generate JDownloader `.crawljob` - files for every profile on disk (see `scripts/jd2-sync.ts`) + files for every profile on disk (see `scripts/jd2-sync.ts` and + `docs/jdownloader.md`) Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not. diff --git a/docs/jdownloader.md b/docs/jdownloader.md new file mode 100644 index 0000000..3f61988 --- /dev/null +++ b/docs/jdownloader.md @@ -0,0 +1,165 @@ +# JDownloader2 — archive fetching quick reference + +How content gets into this archive, and why the setup is shaped the way it is. + +## Why JDownloader and not Instaloader + +There are two surfaces, and they're treated very differently: + +| Surface | What hits it | Risk | +|---|---|---| +| `instagram.com` | profile pages, GraphQL/API metadata | Tied to your session, heavily rate-limited. **This is where bans come from.** | +| `scontent*.cdninstagram.com` | the actual media | Signed URLs, CDN-served, tolerant. Mostly a bandwidth question. | + +JDownloader does nearly all its work on the CDN. Instaloader's value — the rich +`.json.xz` metadata — comes from asking `instagram.com` a question *per post*. + +Concretely, from this archive: `rivvsofficial` has 188 post-metadata files, so +backfilling it cost 188 API requests for one 605-file profile. That's the ban +vector. Downloading the 238 photos was never the problem. + +Instaloader got this account banned once. JDownloader with throttling has not. + +### What the metadata gap actually costs + +Comparing a JDownloader profile against an Instaloader one: + +| | JDownloader | Instaloader | +|---|---|---| +| Media | ✅ | ✅ | +| Captions (`.txt`) | ✅ | ✅ | +| Dates (from filenames) | ✅ | ✅ | +| Bio / full name | ❌ | ✅ | +| Follower counts | ❌ | ✅ | +| External URL | ❌ | ✅ | + +Captions already work — the viewer reads the `.txt` sidecars. Everything missing +lives in a *single* profile-level record, not the per-post ones. That's why +JDownloader-sourced profiles show "0 followers" and a placeholder bio. + +Not worth extra requests. If you ever want it, the zero-request option is a +hand-written `profile.json` sidecar (not implemented yet — ask). + +## Settings that matter + +**Chunks per download → 1.** The single most important one. JDownloader splits +each file into multiple ranged requests by default; that `Range` pattern looks +nothing like a browser or the app. One chunk = one sequential GET per file. +`jd2-sync` sets `chunks=1` per job, so no global change is needed — but set it +globally too if you ever add links by hand. + +**Max simultaneous downloads → 2–3**, connections-per-host low. Concurrency is +what turns "a user" into a statistic. + +**Leave reconnect / IP-change features off.** A mid-session IP change on a live +cookie is a *stronger* anomaly signal than the request rate you'd be avoiding. + +## The cookie + +Exported manually from a real browser session. This is the right approach — no +programmatic login anywhere, which is the thing that actually gets flagged. + +- Use it from the **same public IP** as the browser it came from. A cookie used + from a different network is what session-hijack detection looks for. +- When it expires, **re-export from the browser**. Never add a login step to a tool. +- It's a full account credential. Keep it off the NAS share and out of the repo. + +## Workflow + +Two URLs per profile, because the profile grid misses some reels: + +``` +https://www.instagram.com// +https://www.instagram.com//reels/ +``` + +They overlap slightly — a reel caught by both lands in each directory and shows +up twice in the viewer. That's correct and matches Instagram, which also shows +reels in the profile grid *and* the Reels tab. + +## Generating jobs + +Instead of pasting URLs and setting output folders by hand: + +```bash +npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives --dry-run +``` + +Review, then write it into JDownloader's folder-watch directory: + +```bash +npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives \ + --out ~/.jd2/folderwatch +``` + +JDownloader runs on the desktop while the archive lives on the NAS, so tell it +the path *it* sees: + +```bash +npm run jd2 -- --archives /mnt/nas/Instagram-archive/archives \ + --download-base 'Z:\Instagram-archive\archives' \ + --out ~/.jd2/folderwatch +``` + +| Flag | Purpose | +|---|---| +| `--archives ` | Archive root to scan (or `$ARCHIVES_DIR`) | +| `--out ` | JDownloader folder-watch directory | +| `--download-base ` | Root path as JDownloader sees it (Windows paths fine) | +| `--user ` | Just this profile (repeatable) | +| `--skip ` | Never emit jobs for this directory (repeatable) | +| `--chunks ` | Connections per file (default 1) | +| `--auto-start` | Start immediately instead of parking in LinkGrabber | +| `--all-reels` | Emit a reels job even where no reels directory exists | +| `--dry-run` | Print instead of writing | + +Defaults are deliberately conservative: `chunks=1`, and links park in the +LinkGrabber for review rather than auto-starting. + +Only posts and reels are emitted. Highlight URLs need a numeric id and story +URLs expire, so those stay manual. + +Directories that aren't Instagram profiles are skipped by username shape +(letters, digits, dots, underscores, ≤30 chars) — pointing a crawl at those +spends `instagram.com` requests to be told the profile doesn't exist. For names +that *look* like usernames but aren't, use `--skip` or a `.jd2ignore` file in +the archive root, one name per line. + +Format reference: `src/org/jdownloader/extensions/folderwatchV2/explain.txt`. +JDownloader develops on SVN — read it via the daily mirror at + (`svn_trunk/`), +not one of the abandoned GitHub copies. + +## Expected layout + +Everything downloads into `/`, one directory per source: + +``` +archives/ + 0ct0ber19/ posts + 0ct0ber19 - reels/ reels + story - 0ct0ber19/ stories + story highlights - 0ct0ber19 - Heestory/ a highlight +``` + +Non-archive directories (tool output, exports from elsewhere) live *outside* +`archives/` so they never reach the viewer. + +The server picks up changes automatically — its index is keyed on directory +mtime, so a new file invalidates only that directory. + +## If something goes wrong + +**429 / rate limited** — stop for hours, not seconds. Retrying into a limit is +what converts a soft throttle into something worse. + +**Cookie stops working** — re-export from the browser. Don't add a login step. + +**Files land in the wrong folder** — a Packagizer rule is overriding the job. +Generated jobs set `overwritePackagizerEnabled=TRUE` to prevent this; check that +rules aren't set to run after it. + +**Viewer doesn't show new posts** — check the file is in the right directory and +matches the naming pattern (`YYYY-MM-DD_ - [ - NN].`). +The index refreshes on directory mtime, so a genuinely new file is picked up on +the next request.