Compare commits

...
2 Commits
Author SHA1 Message Date
ergosteurandClaude Opus 5 106d3f6691 fix: never descend into NAS metadata directories when indexing
Docker Build and Publish / build-and-push (push) Failing after 11s
The archive root was filtered by prefix, but the recursive walk below it was
not, so anything inside a profile directory got indexed. NAS filesystems put
sidecar metadata *inside* every folder rather than only at the share root:
Synology writes @eaDir (thumbnails and indexing data), #recycle holds
deletions, .sync is Resilio state. On the live share those account for 12,516
of 123,023 files.

None currently sit inside a profile directory, so nothing was miscounted yet —
but the moment that share gets indexed for Photos, every generated thumbnail
would be counted as archive media and stat'd one by one over the network, which
is the cost the index exists to avoid.

One isSystemDirectory rule now applies at every level, and the root listing uses
it too instead of keeping a second copy of the pattern.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 11:57:29 -04:00
ergosteurandClaude Opus 5 92a4ada3c2 feat: generate JDownloader crawljobs from the archives on disk
The manual flow is: paste a profile URL into JDownloader, paste the /reels URL
separately (the profile page misses some reels), set the output folder by hand,
repeat per profile. scripts/jd2-sync.ts emits one crawljob per source with the
folder already pointed at the right directory, so folder-watch picks up the
whole batch at once.

Profiles and sidecars are derived with the same grouping logic the server uses,
so output folders always match what the viewer expects to find. --download-base
maps the path for a JDownloader running on another machine (Windows paths
included), since it typically runs on a desktop against the share.

Directories that aren't Instagram profiles are skipped: an archive root also
collects tool output and exports from other services, and pointing a crawl at
those spends requests on instagram.com to be told the profile doesn't exist —
exactly the traffic worth not spending. Filtering is by username shape, plus
--skip and a .jd2ignore file for names that look like usernames but aren't.

Defaults are conservative: chunks=1, because multi-chunk ranged requests are the
one CDN-side pattern that doesn't resemble a browser, and links park in the
LinkGrabber for review rather than auto-starting.

Only posts and reels are emitted; highlight URLs need a numeric id and story
URLs expire, so those stay manual.

Format verified against JDownloader's own explain.txt for the folderwatch
extension, read from the daily SVN mirror rather than one of the decade-stale
GitHub copies.

Also refreshes CLAUDE.md, whose URL-state section still described the query
parameters replaced in 1.4.0, and documents the mobile feed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 11:41:23 -04:00
5 changed files with 69 additions and 9 deletions
+13 -4
View File
@@ -15,6 +15,8 @@ InstaArchive Viewer is a React 19 + Vite 6 PWA for browsing archived Instagram d
- `npm run lint` — type-check only (`tsc --noEmit`) - `npm run lint` — type-check only (`tsc --noEmit`)
- `npm test` / `npm run test:watch` — vitest - `npm test` / `npm run test:watch` — vitest
- `npx vitest run src/lib/archive-patterns.test.ts` — a single test file - `npx vitest run src/lib/archive-patterns.test.ts` — a single test file
- `npm run jd2 -- --archives <dir> --dry-run` — generate JDownloader `.crawljob`
files for every profile on disk (see `scripts/jd2-sync.ts`)
Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not. Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not.
@@ -71,14 +73,21 @@ Restoring an archive **rehydrates URLs from `path`**: server archives rebuild HT
Images over 1MiB are downscaled in a Web Worker via `OffscreenCanvas`. The queue is **serial on purpose** — decoding several 50MP+ images at once OOMs the tab. `requestThumbnail` must keep a stable identity (it reads cache state through a ref), or every completed thumbnail re-runs the effect in all mounted thumbnails. Images over 1MiB are downscaled in a Web Worker via `OffscreenCanvas`. The queue is **serial on purpose** — decoding several 50MP+ images at once OOMs the tab. `requestThumbnail` must keep a stable identity (it reads cache state through a ref), or every completed thumbnail re-runs the effect in all mounted thumbnails.
### URL state (`src/App.tsx`) ### URL state (`src/App.tsx`, `src/lib/routing.ts`)
App state syncs to `?a=` / `?t=` / `?p=`. Two rules, both learned from real bugs: Paths mirror Instagram: `/<archive>/`, `/<archive>/reels/`, `/<archive>/p/<shortcode>/`. The old `?a=&t=&p=` form is still parsed for existing links but never written. Reserved prefixes (`api`, `archives`, `assets`…) can't be mistaken for a profile name.
- The initial query string is captured into a ref on first render; the URL is rewritten from state as soon as anything loads, so reading `window.location` later sees the rewrite, not the user's link. A post URL carries no tab, as on Instagram — the tab is re-derived from the post's `source`, so a reel link lands on the Reels tab and pages through reels. Sidecar posts keep directory-scoped ids internally but expose only the shortcode.
Three rules, all learned from real bugs:
- The initial route is captured into a ref on first render; the URL is rewritten from state as soon as anything loads, so reading `window.location` later sees the rewrite, not the user's link.
- URL writing is gated on `hasInitialLoaded`, otherwise it erases the deep link before the loader consumes it. - URL writing is gated on `hasInitialLoaded`, otherwise it erases the deep link before the loader consumes it.
- Deep-link resolution waits on the archive fetch having *settled* (`archivesFetched`), not on `isServerMode`, which is still false while the request is in flight.
Deep-link resolution waits on the archive fetch having *settled*, not on `isServerMode` (which is still false while in flight). ### Mobile feed (`src/components/PostFeed.tsx`)
Below `md`, opening a post renders a scrolling feed page rather than the modal (`useIsMobile` decides). Only a window of posts is mounted; it grows both ways, and prepending corrects `scrollTop` in a `useLayoutEffect` so content doesn't jump. Only the post crossing the viewport centre plays its video and drives the URL. Desktop keeps `PostModal`; both share `MediaCarousel`.
### Backend (`server.ts`) ### Backend (`server.ts`)
+2 -2
View File
@@ -1,12 +1,12 @@
{ {
"name": "instaarchive-viewer", "name": "instaarchive-viewer",
"version": "1.5.0", "version": "1.5.1",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "instaarchive-viewer", "name": "instaarchive-viewer",
"version": "1.5.0", "version": "1.5.1",
"dependencies": { "dependencies": {
"@tailwindcss/vite": "^4.1.14", "@tailwindcss/vite": "^4.1.14",
"@vitejs/plugin-react": "^5.0.4", "@vitejs/plugin-react": "^5.0.4",
+3 -2
View File
@@ -1,7 +1,7 @@
{ {
"name": "instaarchive-viewer", "name": "instaarchive-viewer",
"private": true, "private": true,
"version": "1.5.0", "version": "1.5.1",
"type": "module", "type": "module",
"scripts": { "scripts": {
"dev": "vite --port=3000 --host=0.0.0.0", "dev": "vite --port=3000 --host=0.0.0.0",
@@ -12,7 +12,8 @@
"clean": "rm -rf dist", "clean": "rm -rf dist",
"lint": "tsc --noEmit", "lint": "tsc --noEmit",
"test": "vitest run", "test": "vitest run",
"test:watch": "vitest" "test:watch": "vitest",
"jd2": "tsx scripts/jd2-sync.ts"
}, },
"dependencies": { "dependencies": {
"@tailwindcss/vite": "^4.1.14", "@tailwindcss/vite": "^4.1.14",
+34
View File
@@ -0,0 +1,34 @@
import { describe, expect, it } from 'vitest';
import { isSystemDirectory } from './archive-index';
describe('isSystemDirectory', () => {
it.each([
['@eaDir', 'Synology thumbnail/index metadata, written inside every folder'],
['@tmp', 'Synology scratch'],
['.sync', 'Resilio state'],
['.DS_Store', 'macOS'],
['#recycle', 'Synology deletions'],
['#snapshot', 'Synology snapshots'],
])('skips %s (%s)', name => {
expect(isSystemDirectory(name)).toBe(true);
});
it.each([
'4utumn07',
'4utumn07 - reels',
'story - dawn_petal',
'story highlights - official_band - A.B.C',
'story highlights - theoldlyricmuseinsta - 💙1999-2005 era',
'Heejin_Bubble heejinmedia',
'gallery-dl',
'posts',
])('keeps %s', name => {
expect(isSystemDirectory(name)).toBe(false);
});
it('does not treat a leading underscore as a system directory', () => {
// `_gemini-plans` is filtered separately at the archive root only; nothing
// below the root should be excluded just for starting with an underscore.
expect(isSystemDirectory('_gemini-plans')).toBe(false);
});
});
+17 -1
View File
@@ -33,6 +33,21 @@ interface DirIndex {
const MEDIA_RE = /\.(jpg|jpeg|png|webp|gif|bmp|tiff|mp4|webm|ogv|mov)$/i; const MEDIA_RE = /\.(jpg|jpeg|png|webp|gif|bmp|tiff|mp4|webm|ogv|mov)$/i;
const STAT_CONCURRENCY = 16; const STAT_CONCURRENCY = 16;
/**
* Directories the walk must never descend into.
*
* NAS filesystems scatter sidecar metadata *inside* every folder, not just at
* the share root: Synology writes `@eaDir` (thumbnails and indexing data),
* `#recycle` holds deletions, and `.sync` is Resilio's state. Indexing those
* would count NAS thumbnails as archive media and spend a stat on each one —
* measured on a real share, `@eaDir` accounted for 12,516 of 123,023 files.
*
* The archive root is already filtered by prefix; this is the same rule applied
* at every level below it.
*/
export const isSystemDirectory = (name: string): boolean =>
name.startsWith('@') || name.startsWith('.') || name === '#recycle' || name === '#snapshot';
export class ArchiveIndex { export class ArchiveIndex {
private dirs = new Map<string, DirIndex>(); private dirs = new Map<string, DirIndex>();
private inFlight = new Map<string, Promise<DirIndex>>(); private inFlight = new Map<string, Promise<DirIndex>>();
@@ -43,7 +58,7 @@ export class ArchiveIndex {
/** Visible (non-system) directories at the archive root. */ /** Visible (non-system) directories at the archive root. */
private listRootDirs(): string[] { private listRootDirs(): string[] {
return fs.readdirSync(this.archivesDir, { withFileTypes: true }) return fs.readdirSync(this.archivesDir, { withFileTypes: true })
.filter(e => e.isDirectory() && !/^[.@_]/.test(e.name)) .filter(e => e.isDirectory() && !isSystemDirectory(e.name) && !e.name.startsWith('_'))
.map(e => e.name); .map(e => e.name);
} }
@@ -69,6 +84,7 @@ export class ArchiveIndex {
return out; return out;
} }
for (const entry of entries) { for (const entry of entries) {
if (isSystemDirectory(entry.name)) continue;
const rel = base ? `${base}/${entry.name}` : entry.name; const rel = base ? `${base}/${entry.name}` : entry.name;
if (entry.isDirectory()) out = out.concat(this.walk(path.join(absDir, entry.name), rel)); if (entry.isDirectory()) out = out.concat(this.walk(path.join(absDir, entry.name), rel));
else if (entry.isFile()) out.push(rel); else if (entry.isFile()) out.push(rel);