docs: update CLAUDE.md for the archive index, sidecars and cache rehydration

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7

CLAUDE.md

@@ -4,63 +4,96 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-InstaArchive Viewer is a React 19 + Vite 6 PWA for browsing archived Instagram data (both official Instagram exports and Instaloader archives). All archive parsing and media processing happens client-side in the browser — the Express backend only lists/serves files from disk, it never parses archive contents.
+InstaArchive Viewer is a React 19 + Vite 6 PWA for browsing archived Instagram data (official Instagram exports and Instaloader archives). All archive *parsing* happens client-side in the browser; the Express backend only indexes and serves files from disk, and never parses archive contents.
 
 ## Commands
 
 - `npm install` — install dependencies
-- `npm run dev` — start Vite dev server on port 3000 (proxies `/api` and `/archives` to `http://localhost:3001`)
+- `npm run dev` — Vite dev server on port 3000 (proxies `/api` and `/archives` to `http://localhost:3001`)
-- `npm run server` — start the Express backend (`tsx server.ts`) on port 3001, serving archives from `ARCHIVES_DIR` (defaults to `./_sample-archives`)
+- `npm run server` — Express backend (`tsx server.ts`) on port 3001, serving `ARCHIVES_DIR` (defaults to `./_sample-archives`)
-- `npm run build` — build frontend to `dist/` (`vite build`) and backend to `dist-server/` (`tsc server.ts ...`)
+- `npm run build` — frontend to `dist/`, backend to `dist-server/`
-- `npm run lint` — type-check only (`tsc --noEmit`); there is no separate test suite or linter config
+- `npm run lint` — type-check only (`tsc --noEmit`)
-- `npm run clean` — remove `dist/`
+- `npm test` / `npm run test:watch` — vitest
+- `npx vitest run src/lib/archive-patterns.test.ts` — a single test file
 
-For local development you typically need both `npm run dev` and `npm run server` running concurrently — the frontend alone has nothing to talk to for server-mode archives (local-folder mode works without the backend).
+Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not.
 
 ## Architecture
 
 ### Two archive sources, one data model
 
-The app supports loading archives two ways, unified behind the `ArchiveFile` interface (`src/types/index.ts`, implementations in `src/lib/archive-files.ts`):
+Loading is unified behind the `ArchiveFile` interface (`src/types/index.ts`, implementations in `src/lib/archive-files.ts`):
 
-- **`LocalArchiveFile`** — wraps a browser `File` from a local folder picker (`webkitdirectory`). Fully offline, media is never uploaded anywhere.
+- **`LocalArchiveFile`** — wraps a browser `File`. `createObjectUrl()` mints a **disk-backed** blob URL directly from the File; never route media through `arrayBuffer()`, which pulls whole files into memory.
-- **`RemoteArchiveFile`** — wraps a file served from the Express backend's `/archives/:name/...` static route, fetched on demand.
+- **`RemoteArchiveFile`** — wraps a file served from `/archives/...`, fetched on demand.
 
-All downstream parsing code (`useArchiveScanner`) operates only on `ArchiveFile[]` and doesn't care which backing implementation it got.
+`revocable` tells callers whether the returned URL must be revoked. The scanner tracks every minted URL and releases them on archive teardown.
+
+### Sidecar directories
+
+An archive root holds one directory per profile plus *sidecars* that belong to it:
+
+```
+0ct0ber19                                 -> posts (base)
+0ct0ber19 - reels                         -> reels
+story - 0ct0ber19                         -> stories
+story highlights - 0ct0ber19 - Heestory   -> highlight "Heestory"
+```
+
+`src/lib/archive-grouping.ts` (shared by server and tests) folds these into a single profile with a `sources` list. Sidecars never appear as standalone archives. Each file the server returns carries its `kind`, so the client routes posts / reels / story ring / highlight circles without re-deriving naming rules.
+
+### Server-side archive index (`src/lib/archive-index.ts`)
+
+**Do not reintroduce per-request filesystem walks.** Archives typically live on network storage where per-file `stat` costs ~1.4ms and does not parallelise; a naive walk of a 110k-file root took ~52s per listing. Instead:
+
+- Each source directory is indexed once and cached, keyed by its **directory mtime** (a directory `stat` is effectively free).
+- The index is warmed in the background at startup and persisted to `CACHE_DIR` (mount a volume at `/cache`).
+- `GET /api/archives` does no file walking at all — it returns directory-mtime `signature`s, which the client uses for cache invalidation instead of a file count.
+- Only media files are stat'd (for `size`, which gates thumbnailing) and only highlights need `mtime` (their filenames carry no date).
 
 ### Scanning pipeline (`src/hooks/useArchiveScanner.ts`)
 
-This is the core of the app — a single large `handleFiles` function that:
+`handleFiles` indexes files, detects format, then parses via one of three largely independent paths that all write into a shared `postsMap`:
-1. **Indexes** all files, detecting archive format by filename regex: Instagram "export" format (`YYYY-MM-DD_user - post_id[- idx][- story].ext`), Instaloader format (`YYYY-MM-DD_HH-MM-SS_UTC[_idx][_story].ext`), or a generic JSON-manifest format (`posts_1.json`, `reels_1.json`, `stories_1.json`, possibly `.json.xz`-compressed via `xz-decompress`).
-2. **Parses** according to detected format, building a `Map<postId, Partial<Post>>`. For JSON-manifest format, media files are matched to JSON entries by URI substring match, then by ID substring match, then by filename-derived heuristic — in that fallback order.
-3. **Falls back** to generic filename-prefix grouping when no posts were found via regex/JSON matching (treats files sharing a common basename as one carousel post, chunked into groups of 20).
-4. Detects a **profile picture** from `*_profile_pic.jpg` / `<username>.jpg` files, or falls back to the oldest image in the archive by filename sort ("Smart Fallback").
-5. **Caches** the final `{ posts, stories, profileMetadata, ... }` result to IndexedDB via `idb-keyval`, keyed by archive name (or `local_archive` for unnamed local folders) — this is what makes repeat visits load instantly. Both server and local archives are cached; the cache shape is documented inline in `useArchiveScanner`'s state (mirrors the `CacheData` interface in `GEMINI.md`).
 
-When modifying format-detection or media-matching logic, be aware the three code paths (JSON-manifest, filename-regex export/instaloader, generic fallback) are largely independent and a change to one rarely needs to touch the others — but all three write into the same `postsMap`.
+1. **JSON manifest** (`posts_1.json`, possibly `.json.xz` via `xz-decompress`) — media matched to entries by URI, then ID, then filename heuristic, in that fallback order.
+2. **Filename patterns** — see `src/lib/archive-patterns.ts` for the export / Instaloader / highlight regexes, extracted as pure functions and covered by tests. Prefer changing them there.
+3. **Generic grouping fallback** — when nothing else matched, files sharing a basename become one carousel.
 
-### Thumbnail generation (`src/hooks/useThumbnailQueue.ts` + `src/lib/thumbnail-worker.ts`)
+Results are cached to IndexedDB. Media records store a stable `path`; **`url` is not persistable** for local archives because blob URLs die with the document.
 
-High-res images (>1MiB) are downscaled off the main thread:
+### Cache and local-archive persistence (`src/lib/archive-cache.ts`)
-- `useThumbnailQueue` maintains a **serial** (one-at-a-time) queue — this is deliberate, not a bug: decoding multiple 50MP+ images concurrently causes OOM crashes in the browser.
-- Actual resizing happens in `thumbnail-worker.ts` using `OffscreenCanvas`/`createImageBitmap` inside a Web Worker.
-- Results are cached in IndexedDB under a `thumb_<id>` key, checked before falling back to the worker, so thumbnails persist across sessions.
 
-### URL state sync (`src/App.tsx`)
+IndexedDB keys are namespaced (`archive:`, `thumb:`, `handle:`) so listing archives does not deserialize every cached thumbnail blob, and thumbnails are scoped per archive to avoid cross-archive collisions.
 
-App state (selected archive, active tab, selected post) is synchronized with URL query params (`?a=`, `?t=`, `?p=`) via `URLSearchParams` + `window.history.replaceState` in a cluster of `useEffect` hooks — this is what enables permalinks/deep-linking. When adding new shareable state, follow this pattern rather than introducing a router.
+Restoring an archive **rehydrates URLs from `path`**: server archives rebuild HTTP URLs; local archives re-open a persisted `FileSystemDirectoryHandle` and mint fresh blob URLs. If the folder is unreachable (permission lapsed, or the browser lacks `showDirectoryPicker` — Firefox/Safari), the app re-prompts rather than rendering broken images.
+
+### Thumbnails (`src/hooks/useThumbnailQueue.ts` + `src/lib/thumbnail-worker.ts`)
+
+Images over 1MiB are downscaled in a Web Worker via `OffscreenCanvas`. The queue is **serial on purpose** — decoding several 50MP+ images at once OOMs the tab. `requestThumbnail` must keep a stable identity (it reads cache state through a ref), or every completed thumbnail re-runs the effect in all mounted thumbnails.
+
+### URL state (`src/App.tsx`)
+
+App state syncs to `?a=` / `?t=` / `?p=`. Two rules, both learned from real bugs:
+
+- The initial query string is captured into a ref on first render; the URL is rewritten from state as soon as anything loads, so reading `window.location` later sees the rewrite, not the user's link.
+- URL writing is gated on `hasInitialLoaded`, otherwise it erases the deep link before the loader consumes it.
+
+Deep-link resolution waits on the archive fetch having *settled*, not on `isServerMode` (which is still false while in flight).
 
 ### Backend (`server.ts`)
 
-Minimal Express server, three responsibilities only:
+Serves `/api/archives`, `/api/archives/:name/files`, static `/archives`, and the built SPA. Notes:
-- `GET /api/archives` — lists subdirectories of `ARCHIVES_DIR` (skipping dotfiles/`@`/`_`-prefixed dirs) as `ServerArchive[]`, guessing a thumbnail per archive.
-- `GET /api/archives/:name/files` — recursively lists all files in one archive directory.
-- Static-serves `ARCHIVES_DIR` under `/archives` and, in production, serves the built `dist/` frontend with an SPA fallback.
 
-It does no parsing of archive/JSON contents — that's entirely client-side in `useArchiveScanner`. `ARCHIVES_DIR` is resolved from the `ARCHIVES_DIR` env var (see `.env` / Docker volume mount at `/archives`).
+- Express decodes route params **after** segment matching, so `..%2f` reaches the handler as `../`. All user-supplied archive names go through `resolveArchivePath`.
+- Sets CSP and related security headers. The CSP allows `blob:`/`data:` for media and `unsafe-inline` styles (the animation library sets inline styles); scripts stay same-origin only.
+- `os.userInfo()` throws for a UID with no `/etc/passwd` entry, which is what `--user 1234:1234` produces — use `describeUser()`.
+
+### Deployment
+
+Container runs as non-root. The image defaults to `node`, but the archive share must be *listable* by that UID — a mode-711 share owned by another account needs `user: "<uid>:<gid>"` in compose. Mount a volume at `/cache` so the index survives restarts.
 
 ### PWA / build quirks
 
-- `vite.config.ts` sets `hmr: process.env.DISABLE_HMR !== 'true'` — this is intentionally left alone; it exists to disable file-watch flicker when running under AI Studio-style agent editing. Don't "clean up" or remove it.
+- `vite.config.ts` sets `hmr: process.env.DISABLE_HMR !== 'true'` — intentional, leave it.
-- `workbox.navigateFallbackDenylist` excludes `/api` and `/archives` from the SPA fallback so those routes hit the real server/static files instead of `index.html` (needed for "open original file in new tab").
+- `workbox.navigateFallbackDenylist` excludes `/api` and `/archives` so those hit the real server.
-- Service worker uses `registerType: 'autoUpdate'` with hourly periodic checks — new deployments propagate to open clients automatically.
+- Fonts and icons are vendored in `public/` — do not reintroduce CDN references; the app advertises offline support and local-only processing.