Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8b053b4b2e | ||
|
|
d1fa1a8d2f | ||
|
|
b9ece021d4 | ||
|
|
f300b8d9f5 | ||
|
|
e57be521a2 | ||
|
|
792b834cbe | ||
|
|
106d3f6691 | ||
|
|
92a4ada3c2 |
@@ -15,6 +15,9 @@ InstaArchive Viewer is a React 19 + Vite 6 PWA for browsing archived Instagram d
|
|||||||
- `npm run lint` — type-check only (`tsc --noEmit`)
|
- `npm run lint` — type-check only (`tsc --noEmit`)
|
||||||
- `npm test` / `npm run test:watch` — vitest
|
- `npm test` / `npm run test:watch` — vitest
|
||||||
- `npx vitest run src/lib/archive-patterns.test.ts` — a single test file
|
- `npx vitest run src/lib/archive-patterns.test.ts` — a single test file
|
||||||
|
- `npm run jd2 -- --archives <dir> --dry-run` — generate JDownloader `.crawljob`
|
||||||
|
files for every profile on disk (see `scripts/jd2-sync.ts` and
|
||||||
|
`docs/jdownloader.md`)
|
||||||
|
|
||||||
Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not.
|
Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not.
|
||||||
|
|
||||||
@@ -71,25 +74,74 @@ Restoring an archive **rehydrates URLs from `path`**: server archives rebuild HT
|
|||||||
|
|
||||||
Images over 1MiB are downscaled in a Web Worker via `OffscreenCanvas`. The queue is **serial on purpose** — decoding several 50MP+ images at once OOMs the tab. `requestThumbnail` must keep a stable identity (it reads cache state through a ref), or every completed thumbnail re-runs the effect in all mounted thumbnails.
|
Images over 1MiB are downscaled in a Web Worker via `OffscreenCanvas`. The queue is **serial on purpose** — decoding several 50MP+ images at once OOMs the tab. `requestThumbnail` must keep a stable identity (it reads cache state through a ref), or every completed thumbnail re-runs the effect in all mounted thumbnails.
|
||||||
|
|
||||||
### URL state (`src/App.tsx`)
|
### Profile tabs (`src/lib/post-tabs.ts`)
|
||||||
|
|
||||||
App state syncs to `?a=` / `?t=` / `?p=`. Two rules, both learned from real bugs:
|
The grid holds **everything**, reels included, and the Reels tab is a *filtered view* of that same set — as on Instagram. Only the Reels tab filters. The tabs were mutually exclusive until v1.7.0, which hid a lot: 1100 of `groupfandom`'s 3813 posts and 533 of `for.member`'s 1225 never appeared in the grid at all.
|
||||||
|
|
||||||
- The initial query string is captured into a ref on first render; the URL is rewritten from state as soon as anything loads, so reading `window.location` later sees the rewrite, not the user's link.
|
Deciding *what is a reel* has no good answer for most archives. Instagram's own marker is `product_type` on the post's GraphQL node (`clips` = reel, `feed` = ordinary feed video, `igtv`, `story`) — `__typename` is `GraphVideo` for all three, and aspect ratio does not separate them either. But:
|
||||||
|
|
||||||
|
- Only Instaloader archives carry that metadata, and only newer captures. A survey of `hazelofficial` found `product_type` on 1101 of 5919 sidecars, and just **2** posts marked `clips`.
|
||||||
|
- JDownloader archives carry none at all — media plus a `.txt` holding the bare caption.
|
||||||
|
|
||||||
|
So the viewer believes a `- reels` sidecar directory when one exists, and otherwise falls back to treating a lone video as a reel. **The fallback is a guess**: it cannot tell a reel from a feed video or an old IGTV upload, and it misses videos inside carousels.
|
||||||
|
|
||||||
|
`dedupePostCopies` exists because the JDownloader flow crawls the profile URL and the `/reels/` URL separately (the profile page misses some reels), so the two overlap and a reel can land on disk twice. Those become two posts with distinct directory-scoped ids, which the grid would otherwise render side by side. It dedupes by shortcode, preferring the reels-source copy. It is only safe over `allPosts` — stories and highlights are excluded there, and a shortcode may legitimately appear in both a profile and a highlight.
|
||||||
|
|
||||||
|
### URL state (`src/App.tsx`, `src/lib/routing.ts`)
|
||||||
|
|
||||||
|
Paths mirror Instagram: `/<archive>/`, `/<archive>/reels/`, `/<archive>/p/<shortcode>/`. The old `?a=&t=&p=` form is still parsed for existing links but never written. Reserved prefixes (`api`, `archives`, `assets`…) can't be mistaken for a profile name.
|
||||||
|
|
||||||
|
A post URL carries no tab, as on Instagram — the tab is re-derived from the post's `source`, so a reel link lands on the Reels tab and pages through reels. Sidecar posts keep directory-scoped ids internally but expose only the shortcode.
|
||||||
|
|
||||||
|
Three rules, all learned from real bugs:
|
||||||
|
|
||||||
|
- The initial route is captured into a ref on first render; the URL is rewritten from state as soon as anything loads, so reading `window.location` later sees the rewrite, not the user's link.
|
||||||
- URL writing is gated on `hasInitialLoaded`, otherwise it erases the deep link before the loader consumes it.
|
- URL writing is gated on `hasInitialLoaded`, otherwise it erases the deep link before the loader consumes it.
|
||||||
|
- Deep-link resolution waits on the archive fetch having *settled* (`archivesFetched`), not on `isServerMode`, which is still false while the request is in flight.
|
||||||
|
|
||||||
Deep-link resolution waits on the archive fetch having *settled*, not on `isServerMode` (which is still false while in flight).
|
### Mobile feed (`src/components/PostFeed.tsx`)
|
||||||
|
|
||||||
|
Below `md`, opening a post renders a scrolling feed page rather than the modal (`useIsMobile` decides). Only a window of posts is mounted; it grows both ways, and prepending corrects `scrollTop` in a `useLayoutEffect` so content doesn't jump. Only the post crossing the viewport centre plays its video and drives the URL. Desktop keeps `PostModal`; both share `MediaCarousel`.
|
||||||
|
|
||||||
### Backend (`server.ts`)
|
### Backend (`server.ts`)
|
||||||
|
|
||||||
Serves `/api/archives`, `/api/archives/:name/files`, static `/archives`, and the built SPA. Notes:
|
Serves `/api/archives`, `/api/archives/:name/files`, static `/archives`, and the built SPA. Notes:
|
||||||
|
|
||||||
- Express decodes route params **after** segment matching, so `..%2f` reaches the handler as `../`. All user-supplied archive names go through `resolveArchivePath`.
|
- Express decodes route params **after** segment matching, so `..%2f` reaches the handler as `../`. All user-supplied archive names go through `resolveArchivePath`.
|
||||||
- Sets CSP and related security headers. The CSP allows `blob:`/`data:` for media and `unsafe-inline` styles (the animation library sets inline styles); scripts stay same-origin only.
|
|
||||||
- `os.userInfo()` throws for a UID with no `/etc/passwd` entry, which is what `--user 1234:1234` produces — use `describeUser()`.
|
- `os.userInfo()` throws for a UID with no `/etc/passwd` entry, which is what `--user 1234:1234` produces — use `describeUser()`.
|
||||||
|
|
||||||
|
### CSP: do not tighten `script-src` or `connect-src` without testing xz
|
||||||
|
|
||||||
|
The xz decompressor for Instaloader `.json.xz` sidecars is **WebAssembly**, embedded as a `data:` URL the library fetches at startup. The policy must keep:
|
||||||
|
|
||||||
|
```
|
||||||
|
script-src 'self' 'wasm-unsafe-eval' // compile wasm, without allowing eval() of JS
|
||||||
|
connect-src 'self' data: // fetch the embedded module
|
||||||
|
```
|
||||||
|
|
||||||
|
Removing either breaks decoding with a bare `TypeError: Failed to fetch` **and no stack** — it surfaces through `new Response(stream).json()`, so it reads like a network fault rather than a policy block. The visible symptom is not an error page: archives silently lose captions, story flags and all profile metadata (follower counts, bio, name). This shipped broken for several releases.
|
||||||
|
|
||||||
|
To check quickly, run in the page console:
|
||||||
|
|
||||||
|
```js
|
||||||
|
await fetch('data:application/wasm;base64,AGFzbQEAAAA=') // connect-src
|
||||||
|
await WebAssembly.instantiate(Uint8Array.of(0,97,115,109,1,0,0,0)) // script-src
|
||||||
|
```
|
||||||
|
|
||||||
|
**The Vite dev server does not send these headers**, so anything CSP-related is invisible in `npm run dev`. Verify security-header and PWA behaviour by building and serving `dist/` through `server.js`, not against the dev server.
|
||||||
|
|
||||||
|
### PWA: server-only changes do not reach installed clients
|
||||||
|
|
||||||
|
The service worker precaches `index.html` **together with its response headers**. A change that touches only the server (a CSP fix, a new header) leaves the client build byte-identical, so the precache manifest and `sw.js` are unchanged, the worker never updates, and installed clients keep replaying the old shell with the old headers — indefinitely.
|
||||||
|
|
||||||
|
`vite.config.ts` therefore compiles the package version into the client via `define: { __APP_VERSION__ }`, and `App.tsx` renders it in the footer. That is **load-bearing**: it makes every release change the bundle hash → `index.html` → its precache revision → `sw.js`, which is what browsers byte-compare to decide whether to update. Don't remove it as dead weight.
|
||||||
|
|
||||||
|
To recover a client stuck on an old shell: unregister the service worker, delete its caches, reload.
|
||||||
|
|
||||||
### Deployment
|
### Deployment
|
||||||
|
|
||||||
|
Live archives live in `<share>/Instagram-archive/archives/` — one directory per profile plus sidecars. Directories that are not Instagram profiles (tool output, exports from other services) sit *outside* that folder so they never reach the viewer.
|
||||||
|
|
||||||
Container runs as non-root. The image defaults to `node`, but the archive share must be *listable* by that UID — a mode-711 share owned by another account needs `user: "<uid>:<gid>"` in compose. Mount a volume at `/cache` so the index survives restarts.
|
Container runs as non-root. The image defaults to `node`, but the archive share must be *listable* by that UID — a mode-711 share owned by another account needs `user: "<uid>:<gid>"` in compose. Mount a volume at `/cache` so the index survives restarts.
|
||||||
|
|
||||||
### PWA / build quirks
|
### PWA / build quirks
|
||||||
|
|||||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
|||||||
{
|
{
|
||||||
"name": "instaarchive-viewer",
|
"name": "instaarchive-viewer",
|
||||||
"version": "1.5.0",
|
"version": "1.7.0",
|
||||||
"lockfileVersion": 3,
|
"lockfileVersion": 3,
|
||||||
"requires": true,
|
"requires": true,
|
||||||
"packages": {
|
"packages": {
|
||||||
"": {
|
"": {
|
||||||
"name": "instaarchive-viewer",
|
"name": "instaarchive-viewer",
|
||||||
"version": "1.5.0",
|
"version": "1.7.0",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@tailwindcss/vite": "^4.1.14",
|
"@tailwindcss/vite": "^4.1.14",
|
||||||
"@vitejs/plugin-react": "^5.0.4",
|
"@vitejs/plugin-react": "^5.0.4",
|
||||||
|
|||||||
+3
-2
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "instaarchive-viewer",
|
"name": "instaarchive-viewer",
|
||||||
"private": true,
|
"private": true,
|
||||||
"version": "1.5.0",
|
"version": "1.7.0",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"dev": "vite --port=3000 --host=0.0.0.0",
|
"dev": "vite --port=3000 --host=0.0.0.0",
|
||||||
@@ -12,7 +12,8 @@
|
|||||||
"clean": "rm -rf dist",
|
"clean": "rm -rf dist",
|
||||||
"lint": "tsc --noEmit",
|
"lint": "tsc --noEmit",
|
||||||
"test": "vitest run",
|
"test": "vitest run",
|
||||||
"test:watch": "vitest"
|
"test:watch": "vitest",
|
||||||
|
"jd2": "tsx scripts/jd2-sync.ts"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@tailwindcss/vite": "^4.1.14",
|
"@tailwindcss/vite": "^4.1.14",
|
||||||
|
|||||||
@@ -80,10 +80,16 @@ app.use((req, res, next) => {
|
|||||||
"default-src 'self'",
|
"default-src 'self'",
|
||||||
"img-src 'self' blob: data:",
|
"img-src 'self' blob: data:",
|
||||||
"media-src 'self' blob: data:",
|
"media-src 'self' blob: data:",
|
||||||
"script-src 'self'",
|
// 'wasm-unsafe-eval' permits WebAssembly compilation without allowing
|
||||||
|
// eval() of JavaScript. The xz decompressor used for Instaloader's
|
||||||
|
// .json.xz sidecars is WebAssembly, embedded as a data: URL it fetches at
|
||||||
|
// startup — so connect-src must allow data: too. Without both, decoding
|
||||||
|
// fails with a bare "TypeError: Failed to fetch" and every archive silently
|
||||||
|
// loses its captions, story flags and profile metadata.
|
||||||
|
"script-src 'self' 'wasm-unsafe-eval'",
|
||||||
"style-src 'self' 'unsafe-inline'",
|
"style-src 'self' 'unsafe-inline'",
|
||||||
"font-src 'self'",
|
"font-src 'self'",
|
||||||
"connect-src 'self'",
|
"connect-src 'self' data:",
|
||||||
"worker-src 'self' blob:",
|
"worker-src 'self' blob:",
|
||||||
"frame-ancestors 'self'",
|
"frame-ancestors 'self'",
|
||||||
"object-src 'none'",
|
"object-src 'none'",
|
||||||
|
|||||||
+50
-23
@@ -17,8 +17,9 @@ import {
|
|||||||
import { motion, AnimatePresence } from 'motion/react';
|
import { motion, AnimatePresence } from 'motion/react';
|
||||||
|
|
||||||
import { cn } from './lib/utils';
|
import { cn } from './lib/utils';
|
||||||
import { PRESS } from './lib/motion';
|
import { PRESS, prefersReducedMotion } from './lib/motion';
|
||||||
import { buildPath, findPostBySlug, parseRoute, postSlug, tabForSource } from './lib/routing';
|
import { buildPath, findPostBySlug, parseRoute, postSlug, tabForSource } from './lib/routing';
|
||||||
|
import { postsForTab } from './lib/post-tabs';
|
||||||
import { LocalArchiveFile, RemoteArchiveFile } from './lib/archive-files';
|
import { LocalArchiveFile, RemoteArchiveFile } from './lib/archive-files';
|
||||||
import {
|
import {
|
||||||
deleteCachedArchive,
|
deleteCachedArchive,
|
||||||
@@ -111,6 +112,30 @@ export default function App() {
|
|||||||
|
|
||||||
const [lastLoadedScanningImage, setLastLoadedScanningImage] = useState<string | null>(null);
|
const [lastLoadedScanningImage, setLastLoadedScanningImage] = useState<string | null>(null);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Blurred backdrops behind the scanning UI, newest last.
|
||||||
|
*
|
||||||
|
* Each new image is stacked *over* the previous one and fades in; the one
|
||||||
|
* underneath stays fully opaque until it's covered. Cross-fading by swapping
|
||||||
|
* a single element left the pale backdrop showing through mid-transition,
|
||||||
|
* which read as a white flash between every image.
|
||||||
|
*/
|
||||||
|
const [scanBackdrops, setScanBackdrops] = useState<string[]>([]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!lastLoadedScanningImage) return;
|
||||||
|
setScanBackdrops(prev =>
|
||||||
|
prev[prev.length - 1] === lastLoadedScanningImage
|
||||||
|
? prev
|
||||||
|
: [...prev, lastLoadedScanningImage].slice(-3),
|
||||||
|
);
|
||||||
|
}, [lastLoadedScanningImage]);
|
||||||
|
|
||||||
|
// Don't carry one archive's backdrops into the next scan.
|
||||||
|
useEffect(() => {
|
||||||
|
if (!isScanning) { setScanBackdrops([]); setLastLoadedScanningImage(null); }
|
||||||
|
}, [isScanning]);
|
||||||
|
|
||||||
const {
|
const {
|
||||||
username,
|
username,
|
||||||
fullName,
|
fullName,
|
||||||
@@ -148,20 +173,11 @@ export default function App() {
|
|||||||
const clearCache = async (name: string) => { await deleteCachedArchive(name); await refreshCachedArchives(); };
|
const clearCache = async (name: string) => { await deleteCachedArchive(name); await refreshCachedArchives(); };
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Archives with a `- reels` sidecar directory say outright which posts are
|
* The grid shows everything, reels included, and the Reels tab is a filtered
|
||||||
* reels; only fall back to the "lone video" heuristic for archives that have
|
* view of the same set — see src/lib/post-tabs.ts for the reel test and for
|
||||||
* no such directory.
|
* why a reel can arrive on disk twice.
|
||||||
*/
|
*/
|
||||||
const hasReelSource = useMemo(() => allPosts.some(p => p.source === 'reels'), [allPosts]);
|
const filteredPosts = useMemo(() => postsForTab(allPosts, activeTab), [allPosts, activeTab]);
|
||||||
const isReel = useCallback((p: Post) => (
|
|
||||||
hasReelSource ? p.source === 'reels' : p.media.length === 1 && p.media[0].type === 'video'
|
|
||||||
), [hasReelSource]);
|
|
||||||
|
|
||||||
const filteredPosts = useMemo(() => {
|
|
||||||
if (activeTab === 'reels') return allPosts.filter(isReel);
|
|
||||||
if (activeTab === 'posts') return allPosts.filter(p => !isReel(p));
|
|
||||||
return [];
|
|
||||||
}, [allPosts, activeTab, isReel]);
|
|
||||||
|
|
||||||
/** Story highlights, grouped into the circles shown under the bio. */
|
/** Story highlights, grouped into the circles shown under the bio. */
|
||||||
const highlightGroups = useMemo(() => {
|
const highlightGroups = useMemo(() => {
|
||||||
@@ -460,17 +476,28 @@ export default function App() {
|
|||||||
onLoad={() => setLastLoadedScanningImage(currentScanningImage)}
|
onLoad={() => setLastLoadedScanningImage(currentScanningImage)}
|
||||||
/>
|
/>
|
||||||
)}
|
)}
|
||||||
<div className="absolute inset-0 z-0">
|
{/*
|
||||||
<AnimatePresence initial={false}>
|
The 0.4 lives on the group, not the images: two layers overlap
|
||||||
<motion.img
|
during a cross-fade, and fading them individually would darken the
|
||||||
key={lastLoadedScanningImage}
|
backdrop as they cross. Inside the group each layer goes to full
|
||||||
src={lastLoadedScanningImage || undefined}
|
opacity, so the stack is always completely covered.
|
||||||
|
*/}
|
||||||
|
<div className="absolute inset-0 z-0 opacity-40">
|
||||||
|
{scanBackdrops.map(src => (
|
||||||
|
<motion.img
|
||||||
|
key={src}
|
||||||
|
src={src}
|
||||||
initial={{ opacity: 0 }}
|
initial={{ opacity: 0 }}
|
||||||
animate={{ opacity: 0.4 }}
|
animate={{ opacity: 1 }}
|
||||||
transition={{ duration: 1.5 }}
|
transition={prefersReducedMotion() ? { duration: 0 } : { duration: 0.9, ease: 'easeInOut' }}
|
||||||
|
onAnimationComplete={() => setScanBackdrops(prev => {
|
||||||
|
// Once this layer is opaque it hides everything below it.
|
||||||
|
const i = prev.indexOf(src);
|
||||||
|
return i > 0 ? prev.slice(i) : prev;
|
||||||
|
})}
|
||||||
className="absolute inset-0 w-full h-full object-cover blur-[60px] scale-110"
|
className="absolute inset-0 w-full h-full object-cover blur-[60px] scale-110"
|
||||||
/>
|
/>
|
||||||
</AnimatePresence>
|
))}
|
||||||
</div>
|
</div>
|
||||||
<div className="absolute inset-0 bg-white/40 z-1" />
|
<div className="absolute inset-0 bg-white/40 z-1" />
|
||||||
<div className="relative z-10 w-full max-w-4xl px-4 flex flex-col items-center gap-8 text-black">
|
<div className="relative z-10 w-full max-w-4xl px-4 flex flex-col items-center gap-8 text-black">
|
||||||
@@ -597,7 +624,7 @@ export default function App() {
|
|||||||
{!isScanning && (
|
{!isScanning && (
|
||||||
<footer className="max-w-5xl mx-auto px-4 py-12 text-center text-xs text-gray-400 space-y-4 text-black">
|
<footer className="max-w-5xl mx-auto px-4 py-12 text-center text-xs text-gray-400 space-y-4 text-black">
|
||||||
<div className="flex flex-wrap justify-center gap-x-4 gap-y-2 uppercase tracking-tight text-black"><span>Meta</span><span>About</span><span>Blog</span><span>Jobs</span><span>Help</span><span>API</span><span>Privacy</span><span>Terms</span><span>Locations</span><span>Instagram Lite</span><span>Threads</span><span>Contact Uploading & Non-Users</span><span>Meta Verified</span></div>
|
<div className="flex flex-wrap justify-center gap-x-4 gap-y-2 uppercase tracking-tight text-black"><span>Meta</span><span>About</span><span>Blog</span><span>Jobs</span><span>Help</span><span>API</span><span>Privacy</span><span>Terms</span><span>Locations</span><span>Instagram Lite</span><span>Threads</span><span>Contact Uploading & Non-Users</span><span>Meta Verified</span></div>
|
||||||
<div className="text-black/40 text-black">© 2026 InstaArchive Viewer</div>
|
<div className="text-black/40 text-black">© 2026 InstaArchive Viewer · v{__APP_VERSION__}</div>
|
||||||
</footer>
|
</footer>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -82,11 +82,14 @@ export const PostModal: React.FC<PostModalProps> = ({
|
|||||||
|
|
||||||
useEffect(() => setCurrentIndex(0), [post.id]);
|
useEffect(() => setCurrentIndex(0), [post.id]);
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
|
// Arrows page within the carousel — the thing the arrows visually point at.
|
||||||
|
// Moving between posts stays on the side buttons, with , and . as keyboard
|
||||||
|
// equivalents.
|
||||||
const handleKeyDown = (e: KeyboardEvent) => {
|
const handleKeyDown = (e: KeyboardEvent) => {
|
||||||
if (e.key === 'ArrowRight') goToPost(1, 'x');
|
if (e.key === 'ArrowRight') paginate(1);
|
||||||
else if (e.key === 'ArrowLeft') goToPost(-1, 'x');
|
else if (e.key === 'ArrowLeft') paginate(-1);
|
||||||
else if (e.key === '.') paginate(1);
|
else if (e.key === '.') goToPost(1, 'x');
|
||||||
else if (e.key === ',') paginate(-1);
|
else if (e.key === ',') goToPost(-1, 'x');
|
||||||
else if (e.key === 'Escape') onClose();
|
else if (e.key === 'Escape') onClose();
|
||||||
};
|
};
|
||||||
window.addEventListener('keydown', handleKeyDown);
|
window.addEventListener('keydown', handleKeyDown);
|
||||||
|
|||||||
@@ -107,11 +107,23 @@ export const useArchiveScanner = (
|
|||||||
/** Stable identity for a media file, used to rehydrate URLs after a reload. */
|
/** Stable identity for a media file, used to rehydrate URLs after a reload. */
|
||||||
const mediaPath = (file: ArchiveFile) => file.webkitRelativePath || file.name;
|
const mediaPath = (file: ArchiveFile) => file.webkitRelativePath || file.name;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Decompress an `.xz` metadata sidecar.
|
||||||
|
*
|
||||||
|
* Read fully into memory first rather than handing the live HTTP body to
|
||||||
|
* the decompressor. These sidecars are a few KB, so buffering costs
|
||||||
|
* nothing, and streaming was actively harmful: the decompressor stops
|
||||||
|
* reading at the end of the xz member, leaving the response body neither
|
||||||
|
* drained nor cancelled. Across a couple of hundred sidecars that exhausts
|
||||||
|
* the connection pool and every later fetch fails with "Failed to fetch" —
|
||||||
|
* which silently cost Instaloader archives their captions, story flags and
|
||||||
|
* profile metadata, since all of it lives in these files.
|
||||||
|
*/
|
||||||
const parseXZFile = async (file: ArchiveFile) => {
|
const parseXZFile = async (file: ArchiveFile) => {
|
||||||
try {
|
try {
|
||||||
const stream = new XzReadableStream(file.stream());
|
const compressed = await file.arrayBuffer();
|
||||||
const response = new Response(stream);
|
const stream = new XzReadableStream(new Blob([compressed]).stream());
|
||||||
return await response.json();
|
return await new Response(stream).json();
|
||||||
} catch (e) { console.error(`[Scanner] XZ Parse Error:`, file.name, e); return null; }
|
} catch (e) { console.error(`[Scanner] XZ Parse Error:`, file.name, e); return null; }
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@@ -14,7 +14,6 @@ export class LocalArchiveFile implements ArchiveFile {
|
|||||||
get size() { return this.file.size; }
|
get size() { return this.file.size; }
|
||||||
text() { return this.file.text(); }
|
text() { return this.file.text(); }
|
||||||
arrayBuffer() { return this.file.arrayBuffer(); }
|
arrayBuffer() { return this.file.arrayBuffer(); }
|
||||||
stream() { return this.file.stream(); }
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* A blob: URL backed directly by the on-disk File.
|
* A blob: URL backed directly by the on-disk File.
|
||||||
@@ -55,15 +54,6 @@ export class RemoteArchiveFile implements ArchiveFile {
|
|||||||
const res = await fetch(this.url);
|
const res = await fetch(this.url);
|
||||||
return res.arrayBuffer();
|
return res.arrayBuffer();
|
||||||
}
|
}
|
||||||
stream() {
|
|
||||||
const transform = new TransformStream();
|
|
||||||
fetch(this.url).then(res => {
|
|
||||||
if (res.body) res.body.pipeTo(transform.writable);
|
|
||||||
else transform.writable.getWriter().close();
|
|
||||||
});
|
|
||||||
return transform.readable;
|
|
||||||
}
|
|
||||||
|
|
||||||
createObjectUrl() {
|
createObjectUrl() {
|
||||||
return this.url;
|
return this.url;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,34 @@
|
|||||||
|
import { describe, expect, it } from 'vitest';
|
||||||
|
import { isSystemDirectory } from './archive-index';
|
||||||
|
|
||||||
|
describe('isSystemDirectory', () => {
|
||||||
|
it.each([
|
||||||
|
['@eaDir', 'Synology thumbnail/index metadata, written inside every folder'],
|
||||||
|
['@tmp', 'Synology scratch'],
|
||||||
|
['.sync', 'Resilio state'],
|
||||||
|
['.DS_Store', 'macOS'],
|
||||||
|
['#recycle', 'Synology deletions'],
|
||||||
|
['#snapshot', 'Synology snapshots'],
|
||||||
|
])('skips %s (%s)', name => {
|
||||||
|
expect(isSystemDirectory(name)).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
'4utumn07',
|
||||||
|
'4utumn07 - reels',
|
||||||
|
'story - dawn_petal',
|
||||||
|
'story highlights - official_band - A.B.C',
|
||||||
|
'story highlights - theoldlyricmuseinsta - 💙1999-2005 era',
|
||||||
|
'Heejin_Bubble heejinmedia',
|
||||||
|
'gallery-dl',
|
||||||
|
'posts',
|
||||||
|
])('keeps %s', name => {
|
||||||
|
expect(isSystemDirectory(name)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('does not treat a leading underscore as a system directory', () => {
|
||||||
|
// `_gemini-plans` is filtered separately at the archive root only; nothing
|
||||||
|
// below the root should be excluded just for starting with an underscore.
|
||||||
|
expect(isSystemDirectory('_gemini-plans')).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -33,6 +33,21 @@ interface DirIndex {
|
|||||||
const MEDIA_RE = /\.(jpg|jpeg|png|webp|gif|bmp|tiff|mp4|webm|ogv|mov)$/i;
|
const MEDIA_RE = /\.(jpg|jpeg|png|webp|gif|bmp|tiff|mp4|webm|ogv|mov)$/i;
|
||||||
const STAT_CONCURRENCY = 16;
|
const STAT_CONCURRENCY = 16;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Directories the walk must never descend into.
|
||||||
|
*
|
||||||
|
* NAS filesystems scatter sidecar metadata *inside* every folder, not just at
|
||||||
|
* the share root: Synology writes `@eaDir` (thumbnails and indexing data),
|
||||||
|
* `#recycle` holds deletions, and `.sync` is Resilio's state. Indexing those
|
||||||
|
* would count NAS thumbnails as archive media and spend a stat on each one —
|
||||||
|
* measured on a real share, `@eaDir` accounted for 12,516 of 123,023 files.
|
||||||
|
*
|
||||||
|
* The archive root is already filtered by prefix; this is the same rule applied
|
||||||
|
* at every level below it.
|
||||||
|
*/
|
||||||
|
export const isSystemDirectory = (name: string): boolean =>
|
||||||
|
name.startsWith('@') || name.startsWith('.') || name === '#recycle' || name === '#snapshot';
|
||||||
|
|
||||||
export class ArchiveIndex {
|
export class ArchiveIndex {
|
||||||
private dirs = new Map<string, DirIndex>();
|
private dirs = new Map<string, DirIndex>();
|
||||||
private inFlight = new Map<string, Promise<DirIndex>>();
|
private inFlight = new Map<string, Promise<DirIndex>>();
|
||||||
@@ -43,7 +58,7 @@ export class ArchiveIndex {
|
|||||||
/** Visible (non-system) directories at the archive root. */
|
/** Visible (non-system) directories at the archive root. */
|
||||||
private listRootDirs(): string[] {
|
private listRootDirs(): string[] {
|
||||||
return fs.readdirSync(this.archivesDir, { withFileTypes: true })
|
return fs.readdirSync(this.archivesDir, { withFileTypes: true })
|
||||||
.filter(e => e.isDirectory() && !/^[.@_]/.test(e.name))
|
.filter(e => e.isDirectory() && !isSystemDirectory(e.name) && !e.name.startsWith('_'))
|
||||||
.map(e => e.name);
|
.map(e => e.name);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -69,6 +84,7 @@ export class ArchiveIndex {
|
|||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
|
if (isSystemDirectory(entry.name)) continue;
|
||||||
const rel = base ? `${base}/${entry.name}` : entry.name;
|
const rel = base ? `${base}/${entry.name}` : entry.name;
|
||||||
if (entry.isDirectory()) out = out.concat(this.walk(path.join(absDir, entry.name), rel));
|
if (entry.isDirectory()) out = out.concat(this.walk(path.join(absDir, entry.name), rel));
|
||||||
else if (entry.isFile()) out.push(rel);
|
else if (entry.isFile()) out.push(rel);
|
||||||
|
|||||||
@@ -0,0 +1,100 @@
|
|||||||
|
import { describe, expect, it } from 'vitest';
|
||||||
|
import { dedupePostCopies, hasReelSource, makeIsReel, postsForTab } from './post-tabs';
|
||||||
|
import { MediaFile, Post } from '../types';
|
||||||
|
|
||||||
|
const media = (type: MediaFile['type'], index = 1): MediaFile => ({
|
||||||
|
name: `f${index}.${type === 'video' ? 'mp4' : 'jpg'}`,
|
||||||
|
path: `d/f${index}`, url: '', type, index,
|
||||||
|
});
|
||||||
|
|
||||||
|
const post = (id: string, opts: Partial<Post> = {}): Post => ({
|
||||||
|
id, date: '2024-01-01', username: 'u', caption: '', media: [media('image')], thumbnail: '', ...opts,
|
||||||
|
});
|
||||||
|
|
||||||
|
const video = (id: string, opts: Partial<Post> = {}) => post(id, { media: [media('video')], ...opts });
|
||||||
|
const carousel = (id: string, opts: Partial<Post> = {}) =>
|
||||||
|
post(id, { media: [media('image', 1), media('video', 2)], ...opts });
|
||||||
|
|
||||||
|
describe('hasReelSource', () => {
|
||||||
|
it('is false for an archive with no reels directory', () => {
|
||||||
|
expect(hasReelSource([post('A'), video('B')])).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('is true once any post came from a reels directory', () => {
|
||||||
|
expect(hasReelSource([post('A'), video('u - reels/B', { source: 'reels' })])).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('makeIsReel', () => {
|
||||||
|
it('believes the reels directory when there is one', () => {
|
||||||
|
const posts = [video('A'), video('u - reels/B', { source: 'reels' })];
|
||||||
|
const isReel = makeIsReel(posts);
|
||||||
|
// A is a lone video too, but the archive states which posts are reels.
|
||||||
|
expect(isReel(posts[0])).toBe(false);
|
||||||
|
expect(isReel(posts[1])).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('falls back to the lone-video heuristic without one', () => {
|
||||||
|
const posts = [post('A'), video('B'), carousel('C')];
|
||||||
|
const isReel = makeIsReel(posts);
|
||||||
|
expect(posts.map(isReel)).toEqual([false, true, false]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('dedupePostCopies', () => {
|
||||||
|
it('leaves distinct posts alone', () => {
|
||||||
|
const posts = [post('A'), video('B')];
|
||||||
|
expect(dedupePostCopies(posts).map(p => p.id)).toEqual(['A', 'B']);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('collapses a reel fetched into both the profile and the reels directory', () => {
|
||||||
|
const posts = [video('B'), video('u - reels/B', { source: 'reels' })];
|
||||||
|
const deduped = dedupePostCopies(posts);
|
||||||
|
expect(deduped).toHaveLength(1);
|
||||||
|
// The reels copy wins, so the survivor is still recognised as a reel.
|
||||||
|
expect(deduped[0].source).toBe('reels');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('picks the reels copy regardless of scan order', () => {
|
||||||
|
const profileCopy = video('B');
|
||||||
|
const reelCopy = video('u - reels/B', { source: 'reels' });
|
||||||
|
expect(dedupePostCopies([profileCopy, reelCopy])[0].source).toBe('reels');
|
||||||
|
expect(dedupePostCopies([reelCopy, profileCopy])[0].source).toBe('reels');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('keeps the position of the first copy seen', () => {
|
||||||
|
const posts = [post('A'), video('B'), post('C'), video('u - reels/B', { source: 'reels' })];
|
||||||
|
expect(dedupePostCopies(posts).map(p => p.id.split('/').pop())).toEqual(['A', 'B', 'C']);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('postsForTab', () => {
|
||||||
|
it('shows reels in the profile grid, as Instagram does', () => {
|
||||||
|
const posts = [post('A'), video('u - reels/B', { source: 'reels' })];
|
||||||
|
expect(postsForTab(posts, 'posts').map(p => p.id)).toEqual(['A', 'u - reels/B']);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('shows the same reel in both tabs', () => {
|
||||||
|
const posts = [post('A'), video('u - reels/B', { source: 'reels' })];
|
||||||
|
const inGrid = postsForTab(posts, 'posts').map(p => p.id);
|
||||||
|
const inReels = postsForTab(posts, 'reels').map(p => p.id);
|
||||||
|
expect(inReels).toEqual(['u - reels/B']);
|
||||||
|
expect(inGrid).toContain('u - reels/B');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('shows a duplicated reel once in the grid, not twice', () => {
|
||||||
|
const posts = [post('A'), video('B'), video('u - reels/B', { source: 'reels' })];
|
||||||
|
expect(postsForTab(posts, 'posts')).toHaveLength(2);
|
||||||
|
expect(postsForTab(posts, 'reels')).toHaveLength(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('treats lone videos as reels for archives with no reels directory', () => {
|
||||||
|
const posts = [post('A'), video('B'), carousel('C')];
|
||||||
|
expect(postsForTab(posts, 'posts').map(p => p.id)).toEqual(['A', 'B', 'C']);
|
||||||
|
expect(postsForTab(posts, 'reels').map(p => p.id)).toEqual(['B']);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('has nothing saved', () => {
|
||||||
|
expect(postsForTab([post('A')], 'saved')).toEqual([]);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
import { Post, SourceKind } from '../types';
|
||||||
|
import { Tab } from './routing';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Which posts each profile tab shows.
|
||||||
|
*
|
||||||
|
* Instagram's profile grid holds everything the account posted — photos,
|
||||||
|
* carousels and reels alike — and the Reels tab is a *filtered view* of that
|
||||||
|
* same set rather than a separate one. So a reel belongs in both tabs, and
|
||||||
|
* only the Reels tab does any filtering.
|
||||||
|
*
|
||||||
|
* Kept pure and separate from App.tsx so the reel heuristic and the
|
||||||
|
* duplicate-copy rules can be tested directly.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The shortcode shared by every copy of a post, regardless of which source
|
||||||
|
* directory it came from. Sidecar ids are directory-scoped
|
||||||
|
* (`4utumn07 - reels/Cq8LrxSJAJE`); the trailing segment is the shortcode.
|
||||||
|
*/
|
||||||
|
const shortcode = (post: Post): string => post.id.split('/').pop() ?? post.id;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* True when the archive has a `- reels` sidecar directory, i.e. it states
|
||||||
|
* outright which posts are reels.
|
||||||
|
*/
|
||||||
|
export const hasReelSource = (posts: Post[]): boolean => posts.some(p => p.source === 'reels');
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build the reel test for an archive.
|
||||||
|
*
|
||||||
|
* Instagram's own marker is `product_type: "clips"` on the post's GraphQL
|
||||||
|
* node, but only Instaloader archives carry that metadata, and only on newer
|
||||||
|
* captures — JDownloader grabs are media plus a caption `.txt` and nothing
|
||||||
|
* else (see docs/jdownloader.md). So:
|
||||||
|
*
|
||||||
|
* - archives with a `- reels` directory are believed outright;
|
||||||
|
* - everything else falls back to treating a lone video as a reel.
|
||||||
|
*
|
||||||
|
* The fallback is a guess: it cannot tell a reel from an ordinary feed video
|
||||||
|
* or an old IGTV upload, all three of which are plain `GraphVideo` nodes
|
||||||
|
* distinguished only by `product_type`.
|
||||||
|
*/
|
||||||
|
export const makeIsReel = (posts: Post[]): ((post: Post) => boolean) => (
|
||||||
|
hasReelSource(posts)
|
||||||
|
? (post: Post) => post.source === 'reels'
|
||||||
|
: (post: Post) => post.media.length === 1 && post.media[0]?.type === 'video'
|
||||||
|
);
|
||||||
|
|
||||||
|
/** Preference order when the same post was fetched into more than one directory. */
|
||||||
|
const SOURCE_RANK: Record<SourceKind, number> = { reels: 0, posts: 1, stories: 2, highlight: 3 };
|
||||||
|
|
||||||
|
const rankOf = (post: Post): number => SOURCE_RANK[post.source ?? 'posts'];
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Collapse copies of one post that were fetched into more than one directory.
|
||||||
|
*
|
||||||
|
* The JDownloader flow crawls a profile URL and its `/reels/` URL separately
|
||||||
|
* because the profile page misses some reels — so the two overlap, and a reel
|
||||||
|
* present in both lands on disk twice. Those become two posts with distinct
|
||||||
|
* directory-scoped ids, which the grid would happily render side by side.
|
||||||
|
*
|
||||||
|
* The reels-source copy wins, so the surviving post still reports
|
||||||
|
* `source: 'reels'` and both the Reels tab and `tabForSource` recognise it.
|
||||||
|
*
|
||||||
|
* Only safe because callers pass the grid's posts, which exclude stories and
|
||||||
|
* highlights — a shortcode may legitimately appear in both the profile and a
|
||||||
|
* highlight, and those must stay distinct.
|
||||||
|
*/
|
||||||
|
export const dedupePostCopies = (posts: Post[]): Post[] => {
|
||||||
|
const winners = new Map<string, Post>();
|
||||||
|
|
||||||
|
for (const post of posts) {
|
||||||
|
const code = shortcode(post);
|
||||||
|
const existing = winners.get(code);
|
||||||
|
if (!existing || rankOf(post) < rankOf(existing)) winners.set(code, post);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Preserve input order, keyed on the winner so ordering does not depend on
|
||||||
|
// which copy happened to be scanned first.
|
||||||
|
const emitted = new Set<string>();
|
||||||
|
const result: Post[] = [];
|
||||||
|
for (const post of posts) {
|
||||||
|
const code = shortcode(post);
|
||||||
|
if (emitted.has(code)) continue;
|
||||||
|
emitted.add(code);
|
||||||
|
result.push(winners.get(code)!);
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The posts a tab displays.
|
||||||
|
*
|
||||||
|
* `posts` must already exclude stories and highlights (App passes `allPosts`).
|
||||||
|
*/
|
||||||
|
export const postsForTab = (posts: Post[], tab: Tab): Post[] => {
|
||||||
|
if (tab === 'saved') return [];
|
||||||
|
|
||||||
|
const unique = dedupePostCopies(posts);
|
||||||
|
if (tab === 'posts') return unique;
|
||||||
|
return unique.filter(makeIsReel(posts));
|
||||||
|
};
|
||||||
+1
-1
@@ -14,7 +14,7 @@ const updateSW = registerSW({
|
|||||||
setInterval(() => {
|
setInterval(() => {
|
||||||
r.update();
|
r.update();
|
||||||
}, 60 * 60 * 1000);
|
}, 60 * 60 * 1000);
|
||||||
console.log('[PWA] Service Worker registered and update interval set.');
|
console.log(`[PWA] v${__APP_VERSION__} registered; hourly update checks enabled.`);
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
onNeedRefresh() {
|
onNeedRefresh() {
|
||||||
|
|||||||
Vendored
+9
@@ -0,0 +1,9 @@
|
|||||||
|
/**
|
||||||
|
* Build-time constants.
|
||||||
|
*
|
||||||
|
* This file deliberately has no imports or exports: that keeps it an ambient
|
||||||
|
* script rather than a module, so the declarations below are global.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** Release version, injected by `define` in vite.config.ts. */
|
||||||
|
declare const __APP_VERSION__: string;
|
||||||
@@ -50,7 +50,6 @@ export interface ArchiveFile {
|
|||||||
size: number;
|
size: number;
|
||||||
text(): Promise<string>;
|
text(): Promise<string>;
|
||||||
arrayBuffer(): Promise<ArrayBuffer>;
|
arrayBuffer(): Promise<ArrayBuffer>;
|
||||||
stream(): ReadableStream<Uint8Array>;
|
|
||||||
url?: string;
|
url?: string;
|
||||||
/**
|
/**
|
||||||
* A URL pointing at this file's contents. Local files mint a disk-backed
|
* A URL pointing at this file's contents. Local files mint a disk-backed
|
||||||
|
|||||||
@@ -3,9 +3,24 @@ import react from '@vitejs/plugin-react';
|
|||||||
import path from 'path';
|
import path from 'path';
|
||||||
import {defineConfig} from 'vite';
|
import {defineConfig} from 'vite';
|
||||||
import { VitePWA } from 'vite-plugin-pwa';
|
import { VitePWA } from 'vite-plugin-pwa';
|
||||||
|
import { createRequire } from 'module';
|
||||||
|
|
||||||
|
const { version } = createRequire(import.meta.url)('./package.json');
|
||||||
|
|
||||||
export default defineConfig(() => {
|
export default defineConfig(() => {
|
||||||
return {
|
return {
|
||||||
|
/**
|
||||||
|
* The release version, compiled into the client.
|
||||||
|
*
|
||||||
|
* This is load-bearing, not cosmetic. The service worker precaches
|
||||||
|
* index.html *including its response headers*, so a server-side header
|
||||||
|
* change (a CSP fix, say) never reaches an installed PWA: nothing in the
|
||||||
|
* client build changed, the precache manifest is byte-identical, and the
|
||||||
|
* worker has no reason to update. Baking the version in means every release
|
||||||
|
* changes the bundle hash, which changes index.html, which invalidates the
|
||||||
|
* precache and re-fetches the shell with current headers.
|
||||||
|
*/
|
||||||
|
define: { __APP_VERSION__: JSON.stringify(version) },
|
||||||
plugins: [
|
plugins: [
|
||||||
react(),
|
react(),
|
||||||
tailwindcss(),
|
tailwindcss(),
|
||||||
|
|||||||
Reference in New Issue
Block a user