JDownloader wrote story-shaped names for highlights during one period of
its life, so the same item exists on disk as both
0ct0ber19 - C5dQPEYpd9W.mp4
2024-04-07_0ct0ber19 - 01 - C5dQPEYpd9W.mp4
which parsed to the ids "C5dQPEYpd9W" and "01 - C5dQPEYpd9W" -- two posts
for one item. The leading ordinal is a position within a day's stories
and carries nothing the shortcode does not, so story and highlight ids
drop it. Post ids are untouched, since those are permalinks.
Measured on the two real files, same archive, cache cleared between:
without the fix the profile reads "Heestory - 2 items", with it
"Heestory - 1 item".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit had this backwards: the sidecar date was only
consulted when the existing date came from an mtime, so a filename date
silently outranked what Instagram itself reported.
The order is sidecar, then filename, then mtime -- metadata first,
mtime last, since mtime is when the file hit disk and says nothing about
when the post was made. Ties keep the incumbent so two equally
authoritative files cannot flip a post's date by scan order.
Extracted to src/lib/post-dates.ts rather than left inline, because the
rule is easy to state and easy to get wrong -- the tests include an
order-independence case that would have caught the original mistake.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The .json sidecars published with the ARTMS fetch were inert: the scanner
fed them through the Instaloader path, where `node.edge_media_to_caption`
and `checkIsStory`'s `product_type` are both absent, so nothing happened.
They are now recognised structurally -- flat, with post_shortcode and
type, and none of the markers the other two JSON shapes carry -- and used
for three things:
- `type` sets post.isReel, which post-tabs prefers over every fallback.
This is Instagram's own classification and it disagrees with ours a
lot: of 781 items in "official_artms - reels", the sidecars say only
360 are reels. The other 421 are feed videos the clips endpoint returns
via include_feed_video, and the directory-based rule counted them all.
- `description` fills the caption where no .txt exists.
- `date` dates a post whose filename could not.
Also fixes date precedence. Only JDownloader highlights lack a date in
the filename, so parseArchiveFilename now marks those as mtime-derived
and the scanner lets any real date replace them -- previously the date
depended on which file the scan reached first.
Verified against real published files: a directory of three type=post and
three type=reel renders 6 in the grid and exactly the 3 reels in the
Reels tab.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every claim in docs/gallery-dl.md was measured against the live site and
the archive rather than taken from documentation, because two of the
assumptions turned out to be wrong.
The safety model is the reason the config looks the way it does.
gallery-dl has two API backends: the graphql one issues a request PER
POST for every video and carousel -- the pattern that got this account
banned via Instaloader -- while the default rest one paginates listings
at 30-50 items and carries carousel_media, video_versions and
product_type inline. A 300-post profile costs ~10 requests.
Findings worth recording:
- JD2 stamped filenames in desktop LOCAL time (US Eastern), not UTC.
Across 212 comparable posts: UTC 19 mismatches, UTC-5 10, UTC-4 zero.
{date:Olocal/%Y-%m-%d} reproduces it; the trailing separator must be
omitted or it lands in the strftime format.
- A profile's reels tab returns collab reels owned by OTHER accounts, so
the directory must be forced with -D. JD2 did the same: chuuo3o and
official_artms filenames sit inside "0ct0ber19 - reels".
- Stories and highlights need per-item {shortcode}; {post_shortcode} is
the reel's id and is shared by every item. {date} is per-item, verified
on a 154-item highlight with distinct times.
- gallery-dl reproduces JD2's caption .txt exactly, including writing
nothing for an empty caption and omitting the trailing newline.
- The json sidecar needs `include`, not `fields`; `fields` silently does
nothing in mode:json and leaks audio_user blobs. It yields `type`
(post/reel) -- Instagram's own flag, which can retire the lone-video
heuristic once the scanner reads it.
Naming differences between the two tools are cosmetic: EXPORT_RE already
makes the index optional and parseInt normalises zero-padding, so a mixed
archive parses identically. Tests pin that down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Docker Build and Publish / build-and-push (push) Failing after 9s
The Posts tab filtered reels out, so the grid was not the archive — it
was the archive minus its videos. For `for.heejin` that hid 533 of 1225
posts; for `loonatheworld`, 1100 of 3813. On Instagram the grid holds
everything and the Reels tab is a filtered view of that same set.
Extract the tab logic to src/lib/post-tabs.ts so the reel heuristic is
testable outside the component, and add dedupePostCopies: the jd2 flow
crawls the profile URL and the /reels URL separately because the profile
page misses some reels, so the two overlap and a reel can land on disk
twice. Those are two posts with distinct directory-scoped ids, which the
grid would now render side by side; the reels-source copy wins so the
survivor is still recognised as a reel.
Deciding what *is* a reel stays a guess for most archives. Instagram
marks it with product_type ("clips" vs "feed" vs "igtv" — all three are
GraphVideo, and aspect ratio does not separate them), but only newer
Instaloader captures carry it: 1101 of gibiofficial's 5919 sidecars, and
only 2 marked clips. JDownloader archives carry none, so those still
fall back to treating a lone video as a reel.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Docker Build and Publish / build-and-push (push) Failing after 10s
Instaloader metadata was silently lost
Every .json.xz failed with "Failed to fetch" during a scan, though the same URL
fetched fine on its own. RemoteArchiveFile.stream() started a fetch, piped the
body into a TransformStream and returned the readable immediately — nothing
caught a fetch rejection, and the decompressor stops reading at the end of the
xz member, so the response body was never drained or cancelled. Across ~190
sidecars that exhausted the connection pool.
Everything Instaloader archives carry lives in those files, so the failure was
invisible but total. rivvsofficial reported 188 posts, no stories, 0 followers
and a placeholder bio; it now reports 68 posts, 120 stories, 10,337 followers
and the real name, bio and link — 68 + 120 = 188, matching the sidecars exactly
(106 GraphStoryVideo + 14 GraphStoryImage = 120).
These sidecars are a few KB, so they are now read into memory before
decompressing. stream() was left unused by that change and is removed from the
interface and both implementations rather than kept as a trap.
Arrow keys page the carousel
They moved between posts, which contradicted the arrows drawn on the carousel
itself. Arrows now page slides; , and . move between posts, alongside the side
buttons.
Backdrop cross-fade
AnimatePresence had no exit variant, so the outgoing scan backdrop was removed
instantly while its replacement faded in over 1.5s, exposing the pale page
behind it as a white flash. Layers now stack: the outgoing image holds full
opacity until covered, and the 0.4 moved onto the group so overlapping layers
don't darken as they cross. Measured over a real scan: 152 cross-fades with a
layer always opaque, except the opening fade-in where nothing is underneath.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 11s
The archive root was filtered by prefix, but the recursive walk below it was
not, so anything inside a profile directory got indexed. NAS filesystems put
sidecar metadata *inside* every folder rather than only at the share root:
Synology writes @eaDir (thumbnails and indexing data), #recycle holds
deletions, .sync is Resilio state. On the live share those account for 12,516
of 123,023 files.
None currently sit inside a profile directory, so nothing was miscounted yet —
but the moment that share gets indexed for Photos, every generated thumbnail
would be counted as archive media and stat'd one by one over the network, which
is the cost the index exists to avoid.
One isSystemDirectory rule now applies at every level, and the root listing uses
it too instead of keeping a second copy of the pattern.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 10s
Navigation gestures
Horizontal swipe used to advance the carousel and then, on the last slide,
fling you into the next post — one gesture meaning two things. Horizontal is
now carousel-only. On touch, vertical swipe moves between posts (down on the
first post still dismisses, keeping drag-to-close where it can't mean
"previous"). Desktop keeps the arrows outside the modal.
URLs
Permalinks now mirror Instagram:
/<archive>/ profile
/<archive>/reels/ tab
/<archive>/p/<shortcode>/ post
A post URL carries no tab, as on Instagram; the tab is re-derived from the
post's source, so opening a reel link lands on the Reels tab with next/prev
paging through reels. Sidecar posts keep directory-scoped ids internally but
expose only the shortcode. The old ?a=&t=&p= form is still parsed so existing
links keep working, and reserved prefixes (api, archives, assets…) can never be
mistaken for a profile name.
Animations
Adds a shared motion vocabulary tuned to feel native: critically damped springs
rather than fixed-duration easing, and gestures hand their exit velocity to the
animation so a flick continues instead of restarting. Post transitions animate
along the axis the input implies — vertical for a swipe, horizontal for the
arrows. Modal and story viewer present/dismiss with a scale, tiles and
highlight circles get touch-down feedback, and prefers-reduced-motion is
honoured throughout.
Adds 22 routing tests (58 total).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Security
- Fix path traversal in GET /api/archives/:name/files. Express decodes route
params after segment matching, so `..%2f..%2fetc` escaped ARCHIVES_DIR and
returned a recursive listing of arbitrary directories.
- Add CSP and baseline security headers; disable x-powered-by.
- Stop baking GEMINI_API_KEY into the client bundle (the SDK was unused).
- Run the container as `node` instead of root.
Performance
- Add a directory-mtime-keyed archive index, warmed in the background and
persisted. Listing 110k files went from ~52s to ~0.1s; the largest archive
(24k files) serves in ~0.3s. Per-file stat over CIFS costs ~1.4ms and does
not parallelise, so it is now done once rather than per request.
- Build media URLs from the File directly instead of
`new Blob([await file.arrayBuffer()])`, which read every media file fully
into memory (a 20GB archive tried to become 20GB of resident blobs).
- Track and revoke object URLs; previously none were ever revoked.
- Give `requestThumbnail` a stable identity so a completed thumbnail stops
re-running the effect in every mounted thumbnail.
- Namespace IndexedDB keys so listing archives no longer deserializes every
cached thumbnail blob, and thumbnails no longer collide across archives.
- Serve real file sizes: RemoteArchiveFile was constructed with size 0, which
silently disabled high-res thumbnailing for every server archive.
Correctness
- Local archives cached media as blob: URLs, which die with the document, so
a cached local archive restored as an archive of broken images. Media now
carries a stable path and is rehydrated from a persisted directory handle
(File System Access API), falling back to re-prompting for the folder.
- Fix permalinks: the URL-writing effect erased ?a= on mount before the
archive list arrived to consume it, so deep links never resolved.
- Make cache invalidation detect nested changes via a directory signature.
- Add an error boundary and tolerate unparseable dates, which previously
threw a RangeError and blanked the app.
- Default video to muted so autoplay is not blocked by Safari/Firefox.
Features
- Fold sidecar directories into their base profile: `<user> - reels`,
`story - <user>` and `story highlights - <user> - <title>` now appear as
reels, the story ring and Instagram-style highlight circles rather than as
separate archives.
Housekeeping
- Add @types/react; React was previously type-checked against its JavaScript
source, so `npm run lint` gave almost no type safety on components.
- Vendor fonts and PWA icons locally; the app made third-party CDN requests
despite advertising offline support and local-only processing.
- Drop unused better-sqlite3 (a native module that broke `npm install`).
- Add vitest with 36 tests over the filename and directory-naming rules.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Key changes:
- Added Web Worker for background image thumbnailing with a 1MiB threshold to optimize CPU/memory usage.
- Implemented a serial task queue for memory-safe high-res image processing, preventing OOM crashes.
- Added inter-post preloading in the modal for seamless 'Previous/Next' navigation.
- Refined scanning UI with double-buffering and a dark background to completely eliminate white flashes.
- Renamed project to 'instaarchive-viewer' in package.json.
- Fixed 'Open image in new tab' by denylisting /archives and /api in PWA config.
Summary of changes:
- Extracted archive scanning logic into a modular 'useArchiveScanner' hook for better maintainability and performance.
- Refined PostModal carousel with intelligent media preloading and smoother, jitter-free transitions.
- Optimized image rendering with 'decoding=async' and removed 'black flashes' between slide changes.
- Updated PWA configuration to 'autoUpdate' with hourly periodic checks for fresh content.
- Fixed several bugs including stories sorting, permalink parameter cleanup, and profile metadata cache restoration.
- Comprehensive updates to documentation (README.md and GEMINI.md) reflecting the new architecture.
ergosteurandClaude Opus 5