Compare commits

..
Author SHA1 Message Date
ergosteurandClaude Sonnet 5 600949e475 fix: make the browser Back button close a post instead of exiting the app
The URL was only ever synced with history.replaceState, so the app
never created any history entries of its own -- Back always went
straight to whatever page was open before this one, no matter where
you were in the app.

Opening a post now pushState's a new entry, matching how Instagram's
own back button behaves: Back closes the post and returns to the grid.
Closing a post any other way (the X button, the modal's own close
handler) consumes that same entry via history.back() instead of piling
a fresh one on top, and a popstate listener re-syncs app state for
both directions. Tab switches and archive loads still use
replaceState, unchanged -- only the post view gets its own step in
history, deliberately, to keep the history stack shallow.

Verified in a real browser: open a post, Back closes it and stays in
the app; Forward reopens it; the X button closes it too, consuming the
same entry rather than leaving a stale one behind.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
2026-08-27 14:36:36 -04:00
ergosteurandClaude Opus 5 26d2d3e379 chore: release 1.8.1
Docker Build and Publish / build-and-push (push) Failing after 11s
First build from the redacted history, and the first that does not ship
source comments in the server bundle.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
2026-08-20 15:36:17 -04:00
ergosteurandClaude Opus 5 61c2b62141 fix: stop shipping source comments in the server bundle
`tsc` keeps comments by default, so the doc comment in archive-grouping.ts
describing the sidecar layout was emitted into dist-server and copied into the
runtime image. Every published container image on ghcr carries it — verified by
pulling the dist-server layer of :latest and grepping it:

    app/src/lib/archive-grouping.js:10:  *   <user>  -> posts (base)

That comment names real archived accounts, which is exactly what main was
redacted to remove, so the redaction was incomplete while the build kept
re-emitting them. The frontend was never affected: Vite strips comments, and
the 432K dist layer greps clean.

--removeComments takes dist-server from 0 comment lines. docs/ was never at
risk; the multi-stage build copies only dist/ and dist-server/ into runtime.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
2026-08-20 15:36:00 -04:00
ergosteurandClaude Opus 5 882296b1c0 chore: move the archive-fetching tooling out of this branch
The fetching scripts and their docs now live on the `tooling` branch, which
is not published to GitHub. This removes the two references that would
otherwise dangle here: the `jd2` npm script and the CLAUDE.md bullet
describing it.

The viewer's own gallery-dl support is untouched and stays here —
src/lib/gallery-dl-sidecar.ts and friends parse sidecars at display time and
are app code, not tooling.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
2026-08-20 14:52:03 -04:00
ergosteurandClaude Opus 5 4f8b0021c6 chore: stop tracking compiled Python bytecode
Two .pyc files under scripts/__pycache__ were committed at some point and have
been churning ever since — merely importing gdl-sync.py to check a config
rewrites them and dirties the tree, which is how they surfaced.

.gitignore had no Python entries at all, only Node ones. The files stay on
disk; this just untracks them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
2026-08-20 14:38:41 -04:00
ergosteurandClaude Opus 5 dcd8f2ef1d chore: release 1.8.0
Docker Build and Publish / build-and-push (push) Failing after 10s
Ships the gallery-dl sidecar work to the viewer. The visible change is
reel classification: official_artms' Reels tab drops from 781 items to
360, because the sidecars say the other 421 are ordinary feed videos the
clips endpoint returns via include_feed_video. Directory-based
classification counted them all as reels.

Also in this release: dates ranked by source rather than scan order, and
highlight items no longer appearing twice when the archive holds them
under both JDownloader naming conventions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 13:58:15 -04:00
ergosteurandClaude Opus 5 5d5dea10c8 fix: stop showing a highlight item twice under two naming conventions
JDownloader wrote story-shaped names for highlights during one period of
its life, so the same item exists on disk as both

    0ct0ber19 - C5dQPEYpd9W.mp4
    2024-04-07_0ct0ber19 - 01 - C5dQPEYpd9W.mp4

which parsed to the ids "C5dQPEYpd9W" and "01 - C5dQPEYpd9W" -- two posts
for one item. The leading ordinal is a position within a day's stories
and carries nothing the shortcode does not, so story and highlight ids
drop it. Post ids are untouched, since those are permalinks.

Measured on the two real files, same archive, cache cleared between:
without the fix the profile reads "Heestory - 2 items", with it
"Heestory - 1 item".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 13:57:42 -04:00
ergosteurandClaude Opus 5 816fa970b5 fix: rank date sources instead of letting scan order decide
The previous commit had this backwards: the sidecar date was only
consulted when the existing date came from an mtime, so a filename date
silently outranked what Instagram itself reported.

The order is sidecar, then filename, then mtime -- metadata first,
mtime last, since mtime is when the file hit disk and says nothing about
when the post was made. Ties keep the incumbent so two equally
authoritative files cannot flip a post's date by scan order.

Extracted to src/lib/post-dates.ts rather than left inline, because the
rule is easy to state and easy to get wrong -- the tests include an
order-independence case that would have caught the original mistake.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 12:50:48 -04:00
ergosteurandClaude Opus 5 53b1f80e1d feat: read gallery-dl sidecars for reel type and post dates
The .json sidecars published with the ARTMS fetch were inert: the scanner
fed them through the Instaloader path, where `node.edge_media_to_caption`
and `checkIsStory`'s `product_type` are both absent, so nothing happened.

They are now recognised structurally -- flat, with post_shortcode and
type, and none of the markers the other two JSON shapes carry -- and used
for three things:

- `type` sets post.isReel, which post-tabs prefers over every fallback.
  This is Instagram's own classification and it disagrees with ours a
  lot: of 781 items in "official_artms - reels", the sidecars say only
  360 are reels. The other 421 are feed videos the clips endpoint returns
  via include_feed_video, and the directory-based rule counted them all.
- `description` fills the caption where no .txt exists.
- `date` dates a post whose filename could not.

Also fixes date precedence. Only JDownloader highlights lack a date in
the filename, so parseArchiveFilename now marks those as mtime-derived
and the scanner lets any real date replace them -- previously the date
depended on which file the scan reached first.

Verified against real published files: a directory of three type=post and
three type=reel renders 6 in the grid and exactly the 3 reels in the
Reels tab.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 12:42:54 -04:00
ergosteurandClaude Opus 5 89bd5346db docs: design a gallery-dl replacement for the JDownloader fetcher
Every claim in docs/gallery-dl.md was measured against the live site and
the archive rather than taken from documentation, because two of the
assumptions turned out to be wrong.

The safety model is the reason the config looks the way it does.
gallery-dl has two API backends: the graphql one issues a request PER
POST for every video and carousel -- the pattern that got this account
banned via Instaloader -- while the default rest one paginates listings
at 30-50 items and carries carousel_media, video_versions and
product_type inline. A 300-post profile costs ~10 requests.

Findings worth recording:

- JD2 stamped filenames in desktop LOCAL time (US Eastern), not UTC.
  Across 212 comparable posts: UTC 19 mismatches, UTC-5 10, UTC-4 zero.
  {date:Olocal/%Y-%m-%d} reproduces it; the trailing separator must be
  omitted or it lands in the strftime format.
- A profile's reels tab returns collab reels owned by OTHER accounts, so
  the directory must be forced with -D. JD2 did the same: chuuo3o and
  official_artms filenames sit inside "0ct0ber19 - reels".
- Stories and highlights need per-item {shortcode}; {post_shortcode} is
  the reel's id and is shared by every item. {date} is per-item, verified
  on a 154-item highlight with distinct times.
- gallery-dl reproduces JD2's caption .txt exactly, including writing
  nothing for an empty caption and omitting the trailing newline.
- The json sidecar needs `include`, not `fields`; `fields` silently does
  nothing in mode:json and leaks audio_user blobs. It yields `type`
  (post/reel) -- Instagram's own flag, which can retire the lone-video
  heuristic once the scanner reads it.

Naming differences between the two tools are cosmetic: EXPORT_RE already
makes the index optional and parseInt normalises zero-padding, so a mixed
archive parses identically. Tests pin that down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 21:04:39 -04:00
ergosteurandClaude Opus 5 b153a49bbb fix: count the grid in the post header, and qualify the Instagram claim
Docker Build and Publish / build-and-push (push) Failing after 10s
The header rendered allPosts.length, which is pre-dedupe — 0ct0ber19
showed "303 posts" over a 300-tile grid. Instagram's counter equals its
grid, so count the grid.

Also correct CLAUDE.md. v1.7.0 claimed the grid holds everything "as on
Instagram"; Instagram actually includes a reel in the grid only when the
creator shared it to feed, per post. Measured live: official_artms has
21 reels in its first 34 grid tiles, 0ct0ber19 has 1 in 214. Archives
carry no such flag, so showing everything approximates the behaviour
rather than reproducing it.

Records the DOM trap that caused the wrong reading in the first place:
grid reels link to /reel/<code>/, not /<user>/p/<code>/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 20:00:11 -04:00
ergosteurandClaude Opus 5 8b053b4b2e feat: show reels in the profile grid, as Instagram does
Docker Build and Publish / build-and-push (push) Failing after 9s
The Posts tab filtered reels out, so the grid was not the archive — it
was the archive minus its videos. For `for.heejin` that hid 533 of 1225
posts; for `loonatheworld`, 1100 of 3813. On Instagram the grid holds
everything and the Reels tab is a filtered view of that same set.

Extract the tab logic to src/lib/post-tabs.ts so the reel heuristic is
testable outside the component, and add dedupePostCopies: the jd2 flow
crawls the profile URL and the /reels URL separately because the profile
page misses some reels, so the two overlap and a reel can land on disk
twice. Those are two posts with distinct directory-scoped ids, which the
grid would now render side by side; the reels-source copy wins so the
survivor is still recognised as a reel.

Deciding what *is* a reel stays a guess for most archives. Instagram
marks it with product_type ("clips" vs "feed" vs "igtv" — all three are
GraphVideo, and aspect ratio does not separate them), but only newer
Instaloader captures carry it: 1101 of gibiofficial's 5919 sidecars, and
only 2 marked clips. JDownloader archives carry none, so those still
fall back to treating a lone video as a reel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 19:06:53 -04:00
ergosteurandClaude Opus 5 d1fa1a8d2f docs: record the CSP/wasm and PWA-precache traps in CLAUDE.md
Two failures in this session were expensive because nothing recorded them:

- The CSP must keep 'wasm-unsafe-eval' and connect-src data:, because the xz
  decompressor for Instaloader sidecars is WebAssembly embedded as a data: URL.
  Removing either breaks decoding with a bare "Failed to fetch" and no stack,
  and the visible symptom is silent metadata loss rather than an error.
- The service worker precaches index.html with its headers, so a server-only
  change never reaches installed clients. The version compiled into the client
  is what forces the precache to turn over each release; it is load-bearing,
  not decoration.

Also notes that the Vite dev server sends none of these headers, so CSP and PWA
behaviour must be verified against a built dist/ served by server.js, and that
the live archive root is now the archives/ subdirectory.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 18:36:09 -04:00
ergosteurandClaude Opus 5 b9ece021d4 fix: make server header changes reach installed PWA clients
Docker Build and Publish / build-and-push (push) Failing after 10s
The service worker precaches index.html together with its response headers, so
a server-only change never reaches an installed client: the client build is
byte-identical, the precache manifest is unchanged, and the worker has no
reason to update. That is why the CSP fix in 1.6.1 did not reach a browser that
already had the app cached — it kept replaying a cached shell carrying the old,
broken CSP, indefinitely.

The release version is now compiled into the client, which makes every release
change the bundle hash, hence index.html, hence its precache revision, hence
sw.js itself — the bytes browsers compare to decide whether to update. Verified
by bumping only the version: index-DYufL2Fa.js -> index-60N_3d5j.js, with the
new name carried into the sw.js manifest.

It also surfaces in the footer, so the deployed version is visible without
digging through devtools.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 13:16:25 -04:00
ergosteurandClaude Opus 5 f300b8d9f5 fix: allow WebAssembly in the CSP so xz sidecars can be decoded
Docker Build and Publish / build-and-push (push) Failing after 10s
The xz decompressor for Instaloader's .json.xz sidecars is WebAssembly,
embedded as a data: URL that it fetches at startup. The CSP added in 1.3.0
blocked both halves of that:

  fetch('data:application/wasm;...')  -> TypeError: Failed to fetch
  WebAssembly.instantiate(...)        -> CompileError: violates script-src 'self'

The first surfaces through new Response(stream).json() as a bare "Failed to
fetch" with no stack, which reads like a network fault and is why this was
mis-diagnosed twice. Vite's dev server never sends the CSP, so it reproduced
only in production — every Instaloader archive silently lost its captions,
story flags and profile metadata from 1.3.0 onward.

script-src now allows 'wasm-unsafe-eval', which permits WebAssembly compilation
without permitting eval() of JavaScript, and connect-src allows data: for the
embedded module.

Verified against the production bundle: rivvsofficial goes from 188 posts / 0
followers / no stories to 68 posts, 120 stories, 10,337 followers and its real
name, bio and link — with zero decode errors.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 12:51:42 -04:00
ergosteurandClaude Opus 5 e57be521a2 fix: read xz sidecars by buffer, page carousel with arrows, stop backdrop flash
Docker Build and Publish / build-and-push (push) Failing after 10s
Instaloader metadata was silently lost
Every .json.xz failed with "Failed to fetch" during a scan, though the same URL
fetched fine on its own. RemoteArchiveFile.stream() started a fetch, piped the
body into a TransformStream and returned the readable immediately — nothing
caught a fetch rejection, and the decompressor stops reading at the end of the
xz member, so the response body was never drained or cancelled. Across ~190
sidecars that exhausted the connection pool.

Everything Instaloader archives carry lives in those files, so the failure was
invisible but total. rivvsofficial reported 188 posts, no stories, 0 followers
and a placeholder bio; it now reports 68 posts, 120 stories, 10,337 followers
and the real name, bio and link — 68 + 120 = 188, matching the sidecars exactly
(106 GraphStoryVideo + 14 GraphStoryImage = 120).

These sidecars are a few KB, so they are now read into memory before
decompressing. stream() was left unused by that change and is removed from the
interface and both implementations rather than kept as a trap.

Arrow keys page the carousel
They moved between posts, which contradicted the arrows drawn on the carousel
itself. Arrows now page slides; , and . move between posts, alongside the side
buttons.

Backdrop cross-fade
AnimatePresence had no exit variant, so the outgoing scan backdrop was removed
instantly while its replacement faded in over 1.5s, exposing the pale page
behind it as a white flash. Layers now stack: the outgoing image holds full
opacity until covered, and the 0.4 moved onto the group so overlapping layers
don't darken as they cross. Measured over a real scan: 152 cross-fades with a
layer always opaque, except the opening fade-in where nothing is underneath.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 12:28:26 -04:00
ergosteurandClaude Opus 5 792b834cbe docs: add a JDownloader quick reference
Covers why fetching goes through JDownloader rather than Instaloader (the
instagram.com vs CDN split, and what the metadata gap actually costs), the
settings that matter, cookie handling, the two-URL workflow, jd2-sync usage,
the expected on-disk layout, and what to do when something breaks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 12:02:41 -04:00
ergosteurandClaude Opus 5 106d3f6691 fix: never descend into NAS metadata directories when indexing
Docker Build and Publish / build-and-push (push) Failing after 11s
The archive root was filtered by prefix, but the recursive walk below it was
not, so anything inside a profile directory got indexed. NAS filesystems put
sidecar metadata *inside* every folder rather than only at the share root:
Synology writes @eaDir (thumbnails and indexing data), #recycle holds
deletions, .sync is Resilio state. On the live share those account for 12,516
of 123,023 files.

None currently sit inside a profile directory, so nothing was miscounted yet —
but the moment that share gets indexed for Photos, every generated thumbnail
would be counted as archive media and stat'd one by one over the network, which
is the cost the index exists to avoid.

One isSystemDirectory rule now applies at every level, and the root listing uses
it too instead of keeping a second copy of the pattern.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 11:57:29 -04:00
ergosteurandClaude Opus 5 92a4ada3c2 feat: generate JDownloader crawljobs from the archives on disk
The manual flow is: paste a profile URL into JDownloader, paste the /reels URL
separately (the profile page misses some reels), set the output folder by hand,
repeat per profile. scripts/jd2-sync.ts emits one crawljob per source with the
folder already pointed at the right directory, so folder-watch picks up the
whole batch at once.

Profiles and sidecars are derived with the same grouping logic the server uses,
so output folders always match what the viewer expects to find. --download-base
maps the path for a JDownloader running on another machine (Windows paths
included), since it typically runs on a desktop against the share.

Directories that aren't Instagram profiles are skipped: an archive root also
collects tool output and exports from other services, and pointing a crawl at
those spends requests on instagram.com to be told the profile doesn't exist —
exactly the traffic worth not spending. Filtering is by username shape, plus
--skip and a .jd2ignore file for names that look like usernames but aren't.

Defaults are conservative: chunks=1, because multi-chunk ranged requests are the
one CDN-side pattern that doesn't resemble a browser, and links park in the
LinkGrabber for review rather than auto-starting.

Only posts and reels are emitted; highlight URLs need a numeric id and story
URLs expire, so those stay manual.

Format verified against JDownloader's own explain.txt for the folderwatch
extension, read from the daily SVN mirror rather than one of the decade-stale
GitHub copies.

Also refreshes CLAUDE.md, whose URL-state section still described the query
parameters replaced in 1.4.0, and documents the mobile feed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 11:41:23 -04:00
ergosteurandClaude Opus 5 c54f8d5b09 feat: mobile opens posts as a scrolling feed instead of a modal
Docker Build and Publish / build-and-push (push) Failing after 10s
Tapping a post on a phone now opens a real feed page — header, media, actions,
caption, next post peeking in below — scrolled with the browser's own vertical
scrolling rather than swipe gestures. Desktop keeps the modal, where a centred
sheet with side arrows suits a pointer.

Only a window of posts is mounted: a profile here holds up to 1129 posts and
mounting them all would mean as many full-size images. The window grows in both
directions as you scroll. Growing upwards shifts everything below it, so the
scroll offset is corrected in the same frame, before paint — measured against
the real archive, an anchored post moves exactly one screen per scroll with no
jump.

Only the post crossing the viewport centre plays its video; the rest stay
paused, so a feed of reels doesn't play ten at once. The URL tracks that same
post, so scrolling updates /<archive>/p/<shortcode>/ the way Instagram does,
and the back button returns to the grid with its scroll position intact.

Feed video sizes to the container width rather than its intrinsic size: a
<video> reports 300x150 until metadata loads, which made it render narrow and
then jump to full width. It also gets a taller height ceiling than the modal so
ordinary portrait media fills the width instead of sitting in side bars.

The carousel is extracted into a shared MediaCarousel used by both surfaces, so
horizontal paging behaves identically; touch-action keeps vertical scrolling
passing through to the feed. PostModal loses its now-dead mobile swipe branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 10:23:33 -04:00
ergosteurandClaude Opus 5 877d21ff1f feat: Instagram-shaped URLs, Instagram-shaped gestures, iOS-feel animations
Docker Build and Publish / build-and-push (push) Failing after 10s
Navigation gestures
Horizontal swipe used to advance the carousel and then, on the last slide,
fling you into the next post — one gesture meaning two things. Horizontal is
now carousel-only. On touch, vertical swipe moves between posts (down on the
first post still dismisses, keeping drag-to-close where it can't mean
"previous"). Desktop keeps the arrows outside the modal.

URLs
Permalinks now mirror Instagram:

  /<archive>/                 profile
  /<archive>/reels/           tab
  /<archive>/p/<shortcode>/   post

A post URL carries no tab, as on Instagram; the tab is re-derived from the
post's source, so opening a reel link lands on the Reels tab with next/prev
paging through reels. Sidecar posts keep directory-scoped ids internally but
expose only the shortcode. The old ?a=&t=&p= form is still parsed so existing
links keep working, and reserved prefixes (api, archives, assets…) can never be
mistaken for a profile name.

Animations
Adds a shared motion vocabulary tuned to feel native: critically damped springs
rather than fixed-duration easing, and gestures hand their exit velocity to the
animation so a flick continues instead of restarting. Post transitions animate
along the axis the input implies — vertical for a swipe, horizontal for the
arrows. Modal and story viewer present/dismiss with a scale, tiles and
highlight circles get touch-down feedback, and prefers-reduced-motion is
honoured throughout.

Adds 22 routing tests (58 total).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 10:01:04 -04:00
ergosteurandClaude Opus 5 0cff91edae fix: keep post nav arrows outside the modal and stop scroll chaining
Docker Build and Publish / build-and-push (push) Failing after 11s
The prev/next arrows are fixed to the viewport edges while the modal grows to
fill the available width, so below roughly 1200px the modal slid underneath
them and a white chevron landed on the white caption panel — invisible until
hovered. The overlay now reserves a horizontal gutter (md:px-16 lg:px-24) so
the arrows always sit outside the modal, and they get a solid white pill with a
dark chevron so they read against anything behind them. Verified clearing the
modal at 768, 1024, 1440 and 1920px.

The caption sidebar shrinks to w-80 at md so the narrower modal doesn't squeeze
the media pane.

Also adds overscroll-contain to the overlay: the wheel previously chained
through to the post grid behind it, scrolling the background while a post was
open.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 09:05:24 -04:00
ergosteurandClaude Opus 5 0b4b20e0ff feat: fit full-view media to the viewport and play with sound
Docker Build and Publish / build-and-push (push) Failing after 10s
Media in the post modal used w-full/h-auto, so a portrait video or image grew
taller than the screen (a 720x1280 reel rendered 768x1365 in a 786px viewport)
and forced the modal to scroll. Full view now caps height to the viewport minus
the modal's own padding. Video sizes to its own aspect within the cap so a
portrait clip isn't letterboxed edge to edge; images keep filling the modal
width and only gain a height ceiling.

Opening the modal or a story reel is a user gesture, so playback now starts
unmuted and only falls back to muted if the browser actually refuses the
play() promise — previously it always started muted, and the earlier
muted-by-default fix meant a blocked video could stall the story progress bar.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 08:50:49 -04:00
ergosteurandClaude Opus 5 c0b6b6cf3e docs: update CLAUDE.md for the archive index, sidecars and cache rehydration
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 02:14:11 -04:00
ergosteurandClaude Opus 5 ab7099220a chore: bump version to 1.3.1
Docker Build and Publish / build-and-push (push) Failing after 10s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 02:08:35 -04:00
ergosteurandClaude Opus 5 e6663657b0 fix: don't crash at boot when running under a UID with no passwd entry
os.userInfo() throws ERR_SYSTEM_ERROR (uv_os_get_passwd) for a UID that has no
/etc/passwd entry, which is exactly what `docker run --user 1234:1234` produces
— the very workaround the README recommends. Combined with the switch to a
non-root image user, this crashed the server on startup for any deployment that
needed a custom UID to read its archives.

Also document the non-root default and the /cache index volume.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 02:08:35 -04:00
ergosteurandClaude Opus 5 24ff2727c8 Merge branch 'review-fixes': security, performance and sidecar archive support
Docker Build and Publish / build-and-push (push) Failing after 10s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 01:59:41 -04:00
ergosteurandClaude Opus 5 f089e49e84 chore: bump version to 1.3.0
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 01:59:41 -04:00
ergosteurandClaude Opus 5 1d86fa3583 fix: security, performance and correctness pass; add sidecar archive support
Security
- Fix path traversal in GET /api/archives/:name/files. Express decodes route
  params after segment matching, so `..%2f..%2fetc` escaped ARCHIVES_DIR and
  returned a recursive listing of arbitrary directories.
- Add CSP and baseline security headers; disable x-powered-by.
- Stop baking GEMINI_API_KEY into the client bundle (the SDK was unused).
- Run the container as `node` instead of root.

Performance
- Add a directory-mtime-keyed archive index, warmed in the background and
  persisted. Listing 110k files went from ~52s to ~0.1s; the largest archive
  (24k files) serves in ~0.3s. Per-file stat over CIFS costs ~1.4ms and does
  not parallelise, so it is now done once rather than per request.
- Build media URLs from the File directly instead of
  `new Blob([await file.arrayBuffer()])`, which read every media file fully
  into memory (a 20GB archive tried to become 20GB of resident blobs).
- Track and revoke object URLs; previously none were ever revoked.
- Give `requestThumbnail` a stable identity so a completed thumbnail stops
  re-running the effect in every mounted thumbnail.
- Namespace IndexedDB keys so listing archives no longer deserializes every
  cached thumbnail blob, and thumbnails no longer collide across archives.
- Serve real file sizes: RemoteArchiveFile was constructed with size 0, which
  silently disabled high-res thumbnailing for every server archive.

Correctness
- Local archives cached media as blob: URLs, which die with the document, so
  a cached local archive restored as an archive of broken images. Media now
  carries a stable path and is rehydrated from a persisted directory handle
  (File System Access API), falling back to re-prompting for the folder.
- Fix permalinks: the URL-writing effect erased ?a= on mount before the
  archive list arrived to consume it, so deep links never resolved.
- Make cache invalidation detect nested changes via a directory signature.
- Add an error boundary and tolerate unparseable dates, which previously
  threw a RangeError and blanked the app.
- Default video to muted so autoplay is not blocked by Safari/Firefox.

Features
- Fold sidecar directories into their base profile: `<user> - reels`,
  `story - <user>` and `story highlights - <user> - <title>` now appear as
  reels, the story ring and Instagram-style highlight circles rather than as
  separate archives.

Housekeeping
- Add @types/react; React was previously type-checked against its JavaScript
  source, so `npm run lint` gave almost no type safety on components.
- Vendor fonts and PWA icons locally; the app made third-party CDN requests
  despite advertising offline support and local-only processing.
- Drop unused better-sqlite3 (a native module that broke `npm install`).
- Add vitest with 36 tests over the filename and directory-naming rules.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 01:59:19 -04:00
ergosteur 0ba7a0d9ad docs: update documentation for high-res performance and local persistence
Docker Build and Publish / build-and-push (push) Failing after 9s
Key changes:
- Updated README.md and GEMINI.md with details on background thumbnailing and inter-post preloading.
- Documented persistent local archive caching and smart profile fallback features.
- Added dist-server/ to .gitignore.
- Restored missing feature descriptions and troubleshooting tips in README.
2026-03-07 21:59:43 -05:00
ergosteur 899e8dfbbb feat: enable persistent local archives and smart profile fallback
Key changes:
- Enabled full metadata caching for local folder archives, allowing them to load instantly from IndexedDB without re-uploading.
- Implemented oldest-image fallback for profiles missing an explicit profile picture.
- Restored folder-name-to-username detection for local archive uploads.
- Optimized scan indexing to track all image files for fallback use.
2026-03-07 21:56:25 -05:00
ergosteur 8809b7794b fix: restore white glass scanning UI and resolve small image blur bug
Key changes:
- Corrected logic in PostThumbnail to prevent blur effects on images smaller than 1MiB.
- Restored the white glass aesthetic to the scanning dashboard with improved contrast and transparency.
- Optimized scanning background transitions to ensure a smooth, flicker-free crossfade.
2026-03-07 21:46:11 -05:00
ergosteur 8ec2c07b3f perf: implement background thumbnail generation and inter-post preloading
Key changes:
- Added Web Worker for background image thumbnailing with a 1MiB threshold to optimize CPU/memory usage.
- Implemented a serial task queue for memory-safe high-res image processing, preventing OOM crashes.
- Added inter-post preloading in the modal for seamless 'Previous/Next' navigation.
- Refined scanning UI with double-buffering and a dark background to completely eliminate white flashes.
- Renamed project to 'instaarchive-viewer' in package.json.
- Fixed 'Open image in new tab' by denylisting /archives and /api in PWA config.
2026-03-07 21:42:56 -05:00
ergosteur 3c50be286e chore: bump version to 1.2.0
Docker Build and Publish / build-and-push (push) Failing after 10s
2026-03-07 21:16:39 -05:00
ergosteur 7fc31fed94 feat: modularize scanner, enhance carousel preloading, and improve PWA updates
Summary of changes:
- Extracted archive scanning logic into a modular 'useArchiveScanner' hook for better maintainability and performance.
- Refined PostModal carousel with intelligent media preloading and smoother, jitter-free transitions.
- Optimized image rendering with 'decoding=async' and removed 'black flashes' between slide changes.
- Updated PWA configuration to 'autoUpdate' with hourly periodic checks for fresh content.
- Fixed several bugs including stories sorting, permalink parameter cleanup, and profile metadata cache restoration.
- Comprehensive updates to documentation (README.md and GEMINI.md) reflecting the new architecture.
2026-03-07 21:16:28 -05:00
ergosteur 0ed4cf292c fix: optimize scanning performance and resolve zero-post bug
Docker Build and Publish / build-and-push (push) Failing after 8s
2026-03-07 20:25:23 -05:00
ergosteur f51b82e37a fix: restore missing UI handlers and finalize generic parser
Docker Build and Publish / build-and-push (push) Failing after 9s
2026-03-07 20:18:33 -05:00
ergosteur e539677ec3 fix: refine dockerignore and bump version to v1.1.4
Docker Build and Publish / build-and-push (push) Failing after 10s
2026-03-07 20:10:40 -05:00
ergosteur 418997779c feat: implement permalinks and document PWA cache troubleshooting
Docker Build and Publish / build-and-push (push) Failing after 10s
2026-03-07 05:20:09 -05:00
ergosteur 3896506600 debug: add verbose logging to permalink synchronization 2026-03-07 05:12:14 -05:00
ergosteur bc0f4c230d fix: improve permalink comparison and add debug logging 2026-03-07 05:10:43 -05:00
ergosteur ad2f6ec083 feat: implement permalinks for archives, tabs, and posts 2026-03-07 05:07:04 -05:00
ergosteur bdcedab2da fix: exhaustive generic parser and implement local archive history 2026-03-07 05:05:08 -05:00
ergosteur c196f40cfd fix: address Docker EACCES errors with better logging and SELinux hints
Docker Build and Publish / build-and-push (push) Failing after 9s
2026-03-07 03:02:02 -05:00
ergosteur 2d0501f124 fix: improve Docker archive discovery and switch to compiled server
Docker Build and Publish / build-and-push (push) Failing after 10s
2026-03-07 02:57:19 -05:00
ergosteur be54b42e9e docs: update README and GEMINI with Docker usage and new features 2026-03-07 02:50:18 -05:00
ergosteur 5842a773d3 feat: add Dockerfile and GitHub Actions workflow for GHCR deployment
Docker Build and Publish / build-and-push (push) Failing after 9s
2026-03-07 02:45:17 -05:00
ergosteur eb66250511 feat: refine navigation protection to only warn when leaving the app 2026-03-07 02:41:16 -05:00
ergosteur c34aaae63c feat: add explicit confirmation for back button and refresh in archives 2026-03-07 02:39:39 -05:00
ergosteur 5bd6534adb feat: add navigation protection and refine cached badge visibility 2026-03-07 02:37:17 -05:00
ergosteur c86ee00e9b fix: resolve Firefox media warnings by improving video cleanup 2026-03-07 02:30:46 -05:00
ergosteur ab0001efbc feat: implement persistent caching, glassy scanning UI, and UI refinements 2026-03-07 02:28:22 -05:00
ergosteur afacc61888 feat: implement self-hostable mode with server-side directory scanning 2026-03-07 00:59:31 -05:00
ergosteur ee87e9b523 docs: update README and GEMINI.md, remove AI Studio boilerplate and .env.example 2026-03-07 00:36:46 -05:00
ergosteur 0e30c8b8b0 feat: enhance story viewer and media playback experience 2026-03-07 00:31:04 -05:00
ergosteur c8e3026ffd feat: improve archive parsing, add .json.xz support, and fix profile pic display 2026-03-07 00:03:54 -05:00
ergosteur e4c53364c3 feat: Initialize InstaArchive PWA project
Sets up a new React PWA project with Vite, Tailwind CSS, and basic PWA features. Includes essential files like README, .gitignore, package.json, and initial app structure.
2026-03-06 22:41:48 -05:00
ergosteurandGitHub eb8adf9dc2 Initial commit 2026-03-06 22:41:33 -05:00
20 changed files with 153 additions and 2244 deletions
+2
View File
@@ -9,3 +9,5 @@ coverage/
!.env.example
_sample-archives
_gemini-plans
__pycache__/
*.pyc
+8 -11
View File
@@ -15,9 +15,6 @@ InstaArchive Viewer is a React 19 + Vite 6 PWA for browsing archived Instagram d
- `npm run lint` — type-check only (`tsc --noEmit`)
- `npm test` / `npm run test:watch` — vitest
- `npx vitest run src/lib/archive-patterns.test.ts` — a single test file
- `npm run jd2 -- --archives <dir> --dry-run` — generate JDownloader `.crawljob`
files for every profile on disk (see `scripts/jd2-sync.ts` and
`docs/jdownloader.md`)
Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not.
@@ -37,10 +34,10 @@ Loading is unified behind the `ArchiveFile` interface (`src/types/index.ts`, imp
An archive root holds one directory per profile plus *sidecars* that belong to it:
```
0ct0ber19 -> posts (base)
0ct0ber19 - reels -> reels
story - 0ct0ber19 -> stories
story highlights - 0ct0ber19 - Heestory -> highlight "Heestory"
4utumn07 -> posts (base)
4utumn07 - reels -> reels
story - 4utumn07 -> stories
story highlights - 4utumn07 - Sunstory -> highlight "Sunstory"
```
`src/lib/archive-grouping.ts` (shared by server and tests) folds these into a single profile with a `sources` list. Sidecars never appear as standalone archives. Each file the server returns carries its `kind`, so the client routes posts / reels / story ring / highlight circles without re-deriving naming rules.
@@ -72,7 +69,7 @@ Three different JSON shapes turn up as `.json`, so they are told apart structura
| Instaloader `.json.xz` | GraphQL node under `node` / `__typename` |
| gallery-dl sidecar | flat, `post_shortcode` + `type`, none of the above |
The gallery-dl sidecar is the only source that states what a post *is*: its `type` (`post` / `reel` / `story` / `highlight`) is Instagram's own classification, so `post.isReel` set from it beats every fallback in `post-tabs.ts`. This matters — of the 781 items in `official_artms - reels`, the sidecars say only **360 are reels**; the other 421 are ordinary feed videos the clips endpoint returns via `include_feed_video`. Directory-based classification counted all 781.
The gallery-dl sidecar is the only source that states what a post *is*: its `type` (`post` / `reel` / `story` / `highlight`) is Instagram's own classification, so `post.isReel` set from it beats every fallback in `post-tabs.ts`. This matters — of the 781 items in `official_band - reels`, the sidecars say only **360 are reels**; the other 421 are ordinary feed videos the clips endpoint returns via `include_feed_video`. Directory-based classification counted all 781.
**Dates are ranked, not last-write-wins** (`src/lib/post-dates.ts`): sidecar (what Instagram reported) beats filename (what the fetcher wrote) beats mtime (when the file hit disk, and unrelated to when it was posted). Ties keep the incumbent. Several files describe one post and they are scanned in directory order, not in order of trustworthiness, so without the ranking the date was decided by whichever file came first. Only JDownloader highlights fall to mtime at all — `parseArchiveFilename` flags those via `dateFromMtime`.
@@ -88,15 +85,15 @@ Images over 1MiB are downscaled in a Web Worker via `OffscreenCanvas`. The queue
### Profile tabs (`src/lib/post-tabs.ts`)
The grid holds **everything**, reels included, and the Reels tab is a *filtered view* of that same set. Only the Reels tab filters. The tabs were mutually exclusive until v1.7.0, which hid a lot: 1100 of `loonatheworld`'s 3813 posts and 533 of `for.heejin`'s 1225 never appeared in the grid at all.
The grid holds **everything**, reels included, and the Reels tab is a *filtered view* of that same set. Only the Reels tab filters. The tabs were mutually exclusive until v1.7.0, which hid a lot: 1100 of `groupfandom`'s 3813 posts and 533 of `for.member`'s 1225 never appeared in the grid at all.
This *approximates* Instagram rather than matching it. Instagram's grid includes a reel only if the creator shared it to feed — a per-post choice, measured live on 2026-08-16: `official_artms` had 21 reels in its first 34 grid tiles, `0ct0ber19` just 1 in 214. That flag appears nowhere in an archive (JD2 stores no metadata, and Instaloader's `product_type` says what a post *is*, not whether it was shared to feed), so showing everything is the closest reachable behaviour. Instagram's "N posts" counter equals its grid, which is why the header counts `postsForTab(allPosts, 'posts')` and not `allPosts` — the raw list still holds both copies of a double-fetched post.
This *approximates* Instagram rather than matching it. Instagram's grid includes a reel only if the creator shared it to feed — a per-post choice, measured live on 2026-08-16: `official_band` had 21 reels in its first 34 grid tiles, `4utumn07` just 1 in 214. That flag appears nowhere in an archive (JD2 stores no metadata, and Instaloader's `product_type` says what a post *is*, not whether it was shared to feed), so showing everything is the closest reachable behaviour. Instagram's "N posts" counter equals its grid, which is why the header counts `postsForTab(allPosts, 'posts')` and not `allPosts` — the raw list still holds both copies of a double-fetched post.
When checking the live site, note that grid reels link to `/reel/<code>/` while ordinary posts link to `/<user>/p/<code>/`. Matching only `/p/` silently drops every reel, which once produced a confident and completely wrong conclusion that Instagram never shows reels in the grid.
Deciding *what is a reel* has no good answer for most archives. Instagram's own marker is `product_type` on the post's GraphQL node (`clips` = reel, `feed` = ordinary feed video, `igtv`, `story`) — `__typename` is `GraphVideo` for all three, and aspect ratio does not separate them either. But:
- Only Instaloader archives carry that metadata, and only newer captures. A survey of `gibiofficial` found `product_type` on 1101 of 5919 sidecars, and just **2** posts marked `clips`.
- Only Instaloader archives carry that metadata, and only newer captures. A survey of `hazelofficial` found `product_type` on 1101 of 5919 sidecars, and just **2** posts marked `clips`.
- JDownloader archives carry none at all — media plus a `.txt` holding the bare caption.
So the viewer believes a `- reels` sidecar directory when one exists, and otherwise falls back to treating a lone video as a reel. **The fallback is a guess**: it cannot tell a reel from a feed video or an old IGTV upload, and it misses videos inside carousels.
-6
View File
@@ -1,6 +0,0 @@
https://www.instagram.com/0ct0ber19/
https://www.instagram.com/kimxxlip/
https://www.instagram.com/withaseul/
https://www.instagram.com/cher_ryppo/
https://www.instagram.com/zindoriyam/
https://www.instagram.com/official_artms/
-605
View File
@@ -1,605 +0,0 @@
# gallery-dl — a CLI replacement for JDownloader2
Status: **in production.** All six ARTMS profiles are synced with
`scripts/gdl-sync.py`; JD2 is no longer used for them.
Everything below was measured against the live site and the real archive on
2026-08-16 and 2026-08-20, not inferred from documentation.
## Why gallery-dl and not a hand-rolled script
The hard parts of fetching Instagram are pagination, cookie handling, CDN URL
expiry and resumption. gallery-dl already has all of them, plus extractors that
map 1:1 onto our sidecar directory layout (`posts`, `reels`, `stories`,
`highlights`). Rolling our own would mean reimplementing the ban-sensitive part
by hand.
## The account was suspended on 2026-08-17 — read this first
The account used for all of the below was suspended the same day this tooling
was built, for "activity that doesn't follow our Community Standards on spam".
The fetching was not the expensive part. **Verification was.**
**It was restored, and synced normally again on 2026-08-20** — a full run
across all six profiles with 0 failures and 0 CDN 429s. That is not evidence
the limits were imagined; it is one data point on a restored account that has
been treated carefully since. Everything below still applies, and the budget is
still per session rather than per command.
What was actually spent against `instagram.com` in a few hours, from one
session and one IP:
| activity | rough requests | downloaded |
|---|---:|---|
| enumerating a profile grid by scrolling it in an automated browser | ~18 pages | nothing |
| the same profile again, after a bug in the scraping selector | ~18 pages | nothing |
| a Reels tab enumerated the same way | ~9 pages | nothing |
| full `-j` metadata dumps of one profile, twice | ~16 pages | nothing |
| `--simulate` runs over the same profile, three times | ~24 pages | nothing |
| single-post `/p/<code>/` fetches while testing filename formats | ~8 | a handful |
| an aborted sync that re-ran every listing pass before dying | ~40 pages | ~270 MB |
| the real sync, 24 sources across 6 profiles | ~150 pages | 2.2 GB |
The two rows that actually mattered to the archive are the last one and part of
the second-to-last. **Everything above them produced no files at all**, and
together they were a comparable number of requests.
The warnings arrived in this order and were each rationalised:
1. `429 Too Many Requests` from `scontent-*.cdninstagram.com`, losing two
videos. Treated as a pacing problem — pacing was lowered and the run
continued.
2. `400 Bad Request` from `/api/v1/highlights/<id>/highlights_tray/`, on an
endpoint that had worked hours earlier. Correctly read as a possible block;
requests stopped.
3. Suspension.
**Treat the first CDN 429 as a stop signal for the session, not a tuning
parameter.** It is the tolerant surface complaining; if that surface is
complaining, the rate-limited one has been unhappy for a while.
### Rules that follow from this
- **Count verification requests against the same budget as fetching.** A
`--simulate`, a `-j` dump and a browser scroll all hit `instagram.com` and
download nothing. Being read-only does not make them free; it makes them
invisible, which is worse.
- **Never enumerate the live site with an automated browser.** Scrolling a
214-post grid is ~18 paginated GraphQL loads at machine speed with no dwell
time between them. It is the most obviously non-human thing in this whole
document, and it was done here twice on one profile.
- **Verify against the archive, not against Instagram.** Every naming, dating
and classification question answered in this file could have been answered
from files already on disk plus a single listing pass.
- **`probe_live` is not cached, so every restart re-enumerates everything.**
The aborted run cost a full duplicate set of listing passes for five
profiles. Cache probe output to disk before running anything twice.
- **Budget per session, not per command.** Nothing in the tooling knows what
the last command spent.
### For a replacement account
- Let it exist and be used normally for a while before pointing any tool at it.
- Keep the cookie on one machine and one public IP, as before.
- Start with a single small profile and stop for the day afterwards.
- Prefer Instagram's own "Download a copy" export where possible: it is
first-party, costs no scraping requests, and carries the metadata this whole
document works around not having.
## The safety model — read this before changing any option
The ban vector is **requests to `instagram.com`**, not bandwidth. See
`docs/jdownloader.md` for the history; Instaloader got this account banned by
asking `instagram.com` a question *per post*.
gallery-dl has two API backends and the difference is exactly that vector:
```python
if self.config("api") == "graphql":
self.api = InstagramGraphqlAPI(self) # per-post api.media() for every
else: # video and every carousel
self.api = InstagramRestAPI(self) # <- default, listing-only
```
The REST backend paginates at `count: 30` (feed) / `page_size: 50` (clips), and
those responses already carry `carousel_media`, `image_versions2`,
`video_versions` and `product_type`. **No per-post request.** A 300-post
profile costs roughly 10 requests to `instagram.com`.
Rules, in order of importance:
1. **`"api": "rest"` always.** Never `graphql`. This is the whole ballgame.
2. **Never enable `metadata`-style options that trigger extra calls.** If a
field is not already in the listing response, it is not worth a request.
3. **Pace it.** `"sleep-request": [4.0, 7.0]` — a randomised gap, not a fixed
one. Also `"sleep": [1.0, 3.0]` between downloads.
4. **Cap the download rate** (`downloader.http.rate`) so the CDN side looks like
a person, not a mirror.
5. **Run from the same public IP as the browser the cookie came from.** At time
of writing that is `mattellite` (`66.23.52.196`); the dev workstation is a
*different* public IP and using the cookie from there is precisely what
session-hijack detection looks for.
6. **No programmatic login, ever.** gallery-dl's username/password path is
disabled upstream anyway; use `--cookies-from-browser`.
Do not add proxy rotation, fingerprint spoofing or account rotation. Throttling
and request-avoidance are welcome; evasion is not.
### Cookies
The logged-in Chrome on `mattellite` runs with a non-default profile:
```
--user-data-dir=/home/matt/.config/google-chrome-devtools
```
so the cookie flag is:
```
--cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools"
```
Plain `--cookies-from-browser chrome` fails with "Unable to find chrome cookies
database" because it looks in `~/.config/google-chrome/`.
Anonymous access is **not** a viable fallback: it serves lower-resolution media,
caps profile pagination at 12 posts, and returns `AuthRequired` for stories and
highlights.
## Output format
The viewer's parser is the contract, not JD2's exact bytes. `EXPORT_RE` in
`src/lib/archive-patterns.ts` accepts all of these, and normalises the index
with `parseInt`, so **JD2 and gallery-dl naming interoperate**:
```
"… - CrORBIcJJbM.mp4" -> postId=CrORBIcJJbM index=1
"… - CrORBIcJJbM - 1.mp4" -> postId=CrORBIcJJbM index=1
"… - C53YPQzp7Wj - 09.jpg" -> postId=C53YPQzp7Wj index=9
```
That means zero-padding and the presence/absence of ` - N` on single-media posts
are cosmetic. Don't spend effort forcing them.
### Directory layout
| kind | directory | note |
|---|---|---|
| posts | `<user>` | |
| reels | `<user> - reels` | |
| stories | `story - <user>` | |
| highlights | `story highlights - <user> - <title>` | |
**Force the directory with `-D`; never use `{username}` for it.** A profile's
reels tab returns *collab reels owned by other accounts*`/0ct0ber19/reels/`
served 6 reels owned by `official_artms` and 1 by `chuuo3o`. With
`{username}` those would scatter into `official_artms - reels/`. JD2 got this
right and the archive proves it: `chuuo3o` and `official_artms` filenames sit
inside `0ct0ber19 - reels/`.
So: **owner in the filename, crawl scope in the directory.**
### Filenames
```jsonc
"filename": {
"sidecar_shortcode and count >= 10":
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num:02}.{extension}",
"sidecar_shortcode":
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num}.{extension}",
"":
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.{extension}"
}
```
`sidecar_shortcode` is set only when the post is a carousel, so it is the
carousel discriminator. Conditions are evaluated in order, first match wins
(`path.py:265`).
Stories and highlights use the per-item `{shortcode}`, not `{post_shortcode}`
(which is the *reel's* id, shared by every item in it):
```
"{date:Olocal/%Y-%m-%d}_{username} - {shortcode}.{extension}"
```
`{date}` on a story/highlight file is the **per-item** `taken_at`
(`instagram.py:337` prefers `item["taken_at"]`), verified on a 154-item
highlight whose items carried distinct times while `post_date` stayed pinned to
the reel. Highlights therefore gain real dates — today they fall back to
directory mtime.
### The timezone is not UTC
JD2 stamped filenames in **desktop local time (US Eastern)**. Measured across
212 comparable posts:
| model | mismatches |
|---|---:|
| UTC | 19 |
| UTC5 (EST) | 10 |
| UTC4 (EDT) | **0** |
| America/New_York (DST-aware) | **0** |
`{date:Olocal/%Y-%m-%d}` uses the machine's local zone with per-timestamp DST
awareness, which reproduces it — `mattellite` is `America/Toronto`, the same
offsets. Note the **trailing `/` must be omitted**: `Olocal/%Y-%m-%d/` puts the
separator into the strftime format and it sanitises to an underscore, giving
`2026-08-15__0ct0ber19`.
If the sync ever moves to a host in another timezone, set an explicit
`{date:O-4/…}` or the dates will silently shift for ~9% of posts.
### Caption sidecars
JD2 writes one `.txt` per post, named without the index, containing the caption
with **no trailing newline**, and writes nothing when the caption is empty
(measured: 197 of 217 posts, 86 of 86 reels, 0 of 10 stories, 0 of 16
highlights). gallery-dl reproduces this exactly with the default
`"empty": false`:
```jsonc
{ "name": "metadata", "event": "post", "mode": "custom",
"content-format": "{description}", "extension": "txt",
"filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.txt" }
```
`"event": "post"` is what makes it one file per post rather than per media file.
### Metadata sidecar (new — JD2 had no equivalent)
```jsonc
{ "name": "metadata", "event": "post", "mode": "json",
"filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.json",
"include": ["post_shortcode","post_id","type","date","post_date","username",
"fullname","owner_id","description","count","likes","post_url",
"sidecar_shortcode"] }
```
Use **`include`**, not `fields``fields` is for `mode: custom` and silently
does nothing here, leaving `audio_user` blobs (including another user's profile
picture URL) in the output.
The payoff is `type`, which is Instagram's own classification:
```json
{ "post_shortcode": "DbdG9L9jU4m", "type": "post", "count": 2 } // feed video
{ "post_shortcode": "Db-lNCoib9m", "type": "reel", "count": 1 } // real reel
```
This is the `product_type: "clips"` signal, delivered free in the listing
response. It is the authoritative answer to "is this a reel", and would let the
viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
"Scanner work" below.
**`type` is only populated by listing extractors.** Extracting a single
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
only matters when testing by hand.
## Cadence, and the budget that enforces it
**Monthly for everything, daily for stories only.** Stories expire in 24h and
cannot be backfilled, so they are the one surface where missing a day means
losing the content permanently. Everything else can wait — the skip-archive
means an infrequent full sync costs barely more than a frequent one, because it
only fetches what is new.
```
# monthly, everything
gdl-sync.py --index <viewer-url> --staging ~/gdl/staging \
--publish <user>@<nas>:<archives> --archive-db ~/gdl/artms.db \
--urls-file artms_account_links.txt --execute
# daily, stories only -- one request per profile
gdl-sync.py ... --only stories --execute
```
A stories-only run is one source per profile and **never seeds**, because a
story cannot be in the archive before it is fetched; probing would double the
cost of the cheapest surface for no benefit. Six profiles is a handful of
requests.
When scheduling it, **randomise the minute and avoid the hour boundary**. A job
that fires at exactly 09:00 every day is a machine; one that fires somewhere in
a window looks like someone opening the app.
The tool now refuses to repeat itself:
| flag | default | what it prevents |
|---|---|---|
| `--min-interval` | 20h | re-fetching a source touched recently — the aborted-restart case that re-enumerated five profiles |
| `--probe-ttl` | 24h | paying for a listing pass twice within a run cycle |
| `--max-sources` | off | a runaway list touching more than intended |
| `--force` | off | (escape hatch: ignores both guards) |
State lives beside the archive DB as `<db>.state.json`, recording per source
when it was seeded and last fetched. **Seeding is a one-time bootstrap**: after
the first successful sync the archive DB records everything gallery-dl has
seen, so the source is never probed again. That is the single biggest saving
here — a second full sync costs roughly half what the first did.
## Incremental sync — why the fetch host needs no copy of the archive
gallery-dl can skip already-held media two ways, and the difference decides
whether the fetcher needs the archive mounted:
- **By file existence** (default). Needs the destination to already contain the
files, so it only works if the archive is mounted where gallery-dl writes.
- **By skip-archive** (`--download-archive`). A sqlite DB of ids. Needs nothing
on disk.
We use the second, so the fetch host can write to **local disk and rsync
afterwards**. That avoids writing tens of thousands of small files over CIFS,
and keeps a mid-sync failure from leaving partial files on the live Resilio
share.
The key is `archive_prefix + archive_fmt`, which for this extractor is the
literal `instagram` plus the per-media numeric pk (`instagram.py:25`,
`job.py:713-719`). Verified: a 3-image carousel produced
```
instagram3079387627521318672
instagram3079387627521429433
instagram3079387627529716672
```
and a second run skipped every media file, rewriting only the idempotent
`.txt`/`.json` sidecars.
**Seeding.** `media_id` is not in our filenames, so the DB cannot be built from
names alone — but one listing pass (the pass we make anyway) maps every live
item to its `media_id`, and the archive's *file listing* says which we already
hold. No extra Instagram requests, and no archive content — a listing is
enough, which `GET /api/archives/:name/files` already serves.
Measured on `0ct0ber19`: 2275 live media items, 2248 seeded from the existing
listing, **27 left to download** — precisely the media of the two posts added
since the last crawl.
The one trap, which silently seeds almost nothing if you get it backwards:
| surface | filed under | why |
|---|---|---|
| posts, reels | `post_shortcode` | carousel children each have their own `shortcode`, which never appears in a filename |
| stories, highlights | `shortcode` (per item) | `post_shortcode` is the containing reel's id, shared by every item |
`live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
### The skip-archive saves the CDN, not `instagram.com`
Worth being exact about, because the two costs land on different surfaces and
only one of them bans accounts:
| what | which surface | scales with |
|---|---|---|
| downloading media | `scontent-*.cdninstagram.com` | how much is **new** |
| enumerating the profile to find it | `instagram.com` | how **big** the profile is |
The skip-archive suppresses the first. It does nothing about the second, so a
2275-post profile costs ~76 pages of pagination every run, forever, whether it
has three new posts or none. Seeding (above) saved a *second* full pass, not
the first.
Measured on the 2026-08-20 run, from sidecar write times in staging — free,
since the run was paying for the listing anyway:
```
1787248852 2026-08-19 … DcOeoVxkthi new, +0s
1787248944 2026-08-18 … DcLpfoJCZtp new, +92s
1787249058 2026-08-17 … DcIlGbxCUk0 new, +114s
1787249162 2026-07-24 … DbKr1TxlPSX ┐ all one second: nothing
1787249162 2026-08-15 … DcD-FdBCYGm ┘ downloaded, sidecars only
```
Three posts took ~100s each; the remaining 2272 were enumeration with nothing
to show for it.
**Pinned posts do not break early abort.** Test case 16 previously claimed
`0ct0ber19` returns its 3 pinned posts out of date order — that is true of the
*web grid*, but the REST `/posts/` listing came back strictly
reverse-chronological, newest first, no hoisting. That matters because
front-loaded old posts are the one thing that would make `skip: abort:N`
dangerous: it would trip on them and abort before reaching anything new.
So `skip: abort:N` is viable, and cuts ~420 requests per run to ~40-60:
| surface | live items | pages | with `abort:50` |
|---|---:|---:|---:|
| posts, 6 profiles | 11,248 | ~377 | ~12 |
| reels, 6 profiles | 1,080 | ~24 | ~8 |
| stories + highlights | — | ~20 | ~20 |
N counts consecutive skipped **files**, not posts, so it must clear the largest
already-held carousel — `DcD-FdBCYGm` alone is 22 media. 50 is comfortable; 5
would not be.
**The tradeoff is edited carousels.** Test case 15 is a post that gained items
after we archived it, and only a full enumeration finds those. Suggested
policy: `abort:50` for routine runs, a full sweep occasionally.
Measured the same day, resuming a stopped run with `--abort 50`:
| source | live items | enumerated |
|---|---:|---:|
| `cher_ryppo` posts | 2,151 | **7** |
| `cher_ryppo` reels | 92 | 53 |
One page instead of 72, and every new post was still caught. The 7 is roughly
3 new posts plus 4 already-held carousels making up the 50 skipped files.
Reels need 53 because they are single-media, so 50 consecutive skips really is
50 reels — another reminder that N counts files, and that the same N behaves
very differently on a carousel-heavy surface than on a reels tab.
## Publishing
The fetch host stages to local disk and rsyncs afterwards. `rsync
--ignore-existing` is not an optimisation but the safety property: the archive
deliberately outlives Instagram, so publishing must only ever **add**. No
`--delete`, and nothing already present is overwritten — including sidecars,
which are rewritten every run and would otherwise churn the synced share.
Publishing happens once at the end of a run, so a profile that fails midway
never reaches the archive half-written.
## Status
In use for all six ARTMS profiles.
`withaseul` first — 322 files added (74 media, 241 `.json`, 7 `.txt`), nothing
overwritten or deleted. Of the 74 new media, **zero** duplicated media already
held under a different name, which is the check that says JD2 and gallery-dl
naming really do converge.
**2026-08-20**, the first full incremental sync, four days after the previous
one. 184 new media, 299 files published, 0 failures and **0 CDN 429s**:
| profile | posts | reels | stories | files added |
|---|---:|---:|---:|---:|
| 0ct0ber19 | 58 | 2 | 4 | +77 |
| official_artms | 12 | — | 2 | +85 |
| cher_ryppo | 41 | 1 | 8 | +63 |
| zindoriyam | 23 | — | 4 | +35 |
| kimxxlip | 16 | — | 2 | +23 |
| withaseul | 10 | — | — | +16 |
The 20 story items are the part that could not have been recovered later.
Two things made it cheap, and both are worth keeping:
- The archive DB was already seeded from the previous run, so `--min-interval`
and the recorded `seeded` state meant **no probe passes at all**. A state
file has to exist for this; if one is missing after a manual run, write it
rather than letting the tool re-seed 24 sources.
- `--abort 50` (see above) cut the remaining listing cost by roughly 85%.
The run was deliberately **stopped and resumed** halfway to pick up `--abort`.
That is safe precisely because of the state file: the 12 finished sources were
already marked `fetched`, so the 20h floor skipped them and only the remaining
12 re-ran. Stopping a run is cheap now; it was not before.
Published files land owned by the SSH user rather than `rslsync`. The viewer
reads them fine (world-readable), but Resilio does not own what it syncs; worth
a `chown` if that ever matters. This also makes **`rsync` exit 23**
("some files/attrs were not transferred") the *normal* outcome of a publish —
it is the failed `chown`, not lost data. Confirm by re-running the same rsync
with `--dry-run`: an empty file list means everything arrived.
The profiles to fetch live in `artms_account_links.txt` at the archive root,
passed with `--urls-file`.
## Verified run
`withaseul`, all four surfaces, staged locally and published to a scratch
directory before the live publish above:
```
==> withaseul / posts seeded 915 of 984 live items
==> withaseul / reels seeded 28 of 34 live items
==> withaseul / stories no results (none active)
==> withaseul / highlights no results
```
Output landed correctly, including the collab-reel case — `withaseul - reels`
contains 53 files owned by `withaseul`, 10 by `cher_ryppo`, 3 by `0ct0ber19`
and 2 by `official_artms`, all with the owner in the filename and the crawl
scope as the directory.
### The CDN rate-limits, and the first run tripped it
At `rate: 3M` with `sleep: [1.0, 3.0]`, `scontent-*.cdninstagram.com` returned
**`429 Too Many Requests`** and two videos were lost (gallery-dl retried, then
gave up with exit 4). This is the *tolerant* surface complaining, which is a
clear signal the pacing was too aggressive.
Defaults are now:
| option | value |
|---|---|
| `--rate` | `1M` |
| `--sleep-request` | 610 s |
| `--sleep` | 36 s |
| `sleep-429` | 120 s |
| `retries` (extractor and downloader) | 8 |
Re-running with those recovered both videos and produced **0 failures and 0
429s**. Do not raise them for speed; an archive sync has no deadline.
### yt-dlp is worth installing
Without it, gallery-dl logs `Cannot import yt-dlp or youtube-dl` and falls back
to a progressive URL for DASH videos. The fallback mostly works but is what the
429s hit hardest.
**`pipx install yt-dlp` does not work** — it was the advice here until
2026-08-20, and it is wrong. It gives yt-dlp its own venv, so the binary lands
on `PATH` while gallery-dl, in a *different* venv, still cannot `import yt_dlp`.
The symptom is that everything looks installed and the log keeps saying
`Cannot import yt-dlp`. gallery-dl needs it importable, not runnable:
```sh
pipx inject gallery-dl yt-dlp
```
Verify by asking gallery-dl's own interpreter, not the shell:
```sh
/home/matt/.local/share/pipx/venvs/gallery-dl/bin/python -c 'import yt_dlp'
```
## Known quirks
- **`count` is not the emitted file count.** For 135 of 214 posts it was exactly
one higher than the number of files written. This makes the `count >= 10`
padding condition mis-pad a handful of 9-item posts (10 of 214 measured). Since
the parser normalises the index, this is cosmetic — but it means a re-fetch
over an existing JD2 tree writes `- 01.jpg` beside an existing `- 1.jpg`.
- **Carousels get edited.** Two posts had a different media count live than on
disk. Padding width follows the count *at download time*, so a grown carousel
produces mixed widths — the archive already contains one such post from JD2.
- **Highlights already have two naming styles on disk**, and every undated file
has a dated twin. The scanner dedupes by index so they render once; it is
wasted disk, not a display bug.
## Scanner work (not done yet)
`useArchiveScanner` currently treats any `.json` in the tree as a possible
manifest. Adding gallery-dl sidecars needs it to distinguish three things:
1. Instagram export manifests (`posts_1.json`) — existing path.
2. Instaloader `.json.xz` — existing path, GraphQL node shape.
3. gallery-dl `.json` — new, flat shape, identified by having
`post_shortcode` + `type` at the top level.
Once (3) is read, `source`/`isStory` and the reel flag should come from `type`
rather than from the directory and the lone-video heuristic.
## Test cases
Real subjects, all present in the archive today. See
`scripts/gdl-sync.py --selftest` for the harness.
| # | case | shortcode | expected |
|---|---|---|---|
| 1 | single image | `CwcXnQhOqFG` | one `.jpg`, no index |
| 2 | single feed video | `DbdG9L9jU4m` | one `.mp4`, `type: post` |
| 3 | carousel, images only | `Cq8LrxSJAJE` | `- 1 … - 3` |
| 4 | carousel, image + video | `CtohvHxLnWO` | `- 1.jpg … - 4.mp4`, **no `.txt`** |
| 5 | carousel of exactly 9 | `Cv2Hb_brx_N` | 1-digit index |
| 6 | carousel of 10+ | `CzM8Uf6B6H_` | 2-digit index `- 01 … - 10` |
| 7 | reel shown on the posts grid | `C8FHM6EJl15` | in `<user>`, `type: reel` |
| 8 | reel on the reels tab | `Db-lNCoib9m` | in `<user> - reels`, `type: reel` |
| 9 | collab reel (other owner) | `DYcZOb0h6Sv` | dir `0ct0ber19 - reels`, filename `chuuo3o` |
| 10 | story | live only | `story - <user>`, per-item shortcode + date |
| 11 | story highlight | `C-IImhvpFuk` | `story highlights - <user> - <title>` |
| 12 | highlight, unicode title | `Drawheeing` | trailing U+2800 preserved in dirname |
| 13 | empty caption | `CrdsY5CrSsO` | media written, `.txt` absent |
| 14 | deleted post | `C0TgI7sphfZ` | on disk, absent live — must not be removed |
| 15 | edited carousel | `C7zG7-jJMlq` | 18 on disk, 8 live — must not be removed |
| 16 | pinned posts | `0ct0ber19` | REST listing is strictly reverse-chronological; see below |
| 17 | profile avatar | `0ct0ber19.jpg` | base dir, undated |
Cases 1416 are reconciliation, not naming: **a sync must never delete**, since
the archive deliberately outlives Instagram.
Not covered, decide before relying on them: the `/reposts/` tab (`0ct0ber19`
has one) and `/tagged/`. Neither is fetched today.
-181
View File
@@ -1,181 +0,0 @@
# JDownloader2 — archive fetching quick reference
How content gets into this archive, and why the setup is shaped the way it is.
## Why JDownloader and not Instaloader
There are two surfaces, and they're treated very differently:
| Surface | What hits it | Risk |
|---|---|---|
| `instagram.com` | profile pages, GraphQL/API metadata | Tied to your session, heavily rate-limited. **This is where bans come from.** |
| `scontent*.cdninstagram.com` | the actual media | Signed URLs, CDN-served, tolerant. Mostly a bandwidth question. |
JDownloader does nearly all its work on the CDN. Instaloader's value — the rich
`.json.xz` metadata — comes from asking `instagram.com` a question *per post*.
Concretely, from this archive: `rivvsofficial` has 188 post-metadata files, so
backfilling it cost 188 API requests for one 605-file profile. That's the ban
vector. Downloading the 238 photos was never the problem.
Instaloader got this account banned once. JDownloader with throttling did not.
> **The account was suspended anyway, on 2026-08-17, for "spam".** Not by
> JDownloader, and not by downloading. It was suspended during a day of
> *building and verifying* the gallery-dl replacement — automated browser
> scrolling to enumerate profile grids, repeated `--simulate` and `-j` metadata
> passes, and one aborted sync that re-ran every listing pass before dying.
>
> The framing above is right about which surface is dangerous and wrong about
> what reaches it. **Every read of `instagram.com` counts, including the ones
> that download nothing** — and read-only work is easy not to count precisely
> because it leaves no files behind. See the post-mortem at the top of
> `docs/gallery-dl.md`.
>
> The rule that would have prevented it: *verify against the archive, never
> against the live site*, and treat the first CDN `429` as the end of the
> session rather than a pacing knob.
### What the metadata gap actually costs
Comparing a JDownloader profile against an Instaloader one:
| | JDownloader | Instaloader |
|---|---|---|
| Media | ✅ | ✅ |
| Captions (`.txt`) | ✅ | ✅ |
| Dates (from filenames) | ✅ | ✅ |
| Bio / full name | ❌ | ✅ |
| Follower counts | ❌ | ✅ |
| External URL | ❌ | ✅ |
Captions already work — the viewer reads the `.txt` sidecars. Everything missing
lives in a *single* profile-level record, not the per-post ones. That's why
JDownloader-sourced profiles show "0 followers" and a placeholder bio.
Not worth extra requests. If you ever want it, the zero-request option is a
hand-written `profile.json` sidecar (not implemented yet — ask).
## Settings that matter
**Chunks per download → 1.** The single most important one. JDownloader splits
each file into multiple ranged requests by default; that `Range` pattern looks
nothing like a browser or the app. One chunk = one sequential GET per file.
`jd2-sync` sets `chunks=1` per job, so no global change is needed — but set it
globally too if you ever add links by hand.
**Max simultaneous downloads → 23**, connections-per-host low. Concurrency is
what turns "a user" into a statistic.
**Leave reconnect / IP-change features off.** A mid-session IP change on a live
cookie is a *stronger* anomaly signal than the request rate you'd be avoiding.
## The cookie
Exported manually from a real browser session. This is the right approach — no
programmatic login anywhere, which is the thing that actually gets flagged.
- Use it from the **same public IP** as the browser it came from. A cookie used
from a different network is what session-hijack detection looks for.
- When it expires, **re-export from the browser**. Never add a login step to a tool.
- It's a full account credential. Keep it off the NAS share and out of the repo.
## Workflow
Two URLs per profile, because the profile grid misses some reels:
```
https://www.instagram.com/<user>/
https://www.instagram.com/<user>/reels/
```
They overlap slightly — a reel caught by both lands in each directory and shows
up twice in the viewer. That's correct and matches Instagram, which also shows
reels in the profile grid *and* the Reels tab.
## Generating jobs
Instead of pasting URLs and setting output folders by hand:
```bash
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives --dry-run
```
Review, then write it into JDownloader's folder-watch directory:
```bash
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives \
--out ~/.jd2/folderwatch
```
JDownloader runs on the desktop while the archive lives on the NAS, so tell it
the path *it* sees:
```bash
npm run jd2 -- --archives /mnt/nas/Instagram-archive/archives \
--download-base 'Z:\Instagram-archive\archives' \
--out ~/.jd2/folderwatch
```
| Flag | Purpose |
|---|---|
| `--archives <dir>` | Archive root to scan (or `$ARCHIVES_DIR`) |
| `--out <dir>` | JDownloader folder-watch directory |
| `--download-base <dir>` | Root path as JDownloader sees it (Windows paths fine) |
| `--user <name>` | Just this profile (repeatable) |
| `--skip <name>` | Never emit jobs for this directory (repeatable) |
| `--chunks <n>` | Connections per file (default 1) |
| `--auto-start` | Start immediately instead of parking in LinkGrabber |
| `--all-reels` | Emit a reels job even where no reels directory exists |
| `--dry-run` | Print instead of writing |
Defaults are deliberately conservative: `chunks=1`, and links park in the
LinkGrabber for review rather than auto-starting.
Only posts and reels are emitted. Highlight URLs need a numeric id and story
URLs expire, so those stay manual.
Directories that aren't Instagram profiles are skipped by username shape
(letters, digits, dots, underscores, ≤30 chars) — pointing a crawl at those
spends `instagram.com` requests to be told the profile doesn't exist. For names
that *look* like usernames but aren't, use `--skip` or a `.jd2ignore` file in
the archive root, one name per line.
Format reference: `src/org/jdownloader/extensions/folderwatchV2/explain.txt`.
JDownloader develops on SVN — read it via the daily mirror at
<https://github.com/mycodedoesnotcompile2/jdownloader_mirror> (`svn_trunk/`),
not one of the abandoned GitHub copies.
## Expected layout
Everything downloads into `<archives>/`, one directory per source:
```
archives/
0ct0ber19/ posts
0ct0ber19 - reels/ reels
story - 0ct0ber19/ stories
story highlights - 0ct0ber19 - Heestory/ a highlight
```
Non-archive directories (tool output, exports from elsewhere) live *outside*
`archives/` so they never reach the viewer.
The server picks up changes automatically — its index is keyed on directory
mtime, so a new file invalidates only that directory.
## If something goes wrong
**429 / rate limited** — stop for hours, not seconds. Retrying into a limit is
what converts a soft throttle into something worse.
**Cookie stops working** — re-export from the browser. Don't add a login step.
**Files land in the wrong folder** — a Packagizer rule is overriding the job.
Generated jobs set `overwritePackagizerEnabled=TRUE` to prevent this; check that
rules aren't set to run after it.
**Viewer doesn't show new posts** — check the file is in the right directory and
matches the naming pattern (`YYYY-MM-DD_<user> - <shortcode>[ - NN].<ext>`).
The index refreshes on directory mtime, so a genuinely new file is picked up on
the next request.
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "instaarchive-viewer",
"version": "1.8.0",
"version": "1.8.1",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "instaarchive-viewer",
"version": "1.8.0",
"version": "1.8.1",
"dependencies": {
"@tailwindcss/vite": "^4.1.14",
"@vitejs/plugin-react": "^5.0.4",
+3 -4
View File
@@ -1,19 +1,18 @@
{
"name": "instaarchive-viewer",
"private": true,
"version": "1.8.0",
"version": "1.8.1",
"type": "module",
"scripts": {
"dev": "vite --port=3000 --host=0.0.0.0",
"build": "vite build && npm run build:server",
"build:server": "tsc server.ts --esModuleInterop --module ESNext --target ES2022 --moduleResolution bundler --outDir dist-server",
"build:server": "tsc server.ts --esModuleInterop --module ESNext --target ES2022 --moduleResolution bundler --removeComments --outDir dist-server",
"preview": "vite preview",
"server": "tsx server.ts",
"clean": "rm -rf dist",
"lint": "tsc --noEmit",
"test": "vitest run",
"test:watch": "vitest",
"jd2": "tsx scripts/jd2-sync.ts"
"test:watch": "vitest"
},
"dependencies": {
"@tailwindcss/vite": "^4.1.14",
Binary file not shown.
Binary file not shown.
-843
View File
@@ -1,843 +0,0 @@
#!/usr/bin/env python3
"""
Fetch Instagram profiles into the archive layout using gallery-dl.
The CLI replacement for the JDownloader2 workflow. See docs/gallery-dl.md for
the measurements behind every choice here — especially the safety model, which
is the reason this script exists in this shape rather than a simpler one.
The fetch host needs no copy of the archive. It stages locally and rsyncs
afterwards; what it already holds is learned from a *file listing* alone
(`--index`), which the viewer's own API serves.
Usage:
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \\
--staging /var/tmp/gdl --publish user@host:/path/to/archives \\
--urls-file artms_account_links.txt --dry-run
# ...then swap --dry-run for --execute. --index also accepts a local path,
# and --profile / --all work instead of --urls-file.
Always --dry-run first: it prints the plan, and the publish step it reports is
the one that would touch the archive.
Run it from the host whose public IP matches the browser the cookie came from;
using the cookie from elsewhere is what session-hijack detection looks for.
"""
from __future__ import annotations
import argparse
import datetime as dt
import json
import os
import re
import shutil
import subprocess
import sys
from dataclasses import dataclass, field
from pathlib import Path
# --------------------------------------------------------------------------
# Archive layout
# --------------------------------------------------------------------------
# Mirrors src/lib/archive-grouping.ts. Instagram usernames cannot contain
# spaces, which is what makes the username separable from a highlight title.
RE_HIGHLIGHT = re.compile(r"^story highlights - ([^ ]+) - (.+)$")
RE_STORIES = re.compile(r"^story - ([^ ]+)$")
RE_REELS = re.compile(r"^([^ ]+) - reels$")
DATE_FMT = "{date:Olocal/%Y-%m-%d}"
"""Local-time date. JD2 stamped US Eastern, NOT UTC (0/212 mismatches vs 19 for
UTC). `Olocal` is DST-aware per timestamp. The trailing separator must be
omitted or it lands in the strftime format and sanitises to an underscore."""
POST_STEM = DATE_FMT + "_{username} - {post_shortcode}"
ITEM_STEM = DATE_FMT + "_{username} - {shortcode}"
@dataclass
class Source:
"""One gallery-dl invocation: a URL fetched into a specific directory."""
kind: str # posts | reels | stories | highlights
url: str
directory: str # relative to the archives root
subcategory: str # gallery-dl config key
title: str | None = None # highlight title, when known
@dataclass
class Profile:
user: str
existing: dict[str, str] = field(default_factory=dict) # kind -> dirname
def sources(self, kinds: set[str]) -> list[Source]:
u = self.user
base = f"https://www.instagram.com/{u}"
all_sources = [
Source("posts", f"{base}/posts/", u, "posts"),
Source("reels", f"{base}/reels/", f"{u} - reels", "reels"),
# Stories expire after 24h, so these can only ever be captured
# live. There is no backfill and no re-fetch -- which is why they
# are the one surface worth visiting daily.
Source("stories", f"https://www.instagram.com/stories/{u}/",
f"story - {u}", "stories"),
# Highlight directories embed the title, which gallery-dl only
# learns mid-extraction -- so this one source fans out into many
# directories and is handled with a directory format string.
Source("highlights", f"{base}/highlights", "", "highlights"),
]
return [s for s in all_sources if s.kind in kinds]
def scan_archives(root: Path) -> dict[str, Profile]:
"""Group existing directories into profiles, as the server does."""
profiles: dict[str, Profile] = {}
def get(user: str) -> Profile:
return profiles.setdefault(user, Profile(user))
for entry in sorted(os.listdir(root)):
if not (root / entry).is_dir() or entry.startswith("."):
continue
if m := RE_HIGHLIGHT.match(entry):
get(m.group(1)).existing.setdefault("highlights", entry)
elif m := RE_STORIES.match(entry):
get(m.group(1)).existing["stories"] = entry
elif m := RE_REELS.match(entry):
get(m.group(1)).existing["reels"] = entry
else:
get(entry).existing["posts"] = entry
return profiles
RE_PROFILE_URL = re.compile(
r"^(?:https?://)?(?:www\.)?instagram\.com/(?P<user>[^/?#\s]+)/?", re.I)
# Path segments that are Instagram features, not profiles. A line like
# ".../p/ABC123/" names a post, and treating "p" as a username would silently
# sync nothing under a nonsense directory.
RESERVED_SEGMENTS = {
"p", "reel", "reels", "stories", "explore", "accounts", "direct",
"tv", "s", "invites", "challenge", "about", "developer",
}
def read_urls_file(path: Path) -> list[str]:
"""
Read profile URLs (or bare usernames) from a file, one per line.
Written for hand-maintained lists: blank lines are skipped, `#` starts a
comment, and either a full URL or a bare username works. Order is kept and
duplicates dropped, so a list can be appended to without care.
"""
users: list[str] = []
seen: set[str] = set()
for lineno, raw in enumerate(path.read_text().splitlines(), 1):
line = raw.split("#", 1)[0].strip()
if not line:
continue
m = RE_PROFILE_URL.match(line)
user = m.group("user") if m else line.strip("/")
if not user or "/" in user or " " in user:
print(f"{path}:{lineno}: cannot read a username from {raw.strip()!r}",
file=sys.stderr)
continue
if user.lower() in RESERVED_SEGMENTS:
print(f"{path}:{lineno}: {user!r} is an Instagram path, not a "
f"profile — skipping", file=sys.stderr)
continue
if user in seen:
continue
seen.add(user)
users.append(user)
return users
class ArchiveIndex:
"""
What the archive already holds, as filenames only.
Deliberately never reads file *contents*, so the fetch host does not need a
copy of the archive — it can stage locally and rsync afterwards. Backed
either by a local directory or by the viewer's own API, which already
serves exactly this listing and is the cheaper option when the archive
lives on network storage (a full walk there took ~52s).
"""
def __init__(self, source: str):
self.remote = source.startswith(("http://", "https://"))
self.source = source.rstrip("/") if self.remote else None
self.root = None if self.remote else Path(source)
if self.root and not self.root.is_dir():
raise SystemExit(f"archive index not found: {source}")
self._cache: dict[str, list[str]] = {}
def _get(self, path: str):
from urllib.request import urlopen
with urlopen(f"{self.source}{path}", timeout=60) as resp:
return json.load(resp)
def profiles(self) -> set[str]:
if self.remote:
return {a["name"] for a in self._get("/api/archives")}
return set(scan_archives(self.root))
def listing(self, user: str) -> list[str]:
"""Every filename belonging to a profile, across all its sidecars."""
if user in self._cache:
return self._cache[user]
names: list[str] = []
if self.remote:
try:
data = self._get(f"/api/archives/{user}/files")
except Exception:
data = []
files = data if isinstance(data, list) else data.get("files", [])
names = [f["path"] for f in files]
else:
prof = scan_archives(self.root).get(user)
for dirname in (prof.existing.values() if prof else ()):
d = self.root / dirname
if d.is_dir():
names += [f"{dirname}/{n}" for n in os.listdir(d)]
self._cache[user] = names
return names
# --------------------------------------------------------------------------
# gallery-dl configuration
# --------------------------------------------------------------------------
def build_config(rate: str, sleep_request: list[float],
sleep: list[float], abort: int = 0) -> dict:
"""
The config is generated rather than checked in so the safety-critical
options cannot drift out of sync with the docs.
`api: rest` is the single most important line in this file. The graphql
backend issues one request PER POST for every video and carousel, which is
the pattern that got this account banned once already.
"""
caption_pp = {
"name": "metadata",
"event": "post",
"mode": "custom",
"content-format": "{description}",
"extension": "txt",
# JD2 wrote no .txt when the caption was empty; "empty": false (the
# default) reproduces that.
}
meta_pp = {
"name": "metadata",
"event": "post",
"mode": "json",
# `include`, NOT `fields` -- `fields` applies to mode:custom and
# silently does nothing here, dumping audio_user blobs that contain
# unrelated users' profile picture URLs.
"include": [
"post_shortcode", "post_id", "type", "date", "post_date",
"username", "fullname", "owner_id", "description", "count",
"likes", "post_url", "sidecar_shortcode",
],
}
def post_like(stem: str) -> dict:
"""Naming for surfaces whose unit is a post (posts, reels)."""
skip: dict = {}
if abort:
# Stop enumerating once `abort` consecutive files are already in
# the skip-archive. The listing pass -- not the downloading -- is
# what costs `instagram.com` requests, and it otherwise walks the
# whole profile every run to find three new posts.
#
# Safe here only because the REST listing is strictly
# reverse-chronological: the web grid hoists pinned posts to the
# front, but this endpoint does not (measured 2026-08-20), so old
# posts never appear before new ones.
#
# Counted in FILES, not posts, so it must clear the largest
# already-held carousel -- 22 media for one real post in this
# archive. It also means edited carousels (test case 15) stop
# being noticed, so a full sweep is still worth running
# occasionally.
skip["skip"] = f"abort:{abort}"
return {
**skip,
# `sidecar_shortcode` is set only for carousels, so it is the
# carousel discriminator. First matching condition wins.
"filename": {
"sidecar_shortcode and count >= 10":
stem + " - {num:02}.{extension}",
"sidecar_shortcode":
stem + " - {num}.{extension}",
"":
stem + ".{extension}",
},
"postprocessors": [
{**caption_pp, "filename": stem + ".txt"},
{**meta_pp, "filename": stem + ".json"},
],
}
def item_like(stem: str) -> dict:
"""
Naming for surfaces whose unit is an item inside a reel (stories,
highlights). `{shortcode}` is per item; `{post_shortcode}` is the
reel's id and is shared by every item in it.
The media filename uses the per-item shortcode, but the sidecar cannot:
it runs at `event: post`, where the kwdict describes the *reel* and has
no `shortcode` at all -- which silently formatted as the literal
"None", producing one "<date>_<user> - None.json" per reel. It is keyed
by `post_shortcode` instead, and is genuinely reel-level data (the
reel's own date and item count); per-item dates live in the media
filenames, which is the more precise source anyway.
"""
return {
"filename": stem + ".{extension}",
"postprocessors": [
{**meta_pp,
"filename": DATE_FMT + "_{username} - {post_shortcode}.json"},
],
}
return {
"extractor": {
"base-directory": ".",
"instagram": {
"api": "rest", # never "graphql" -- see docstring
"sleep-request": sleep_request,
"sleep": sleep,
# The CDN does rate-limit: a first run at 3M/1-3s drew
# '429 Too Many Requests' from scontent-*.cdninstagram.com and
# lost two videos. Back off hard rather than retry fast.
"sleep-429": 120.0,
"retries": 8,
"videos": True,
"include": "", # never "all"; sources are explicit
# Directory is forced per-invocation with -D, because a reels
# tab returns collab reels owned by OTHER accounts and
# {username} would scatter them into the wrong profile.
"directory": [],
"posts": post_like(POST_STEM),
"reels": post_like(POST_STEM),
"stories": item_like(ITEM_STEM),
"highlights": {
**item_like(ITEM_STEM),
# The only surface that must derive its own directory,
# since the title is not known until extraction.
"directory": ["story highlights - {username} - {highlight_title}"],
},
},
},
# `retries` here is the CDN-side counterpart to sleep-429 above.
"downloader": {"http": {"rate": rate, "retries": 8}},
"output": {"mode": "null"},
}
# --------------------------------------------------------------------------
# Planning and execution
# --------------------------------------------------------------------------
def gdl_command(src: Source, staging: Path, config: Path, cookies: str,
archive_db: Path | None) -> list[str]:
cmd = [
"gallery-dl",
"--config", str(config),
"--cookies-from-browser", cookies,
]
if archive_db:
# Without a seeded skip-archive, staging is empty and every file is
# re-downloaded; see seed_archive_db.
cmd += ["--download-archive", str(archive_db)]
# Forced destination -- never `{username}` -- because a reels tab returns
# collab reels owned by other accounts, which would otherwise be filed
# under the wrong profile. Highlights are the exception: their directory
# embeds a title only known mid-extraction, so the config formats it.
dest = staging if src.subcategory == "highlights" else staging / src.directory
cmd += ["--destination", str(dest)]
cmd.append(src.url)
return cmd
# gallery-dl keys its skip-archive on `archive_prefix + archive_fmt`, which for
# this extractor is the literal "instagram" followed by the per-media numeric
# pk (`instagram.py:25`, `job.py:713-719`). Verified against a real run: a
# 3-image carousel produced 3 rows, one per item.
ARCHIVE_KEY = "instagram{}".format
ARCHIVE_SCHEMA = "CREATE TABLE IF NOT EXISTS archive (entry TEXT PRIMARY KEY)"
RE_ARCHIVED = re.compile(
r"^(\d{4}-\d{2}-\d{2})_(.+?) - ([A-Za-z0-9_-]+?)(?: - (\d+))?\.(\w+)$")
NON_MEDIA = {"txt", "json"}
def index_existing(listing: list[str]) -> set[tuple[str, int]]:
"""
Reduce a flat list of filenames to the (shortcode, index) pairs already
held. Only names matter — never the bytes — which is what lets the sync run
on a host that has no copy of the archive.
"""
have: set[tuple[str, int]] = set()
for name in listing:
m = RE_ARCHIVED.match(name.rsplit("/", 1)[-1])
if not m or m.group(5).lower() in NON_MEDIA:
continue
# An absent index means a single-media post, which is index 1 — the
# same normalisation the viewer's EXPORT_RE applies.
have.add((m.group(3), int(m.group(4) or 1)))
return have
def live_key(item: dict, kind: str) -> tuple[str, int]:
"""
The (shortcode, index) a live item *would* be filed under, mirroring the
filename template exactly.
The two surfaces disagree about which shortcode identifies a file, and
getting this wrong silently seeds almost nothing:
posts/reels filed under {post_shortcode} — for a carousel, each
child item ALSO has its own `shortcode`, which is not
what appears in the filename.
stories/highlights filed under the per-item {shortcode}, because
`post_shortcode` there is the containing reel's id and
is shared by every item in it.
"""
if kind in ("stories", "highlights"):
return (item.get("shortcode"), 1)
return (item.get("post_shortcode"), item.get("num"))
def seed_archive_db(db: Path, existing: set[tuple[str, int]],
live: list[dict], kind: str) -> int:
"""
Mark everything already held as downloaded, so a fetch into an empty
directory pulls only what is missing.
`live` is the metadata of one listing pass — the pass we have to make
anyway — each entry carrying at least `media_id` plus the shortcode fields
`live_key` needs. Seeding costs no additional Instagram requests, and needs
only a *listing* of the archive, never its contents.
"""
import sqlite3
db.parent.mkdir(parents=True, exist_ok=True)
con = sqlite3.connect(db)
con.execute(ARCHIVE_SCHEMA)
rows = [
(ARCHIVE_KEY(item["media_id"]),)
for item in live
if live_key(item, kind) in existing
]
con.executemany("INSERT OR IGNORE INTO archive (entry) VALUES (?)", rows)
con.commit()
con.close()
return len(rows)
def probe_live(src: Source, config: Path, cookies: str) -> list[dict]:
"""
One metadata-only listing pass. `sleep` is forced to 0 because it otherwise
applies per *file* even with no download — 2275 files at 1-3s each is over
an hour for a single profile.
"""
out = subprocess.run(
["gallery-dl", "-j", "--config", str(config),
"--cookies-from-browser", cookies, "-o", "sleep=0", src.url],
capture_output=True, text=True, check=True,
)
items: list[dict] = []
def walk(node):
if isinstance(node, dict):
if "media_id" in node and "shortcode" in node:
items.append(node)
for value in node.values():
walk(value)
elif isinstance(node, list):
for value in node:
walk(value)
walk(json.loads(out.stdout))
return items
ALL_KINDS = ("posts", "reels", "stories", "highlights")
# Stories cannot be backfilled and expire in 24h, so a run that only wants
# stories is both cheap and the one worth scheduling daily.
STORIES_ONLY = {"stories"}
class SyncState:
"""
What has already been spent against `instagram.com`.
Exists because nothing else in this tool has any memory: every invocation
used to start from zero and happily re-enumerate profiles it had listed
minutes earlier. That is what suspended the account — the listing passes,
not the downloads.
Two facts are tracked per source:
seeded the skip-archive has been primed from the archive listing.
This is a ONE-TIME bootstrap: afterwards the archive DB records
every item gallery-dl has seen, so the source never needs
probing again. This is the single biggest request saving here.
fetched when it was last downloaded, so a re-run soon after is refused
rather than silently repeating the whole pass.
"""
VERSION = 1
def __init__(self, path: Path):
self.path = path
self.data = {"version": self.VERSION, "sources": {}}
if path.is_file():
try:
loaded = json.loads(path.read_text())
if loaded.get("version") == self.VERSION:
self.data = loaded
except Exception:
pass # a corrupt state file must never block a sync
def _entry(self, url: str) -> dict:
return self.data.setdefault("sources", {}).setdefault(url, {})
def needs_seed(self, url: str) -> bool:
return not self._entry(url).get("seeded")
def mark_seeded(self, url: str, stamp: str) -> None:
self._entry(url)["seeded"] = stamp
def last_fetch(self, url: str) -> str | None:
return self._entry(url).get("fetched")
def mark_fetched(self, url: str, stamp: str) -> None:
self._entry(url)["fetched"] = stamp
def save(self) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
self.path.write_text(json.dumps(self.data, indent=1, sort_keys=True))
def hours_since(stamp: str | None, now: float) -> float:
"""Hours between an ISO stamp and `now`; infinite when never."""
if not stamp:
return float("inf")
try:
then = dt.datetime.fromisoformat(stamp)
except ValueError:
return float("inf")
if then.tzinfo is None:
then = then.replace(tzinfo=dt.timezone.utc)
return (now - then.timestamp()) / 3600.0
def plan_source(src: Source, state: SyncState, now: float,
min_interval: float) -> tuple[bool, bool, str]:
"""
Decide what a source needs: (fetch, seed, reason).
Seeding is skipped once done, and skipped entirely for stories — a story
cannot exist in the archive before it is fetched, so there is nothing to
seed from, and probing would double the request cost of the cheapest
surface we have.
"""
since = hours_since(state.last_fetch(src.url), now)
if since < min_interval:
return (False, False, f"fetched {since:.1f}h ago, under the "
f"{min_interval:g}h floor")
if src.kind == "stories":
return (True, False, "stories: no seed needed")
if state.needs_seed(src.url):
return (True, True, "first run: seeding from the archive listing")
return (True, False, "already seeded; the skip-archive knows what we hold")
class ProbeCache:
"""
Listing-pass results, kept so an interrupted run does not pay for them
twice. Yesterday an aborted sync re-enumerated five profiles on restart.
"""
def __init__(self, path: Path, ttl_hours: float):
self.path = path
self.ttl = ttl_hours
self.data: dict = {}
if path.is_file():
try:
self.data = json.loads(path.read_text())
except Exception:
self.data = {}
def get(self, url: str, now: float) -> list[dict] | None:
entry = self.data.get(url)
if not entry or hours_since(entry.get("at"), now) > self.ttl:
return None
return entry.get("items")
def put(self, url: str, items: list[dict], stamp: str) -> None:
# Only the fields seeding needs, so the cache stays small.
self.data[url] = {"at": stamp, "items": [
{k: i.get(k) for k in ("shortcode", "post_shortcode", "num", "media_id")}
for i in items
]}
def save(self) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
self.path.write_text(json.dumps(self.data))
def rsync_command(staging: Path, dest: str, dry_run: bool) -> list[str]:
"""
Publish a staging tree into the archive.
`--ignore-existing` is not an optimisation, it is the safety property: the
archive deliberately outlives Instagram (posts exist here that Instagram no
longer serves), so publishing must only ever *add*. No `--delete`, and
nothing already present is overwritten — including sidecars, which get
rewritten on every run and would otherwise churn the synced share.
`dest` may be a local path or any rsync destination (`user@host:/path`),
because the archive usually is not writable from the fetch host.
"""
cmd = ["rsync", "-a", "--ignore-existing", "--partial", "--info=stats2",
# Belt and braces: the config lives outside staging, but nothing
# resembling tooling output should ever reach the archive. Archive
# sidecars are always "<date>_<user> - <code>.json", so none of
# these can match real content.
"--exclude", "gdl-sync*.json",
"--exclude", "*.gdl-config.json",
"--exclude", ".gdl-*",
"--exclude", "*.sqlite", "--exclude", "*.db"]
if dry_run:
cmd.append("--dry-run")
# Trailing slash: copy the *contents* of staging into dest.
cmd += [f"{staging}/", dest if dest.endswith("/") else dest + "/"]
return cmd
def publish(staging: Path, dest: str, dry_run: bool) -> int:
if not any(staging.iterdir()):
print(" nothing staged; skipping publish")
return 0
cmd = rsync_command(staging, dest, dry_run)
print(" " + " ".join(cmd))
return subprocess.run(cmd).returncode
def main() -> int:
# A sync runs for hours and is normally watched through a redirected log,
# where Python's block buffering would withhold progress until it happened
# to flush -- and the gallery-dl subprocesses write to the same descriptor
# unbuffered, so the log would also interleave out of order.
sys.stdout.reconfigure(line_buffering=True)
sys.stderr.reconfigure(line_buffering=True)
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--index", required=True,
help="existing archive listing: a local root, or the "
"viewer's base URL (only a FILE LISTING is needed, "
"never the contents)")
ap.add_argument("--publish", required=True,
help="rsync destination for fetched files; a local path or "
"user@host:/path")
ap.add_argument("--staging", type=Path, required=True,
help="local scratch directory gallery-dl writes into")
g = ap.add_mutually_exclusive_group(required=True)
g.add_argument("--profile", action="append", default=[],
help="profile to sync; repeatable")
g.add_argument("--all", action="store_true", help="every profile on disk")
g.add_argument("--urls-file", type=Path,
help="file of Instagram profile URLs or usernames, one per "
"line; # comments and blank lines allowed")
ap.add_argument("--cookies", default="chrome:/home/matt/.config/google-chrome-devtools",
help="gallery-dl --cookies-from-browser value")
ap.add_argument("--archive-db", type=Path, default=None,
help="gallery-dl skip-archive sqlite path")
ap.add_argument("--rate", default="1M", help="per-download rate cap")
ap.add_argument("--sleep-request", nargs=2, type=float, default=[6.0, 10.0],
metavar=("MIN", "MAX"))
ap.add_argument("--sleep", nargs=2, type=float, default=[3.0, 6.0],
metavar=("MIN", "MAX"))
ap.add_argument("--only", default=",".join(ALL_KINDS),
help="comma-separated surfaces to sync: "
"posts,reels,stories,highlights. Use --only stories "
"for the cheap daily run.")
ap.add_argument("--min-interval", type=float, default=20.0, metavar="HOURS",
help="refuse to re-fetch a source touched more recently "
"than this (default 20h); the guard that makes a "
"restart cheap instead of a repeat")
ap.add_argument("--max-sources", type=int, default=0, metavar="N",
help="hard ceiling on sources touched in one run "
"(0 = no limit)")
ap.add_argument("--abort", type=int, default=0, metavar="N",
help="stop enumerating posts/reels after N consecutive "
"already-archived FILES (0 = walk everything, the "
"default). 50 is a safe routine value; it cuts the "
"per-run listing cost by roughly 85%%, at the price "
"of no longer noticing edited carousels")
ap.add_argument("--probe-ttl", type=float, default=24.0, metavar="HOURS",
help="reuse cached listing results younger than this")
ap.add_argument("--force", action="store_true",
help="ignore --min-interval and the probe cache")
mode = ap.add_mutually_exclusive_group()
mode.add_argument("--dry-run", action="store_true", default=True,
help="print the plan and the config; default")
mode.add_argument("--execute", action="store_true",
help="actually run gallery-dl")
args = ap.parse_args()
if not shutil.which("gallery-dl"):
print("gallery-dl not on PATH", file=sys.stderr)
return 2
if not shutil.which("rsync"):
print("rsync not on PATH", file=sys.stderr)
return 2
index = ArchiveIndex(args.index)
names = index.profiles()
if args.urls_file:
if not args.urls_file.is_file():
print(f"urls file not found: {args.urls_file}", file=sys.stderr)
return 2
wanted = read_urls_file(args.urls_file)
if not wanted:
print(f"no usable profiles in {args.urls_file}", file=sys.stderr)
return 2
print(f"read {len(wanted)} profile(s) from {args.urls_file}")
selected = [Profile(p) for p in wanted]
elif args.profile:
for p in args.profile:
if p not in names:
print(f"note: {p} is not in the index yet; it will be created")
selected = [Profile(p) for p in args.profile]
else:
selected = [Profile(p) for p in sorted(names)]
config = build_config(args.rate, list(args.sleep_request),
list(args.sleep), args.abort)
args.staging.mkdir(parents=True, exist_ok=True)
# Deliberately a SIBLING of the staging directory, not inside it: staging is
# rsynced wholesale into the archive, and a dry run caught this file being
# published to the archive root.
config_path = args.staging.parent / f"{args.staging.name}.gdl-config.json"
kinds = {k.strip() for k in args.only.split(",") if k.strip()}
unknown = kinds - set(ALL_KINDS)
if unknown:
print(f"unknown surface(s): {', '.join(sorted(unknown))}", file=sys.stderr)
return 2
state_path = (args.archive_db.with_suffix(".state.json") if args.archive_db
else args.staging.parent / f"{args.staging.name}.state.json")
state = SyncState(state_path)
now = dt.datetime.now(dt.timezone.utc)
now_ts, stamp = now.timestamp(), now.isoformat()
min_interval = 0.0 if args.force else args.min_interval
plan: list[tuple[Profile, Source, bool]] = []
skipped = 0
for prof in selected:
for src in prof.sources(kinds):
fetch, seed, reason = plan_source(src, state, now_ts, min_interval)
if not fetch:
skipped += 1
print(f" skip {prof.user}/{src.kind}: {reason}")
continue
if args.max_sources and len(plan) >= args.max_sources:
skipped += 1
continue
plan.append((prof, src, seed))
print(f"profiles : {len(selected)}")
print(f"surfaces : {','.join(k for k in ALL_KINDS if k in kinds)}")
print(f"sources : {len(plan)} to sync, {skipped} skipped")
print(f"pacing : {args.sleep_request[0]}-{args.sleep_request[1]}s between "
f"requests, rate cap {args.rate}")
print(f"staging : {args.staging}")
print(f"publish : {args.publish}")
print()
if not args.execute:
for prof, src, seed in plan:
dest = src.directory or "(per-highlight)"
note = " [will seed]" if seed else ""
print(f" {prof.user:<20} {src.kind:<11} -> {dest}{note}")
print()
print(" " + " ".join(rsync_command(args.staging, args.publish, True)))
print("\ndry run; nothing fetched. pass --execute to run.")
return 0
config_path.write_text(json.dumps(config, indent=2))
probes = ProbeCache(state_path.with_suffix(".probes.json"),
0.0 if args.force else args.probe_ttl)
failures = 0
for prof, src, seed in plan:
print(f"==> {prof.user} / {src.kind}")
stage_dir = args.staging / (src.directory or ".")
stage_dir.mkdir(parents=True, exist_ok=True)
# Prime the skip-archive from what the archive already holds, so
# fetching into an empty staging directory pulls only what is missing.
# Done once per source, ever: afterwards the archive DB records
# everything gallery-dl has seen and no listing pass is needed.
if seed and args.archive_db:
try:
live = probes.get(src.url, now_ts)
if live is None:
live = probe_live(src, config_path, args.cookies)
probes.put(src.url, live, stamp)
probes.save()
else:
print(f" reusing {len(live)} cached listing items")
held = index_existing(index.listing(prof.user))
seeded = seed_archive_db(args.archive_db, held, live,
src.subcategory)
print(f" seeded {seeded} of {len(live)} live items")
state.mark_seeded(src.url, stamp)
state.save()
except subprocess.CalledProcessError as exc:
failures += 1
print(f" probe FAILED: {exc}", file=sys.stderr)
continue
cmd = gdl_command(src, args.staging, config_path, args.cookies,
args.archive_db)
result = subprocess.run(cmd)
if result.returncode != 0:
failures += 1
# Keep going: one private or renamed profile must not abort the run.
print(f" FAILED (exit {result.returncode})", file=sys.stderr)
else:
# Recorded even for an empty fetch: the request was still spent.
state.mark_fetched(src.url, stamp)
state.save()
# Publish once, at the end, so a partially-fetched profile never reaches
# the archive mid-run. Only ever adds -- see rsync_command.
print("\n==> publish")
if publish(args.staging, args.publish, dry_run=False) != 0:
failures += 1
print(f"\ndone; {failures} step(s) failed")
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main())
-277
View File
@@ -1,277 +0,0 @@
/**
* Generate JDownloader2 .crawljob files for the archives on disk.
*
* The manual flow is: paste a profile URL into JDownloader, paste the /reels
* URL separately (the profile page misses some reels), and set the output
* folder by hand times however many profiles you keep. This emits one
* crawljob per source with the folder already pointed at the right directory,
* so JDownloader's folder-watch picks the whole batch up at once.
*
* Profiles and their sidecar directories are derived with the same grouping
* logic the server uses, so the output folders always match what the viewer
* expects to find.
*
* Only posts and reels are emitted. Story and highlight URLs can't be rebuilt
* from a directory name highlights need their numeric id and stories expire
* so those stay manual.
*
* Crawljob format verified against JDownloader's own docs for the extension:
* src/org/jdownloader/extensions/folderwatchV2/explain.txt. JDownloader
* develops on SVN; read it via the daily mirror at
* https://github.com/mycodedoesnotcompile2/jdownloader_mirror (svn_trunk/),
* not one of the abandoned GitHub copies several are a decade stale.
*
* Entries are separated by `->NEW ENTRY<-` and any property may be omitted.
* There is also a `setBeforePackagizerEnabled` companion to
* `overwritePackagizerEnabled`, if the Packagizer ever needs to see these
* values before they're applied.
*
* Usage:
* npx tsx scripts/jd2-sync.ts --archives <dir> [options]
*
* --archives <dir> Archive root to scan (default: $ARCHIVES_DIR)
* --out <dir> JDownloader folder-watch directory to write into
* --download-base <dir> Root path as *JDownloader* sees it, when it runs on
* a different machine than this script (e.g. a mapped
* drive). Defaults to --archives.
* --user <name> Only this profile (repeatable)
* --skip <name> Never emit jobs for this directory (repeatable).
* Also read from a `.jd2ignore` file in the archive
* root, one name per line.
* --chunks <n> Connections per file (default 1: multi-chunk ranged
* requests are the one CDN pattern that doesn't look
* like a browser)
* --auto-start Start downloads immediately instead of parking them
* in the LinkGrabber for review
* --all-reels Emit a reels job even where no reels directory
* exists yet
* --dry-run Print the crawljob instead of writing it
*/
import fs from 'fs';
import path from 'path';
import { groupArchiveDirectories, ArchiveSource } from '../src/lib/archive-grouping.js';
interface Options {
archives: string;
out: string | null;
downloadBase: string;
users: string[];
skip: Set<string>;
chunks: number;
autoStart: boolean;
allReels: boolean;
dryRun: boolean;
}
const parseArgs = (argv: string[]): Options => {
const opts: Options = {
archives: process.env.ARCHIVES_DIR ?? '',
out: null,
downloadBase: '',
users: [],
skip: new Set(),
chunks: 1,
autoStart: false,
allReels: false,
dryRun: false,
};
for (let i = 0; i < argv.length; i++) {
const arg = argv[i];
const next = () => argv[++i];
switch (arg) {
case '--archives': opts.archives = path.resolve(next()); break;
case '--out': opts.out = path.resolve(next()); break;
case '--download-base': opts.downloadBase = next(); break;
case '--user': opts.users.push(next()); break;
case '--skip': opts.skip.add(next()); break;
case '--chunks': opts.chunks = parseInt(next(), 10); break;
case '--auto-start': opts.autoStart = true; break;
case '--all-reels': opts.allReels = true; break;
case '--dry-run': opts.dryRun = true; break;
case '--help': case '-h': printUsage(); process.exit(0);
default:
console.error(`Unknown argument: ${arg}`);
process.exit(1);
}
}
if (!opts.archives) {
console.error('No archive root. Pass --archives <dir> or set ARCHIVES_DIR.');
process.exit(1);
}
if (!opts.downloadBase) opts.downloadBase = opts.archives;
if (!opts.out && !opts.dryRun) {
console.error('No destination. Pass --out <folder-watch dir>, or --dry-run to preview.');
process.exit(1);
}
return opts;
};
const printUsage = () => {
const header = readHeaderComment();
console.log(header);
};
/** Print the usage block from this file's own header comment. */
const readHeaderComment = () => {
try {
const self = fs.readFileSync(new URL(import.meta.url), 'utf8');
const usage = self.slice(self.indexOf(' * Usage:'), self.indexOf(' */'));
return usage.split('\n').map(l => l.replace(/^ \* ?/, '')).join('\n');
} catch {
return 'See the comment at the top of scripts/jd2-sync.ts';
}
};
/**
* JDownloader escapes nothing in crawljob values, so a stray newline would
* silently split a property. Paths with spaces are fine as-is.
*/
const sanitise = (value: string) => value.replace(/[\r\n]+/g, ' ').trim();
/**
* Instagram usernames are 130 characters of letters, digits, dots and
* underscores. Archive roots also collect directories that aren't profiles at
* all tool output, exports from other services and pointing a crawl at
* those spends requests on instagram.com to be told the profile doesn't exist.
* That's the exact traffic worth not spending.
*/
const USERNAME_RE = /^[A-Za-z0-9._]{1,30}$/;
/** Directory names to skip, from `.jd2ignore` in the archive root. */
const readIgnoreFile = (archives: string): string[] => {
try {
return fs.readFileSync(path.join(archives, '.jd2ignore'), 'utf8')
.split('\n').map(l => l.trim()).filter(l => l && !l.startsWith('#'));
} catch {
return [];
}
};
interface Job {
user: string;
kind: 'posts' | 'reels';
url: string;
packageName: string;
downloadFolder: string;
fileCount: number | null;
}
const buildJobs = (opts: Options): Job[] => {
const dirNames = fs.readdirSync(opts.archives, { withFileTypes: true })
.filter(e => e.isDirectory() && !/^[.@_]/.test(e.name))
.map(e => e.name);
const groups = groupArchiveDirectories(dirNames);
const jobs: Job[] = [];
const skipped: string[] = [];
for (const name of readIgnoreFile(opts.archives)) opts.skip.add(name);
const countFiles = (dir: string): number | null => {
try {
return fs.readdirSync(path.join(opts.archives, dir)).length;
} catch {
return null;
}
};
// JDownloader must be given the path *it* can see, which differs from the
// scan path whenever the archive lives on a share.
const downloadFolderFor = (dir: string) =>
opts.downloadBase.includes('\\')
? `${opts.downloadBase.replace(/\\$/, '')}\\${dir}`
: path.posix.join(opts.downloadBase, dir);
for (const [user, sources] of [...groups].sort(([a], [b]) => a.localeCompare(b))) {
if (opts.users.length && !opts.users.includes(user)) continue;
if (opts.skip.has(user)) { skipped.push(`${user} (ignored)`); continue; }
if (!USERNAME_RE.test(user)) { skipped.push(`${user} (not a username)`); continue; }
const has = (kind: ArchiveSource['kind']) => sources.find(s => s.kind === kind);
const base = has('posts');
if (!base) continue; // sidecar-only group: nothing sensible to point a URL at
jobs.push({
user, kind: 'posts',
url: `https://www.instagram.com/${encodeURIComponent(user)}/`,
packageName: base.dir,
downloadFolder: downloadFolderFor(base.dir),
fileCount: countFiles(base.dir),
});
const reels = has('reels');
if (reels || opts.allReels) {
const dir = reels?.dir ?? `${user} - reels`;
jobs.push({
user, kind: 'reels',
url: `https://www.instagram.com/${encodeURIComponent(user)}/reels/`,
packageName: dir,
downloadFolder: downloadFolderFor(dir),
fileCount: reels ? countFiles(dir) : null,
});
}
}
if (skipped.length) {
console.error(`Skipped ${skipped.length} director${skipped.length === 1 ? 'y' : 'ies'}:`);
for (const s of skipped) console.error(` - ${s}`);
console.error('');
}
return jobs;
};
const renderCrawljob = (jobs: Job[], opts: Options): string =>
jobs.map(job => [
`text=${sanitise(job.url)}`,
`packageName=${sanitise(job.packageName)}`,
`downloadFolder=${sanitise(job.downloadFolder)}`,
`chunks=${opts.chunks}`,
// Without this a Packagizer rule can override downloadFolder and scatter
// files away from the directory the viewer reads.
'overwritePackagizerEnabled=TRUE',
`autoStart=${opts.autoStart ? 'TRUE' : 'FALSE'}`,
`autoConfirm=${opts.autoStart ? 'TRUE' : 'FALSE'}`,
'enabled=TRUE',
`comment=instaarchive jd2-sync (${job.kind})`,
].join('\n')).join('\n->NEW ENTRY<-\n');
const main = () => {
const opts = parseArgs(process.argv.slice(2));
const jobs = buildJobs(opts);
if (!jobs.length) {
console.error('No profiles matched.');
process.exit(1);
}
console.error(`Archive root : ${opts.archives}`);
console.error(`JD sees root : ${opts.downloadBase}`);
console.error(`Jobs : ${jobs.length} (${new Set(jobs.map(j => j.user)).size} profiles)\n`);
for (const job of jobs) {
const count = job.fileCount === null ? 'new' : `${job.fileCount} files`;
console.error(` ${job.kind.padEnd(5)} ${job.user.padEnd(24)} -> ${job.packageName} (${count})`);
}
console.error('');
const body = renderCrawljob(jobs, opts);
if (opts.dryRun || !opts.out) {
console.log(body);
return;
}
fs.mkdirSync(opts.out, { recursive: true });
const file = path.join(opts.out, `instaarchive-${new Date().toISOString().replace(/[:.]/g, '-')}.crawljob`);
fs.writeFileSync(file, body, 'utf8');
console.error(`Wrote ${file}`);
console.error(opts.autoStart
? 'Downloads will start automatically.'
: 'Links land in the LinkGrabber for review; start them when ready.');
};
main();
-231
View File
@@ -1,231 +0,0 @@
#!/usr/bin/env python3
"""
Tests for the request-budget logic in gdl-sync.py.
python3 -m unittest discover -s scripts -p 'test_*.py'
Deliberately stdlib-only, so it runs anywhere the sync itself runs. What is
covered here is the part that decides whether to spend a request the part
whose absence got the archive's Instagram account suspended.
"""
import datetime as dt
import importlib.util
import json
import sys
import tempfile
import unittest
from pathlib import Path
_spec = importlib.util.spec_from_file_location(
"gdl_sync", Path(__file__).with_name("gdl-sync.py"))
gdl = importlib.util.module_from_spec(_spec)
sys.modules["gdl_sync"] = gdl
_spec.loader.exec_module(gdl)
NOW = dt.datetime(2026, 8, 18, 12, 0, tzinfo=dt.timezone.utc)
NOW_TS = NOW.timestamp()
def ago(hours: float) -> str:
return (NOW - dt.timedelta(hours=hours)).isoformat()
class SourceSelection(unittest.TestCase):
def test_only_stories_is_a_single_cheap_source(self):
srcs = gdl.Profile("u").sources(gdl.STORIES_ONLY)
self.assertEqual([s.kind for s in srcs], ["stories"])
self.assertEqual(srcs[0].directory, "story - u")
def test_full_sync_covers_every_surface(self):
srcs = gdl.Profile("u").sources(set(gdl.ALL_KINDS))
self.assertEqual([s.kind for s in srcs], list(gdl.ALL_KINDS))
def test_reels_and_stories_go_to_their_own_directories(self):
by_kind = {s.kind: s for s in gdl.Profile("u").sources(set(gdl.ALL_KINDS))}
self.assertEqual(by_kind["posts"].directory, "u")
self.assertEqual(by_kind["reels"].directory, "u - reels")
# Highlights derive their directory from the title mid-extraction.
self.assertEqual(by_kind["highlights"].directory, "")
class PlanSource(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.state = gdl.SyncState(Path(self.tmp.name) / "state.json")
self.posts = gdl.Profile("u").sources({"posts"})[0]
self.stories = gdl.Profile("u").sources({"stories"})[0]
def tearDown(self):
self.tmp.cleanup()
def test_first_run_seeds(self):
fetch, seed, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
self.assertTrue(fetch)
self.assertTrue(seed)
def test_seeding_happens_only_once(self):
self.state.mark_seeded(self.posts.url, ago(720))
fetch, seed, reason = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
self.assertTrue(fetch)
self.assertFalse(seed, "a seeded source must never be re-probed")
self.assertIn("already seeded", reason)
def test_stories_never_seed(self):
# A story cannot be in the archive before it is fetched, so probing
# would double the cost of the cheapest surface for no benefit.
_, seed, reason = gdl.plan_source(self.stories, self.state, NOW_TS, 20)
self.assertFalse(seed)
self.assertIn("no seed", reason)
def test_recent_fetch_is_refused(self):
self.state.mark_fetched(self.posts.url, ago(3))
fetch, _, reason = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
self.assertFalse(fetch)
self.assertIn("under the", reason)
def test_an_old_fetch_is_allowed_again(self):
self.state.mark_fetched(self.posts.url, ago(30))
fetch, _, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
self.assertTrue(fetch)
def test_daily_stories_pass_a_20h_floor(self):
# The cadence this is built for: once a day, every day.
self.state.mark_fetched(self.stories.url, ago(24))
fetch, _, _ = gdl.plan_source(self.stories, self.state, NOW_TS, 20)
self.assertTrue(fetch)
def test_force_disables_the_floor(self):
self.state.mark_fetched(self.posts.url, ago(1))
fetch, _, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 0.0)
self.assertTrue(fetch)
def test_the_aborted_run_scenario(self):
"""
Yesterday's failure: a run died mid-way and the restart re-enumerated
every profile. Seeded-but-not-fetched must not re-probe.
"""
self.state.mark_seeded(self.posts.url, ago(0.5))
fetch, seed, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
self.assertTrue(fetch, "the fetch still needs to happen")
self.assertFalse(seed, "but the listing pass must not be paid for twice")
class StatePersistence(unittest.TestCase):
def test_state_survives_a_reload(self):
with tempfile.TemporaryDirectory() as d:
path = Path(d) / "state.json"
a = gdl.SyncState(path)
a.mark_seeded("https://x/", ago(1))
a.mark_fetched("https://x/", ago(1))
a.save()
b = gdl.SyncState(path)
self.assertFalse(b.needs_seed("https://x/"))
self.assertEqual(b.last_fetch("https://x/"), ago(1))
def test_a_corrupt_state_file_never_blocks_a_sync(self):
with tempfile.TemporaryDirectory() as d:
path = Path(d) / "state.json"
path.write_text("{ not json")
self.assertTrue(gdl.SyncState(path).needs_seed("https://x/"))
class ProbeCaching(unittest.TestCase):
def test_fresh_entries_are_reused_and_stale_ones_are_not(self):
with tempfile.TemporaryDirectory() as d:
cache = gdl.ProbeCache(Path(d) / "p.json", ttl_hours=24)
cache.put("https://x/", [{"shortcode": "A", "post_shortcode": "A",
"num": 1, "media_id": "1"}], ago(1))
self.assertEqual(len(cache.get("https://x/", NOW_TS)), 1)
cache.put("https://y/", [{"shortcode": "B", "post_shortcode": "B",
"num": 1, "media_id": "2"}], ago(48))
self.assertIsNone(cache.get("https://y/", NOW_TS))
def test_cache_keeps_only_the_fields_seeding_needs(self):
with tempfile.TemporaryDirectory() as d:
path = Path(d) / "p.json"
cache = gdl.ProbeCache(path, ttl_hours=24)
cache.put("https://x/", [{"shortcode": "A", "post_shortcode": "A",
"num": 1, "media_id": "1",
"description": "x" * 5000}], ago(0))
cache.save()
self.assertNotIn("description", path.read_text())
def test_a_miss_is_reported_rather_than_guessed(self):
with tempfile.TemporaryDirectory() as d:
cache = gdl.ProbeCache(Path(d) / "p.json", ttl_hours=24)
self.assertIsNone(cache.get("https://never-seen/", NOW_TS))
class Seeding(unittest.TestCase):
"""The bug that seeded 5 of 2275: matching the wrong shortcode field."""
def test_posts_are_keyed_by_post_shortcode(self):
item = {"shortcode": "childcode", "post_shortcode": "POSTCODE",
"num": 2, "media_id": "9"}
self.assertEqual(gdl.live_key(item, "posts"), ("POSTCODE", 2))
def test_stories_are_keyed_by_the_per_item_shortcode(self):
item = {"shortcode": "ITEMCODE", "post_shortcode": "reelid",
"num": 3, "media_id": "9"}
self.assertEqual(gdl.live_key(item, "stories"), ("ITEMCODE", 1))
self.assertEqual(gdl.live_key(item, "highlights"), ("ITEMCODE", 1))
def test_index_existing_normalises_a_missing_index_to_one(self):
held = gdl.index_existing([
"u/2023-04-19_u - ABC.mp4",
"u/2023-04-12_u - DEF - 3.jpg",
"u/2023-04-12_u - DEF.txt", # sidecars are not media
"u/2023-04-12_u - DEF.json",
])
self.assertEqual(held, {("ABC", 1), ("DEF", 3)})
def test_seeding_marks_only_what_is_already_held(self):
with tempfile.TemporaryDirectory() as d:
db = Path(d) / "a.db"
live = [
{"post_shortcode": "HELD", "shortcode": "x", "num": 1, "media_id": "11"},
{"post_shortcode": "NEW", "shortcode": "y", "num": 1, "media_id": "22"},
]
n = gdl.seed_archive_db(db, {("HELD", 1)}, live, "posts")
self.assertEqual(n, 1)
import sqlite3
rows = {r[0] for r in sqlite3.connect(db).execute(
"SELECT entry FROM archive")}
self.assertEqual(rows, {"instagram11"})
class Publishing(unittest.TestCase):
def test_publish_only_ever_adds(self):
cmd = gdl.rsync_command(Path("/stage"), "host:/archives", dry_run=False)
self.assertIn("--ignore-existing", cmd)
self.assertNotIn("--delete", cmd)
def test_tooling_files_are_excluded_from_the_archive(self):
cmd = " ".join(gdl.rsync_command(Path("/stage"), "/dest", dry_run=True))
for pattern in ("gdl-sync*.json", "*.db"):
self.assertIn(pattern, cmd)
self.assertIn("--dry-run", cmd)
class UrlsFile(unittest.TestCase):
def test_reads_every_form_a_person_might_paste(self):
with tempfile.TemporaryDirectory() as d:
p = Path(d) / "urls.txt"
p.write_text(
"# comment\n"
"https://www.instagram.com/a/\n"
"https://instagram.com/b\n"
"www.instagram.com/c/\n"
"d\n"
" e # trailing\n"
"\n"
"https://www.instagram.com/a/\n" # duplicate
"https://www.instagram.com/p/ABC123/\n" # a post, not a profile
"not a username\n")
self.assertEqual(gdl.read_urls_file(p), ["a", "b", "c", "d", "e"])
if __name__ == "__main__":
unittest.main(verbosity=2)
+56 -2
View File
@@ -73,6 +73,17 @@ export default function App() {
*/
const initialRouteRef = useRef(parseRoute(window.location.pathname, window.location.search));
/**
* Back-button support for the post view. Every other URL change
* (`replaceState`s the tab/archive) is intentionally NOT pushed only
* opening a post gets its own history entry, matching Instagram's own
* back-button behaviour: Back closes the post instead of leaving the app.
*/
const pushedPostRef = useRef(false);
/** Set while reacting to a popstate, so the URL-sync effect below does not
* try to push/replace/back() again for a change the browser already made. */
const suppressNextSyncRef = useRef(false);
const isMobile = useIsMobile();
const fileInputRef = useRef<HTMLInputElement>(null);
const profilePicInputRef = useRef<HTMLInputElement>(null);
@@ -367,6 +378,13 @@ export default function App() {
// loader below is waiting to read.
if (!hasInitialLoaded) return;
// Consumed exactly once per popstate, regardless of what happens below:
// a popstate-driven change often already matches the URL (the browser
// already moved the pointer), which used to leave this flag stuck true
// and silently no-op the NEXT real close/open until something reset it.
const wasPopState = suppressNextSyncRef.current;
suppressNextSyncRef.current = false;
const archive = currentArchive?.name ?? (allPosts.length > 0 ? username : null) ?? null;
const nextPath = buildPath({
archive,
@@ -374,12 +392,48 @@ export default function App() {
post: selectedPost ? postSlug(selectedPost) : null,
});
if (nextPath !== window.location.pathname + window.location.search) {
console.log(`[Permalink] Updating URL to: ${nextPath}`);
if (nextPath === window.location.pathname + window.location.search) return;
if (wasPopState) return; // the browser already navigated; nothing to add
console.log(`[Permalink] Updating URL to: ${nextPath}`);
if (selectedPost && !pushedPostRef.current) {
// Opening a post: push, so Back closes it instead of leaving the app.
window.history.pushState(null, '', nextPath);
pushedPostRef.current = true;
} else if (!selectedPost && pushedPostRef.current) {
// Closing a post that was pushed for: consume that entry rather than
// piling a new one on top of it, so Back still means "one step".
pushedPostRef.current = false;
window.history.back();
} else {
window.history.replaceState(null, '', nextPath);
}
}, [hasInitialLoaded, currentArchive?.name, username, allPosts.length, activeTab, selectedPost?.id]);
/**
* Back/forward support for the post view. Only a post push (above) ever
* creates an entry, so this only ever needs to open or close a post
* never re-derive the tab or archive, which stayed on replaceState.
*/
useEffect(() => {
const onPopState = () => {
suppressNextSyncRef.current = true;
const route = parseRoute(window.location.pathname, window.location.search);
const post = route.post ? findPostBySlug(allPosts, route.post) : null;
if (post) {
setActiveTab(tabForSource(post.source));
setSelectedPost(post);
pushedPostRef.current = true; // forward navigation can land back here
} else {
setSelectedPost(null);
pushedPostRef.current = false;
}
};
window.addEventListener('popstate', onPopState);
return () => window.removeEventListener('popstate', onPopState);
}, [allPosts]);
useEffect(() => {
if (hasInitialLoaded) return;
+28 -28
View File
@@ -3,39 +3,39 @@ import { classifyDirectory, groupArchiveDirectories } from './archive-grouping';
describe('classifyDirectory', () => {
it('treats a bare profile directory as the base', () => {
expect(classifyDirectory('0ct0ber19')).toEqual({
owner: '0ct0ber19',
source: { kind: 'posts', dir: '0ct0ber19' },
expect(classifyDirectory('4utumn07')).toEqual({
owner: '4utumn07',
source: { kind: 'posts', dir: '4utumn07' },
});
});
it('recognises a reels sidecar', () => {
expect(classifyDirectory('0ct0ber19 - reels')).toEqual({
owner: '0ct0ber19',
source: { kind: 'reels', dir: '0ct0ber19 - reels' },
expect(classifyDirectory('4utumn07 - reels')).toEqual({
owner: '4utumn07',
source: { kind: 'reels', dir: '4utumn07 - reels' },
});
});
it('recognises a stories sidecar', () => {
expect(classifyDirectory('story - cher_ryppo')).toEqual({
owner: 'cher_ryppo',
source: { kind: 'stories', dir: 'story - cher_ryppo' },
expect(classifyDirectory('story - dawn_petal')).toEqual({
owner: 'dawn_petal',
source: { kind: 'stories', dir: 'story - dawn_petal' },
});
});
it('splits highlight owner from title', () => {
const { owner, source } = classifyDirectory('story highlights - 0ct0ber19 - Heestory');
expect(owner).toBe('0ct0ber19');
const { owner, source } = classifyDirectory('story highlights - 4utumn07 - Sunstory');
expect(owner).toBe('4utumn07');
expect(source.kind).toBe('highlight');
expect(source.title).toBe('Heestory');
expect(source.title).toBe('Sunstory');
});
it.each([
['story highlights - theoldtaylorswiftinsta - 💙2014-1989 era', 'theoldtaylorswiftinsta', '💙2014-1989 era'],
['story highlights - heejin_theworld - [Dall]', 'heejin_theworld', '[Dall]'],
['story highlights - official_artms - Cosmo Schedule', 'official_artms', 'Cosmo Schedule'],
['story highlights - 0ct0ber19 - Drawheeing', '0ct0ber19', 'Drawheeing'],
['story highlights - official_artms - G.C.I', 'official_artms', 'G.C.I'],
['story highlights - theoldlyricmuseinsta - 💙1999-2005 era', 'theoldlyricmuseinsta', '💙1999-2005 era'],
['story highlights - member_theworld - [Bracket]', 'member_theworld', '[Bracket]'],
['story highlights - official_band - Tour Schedule', 'official_band', 'Tour Schedule'],
['story highlights - 4utumn07 - Sketching', '4utumn07', 'Sketching'],
['story highlights - official_band - A.B.C', 'official_band', 'A.B.C'],
])('handles real-world title %s', (dir, owner, title) => {
const result = classifyDirectory(dir);
expect(result.owner).toBe(owner);
@@ -57,25 +57,25 @@ describe('classifyDirectory', () => {
describe('groupArchiveDirectories', () => {
const dirs = [
'0ct0ber19',
'0ct0ber19 - reels',
'story - 0ct0ber19',
'story highlights - 0ct0ber19 - Heestory',
'story highlights - 0ct0ber19 - Drawheeing',
'carlyraejepsen',
'4utumn07',
'4utumn07 - reels',
'story - 4utumn07',
'story highlights - 4utumn07 - Sunstory',
'story highlights - 4utumn07 - Sketching',
'kestrelsings',
];
it('folds sidecars into their base profile', () => {
const groups = groupArchiveDirectories(dirs);
expect([...groups.keys()].sort()).toEqual(['0ct0ber19', 'carlyraejepsen']);
expect(groups.get('0ct0ber19')).toHaveLength(5);
expect(groups.get('carlyraejepsen')).toHaveLength(1);
expect([...groups.keys()].sort()).toEqual(['4utumn07', 'kestrelsings']);
expect(groups.get('4utumn07')).toHaveLength(5);
expect(groups.get('kestrelsings')).toHaveLength(1);
});
it('orders sources posts, reels, stories, then highlights by title', () => {
const sources = groupArchiveDirectories(dirs).get('0ct0ber19')!;
const sources = groupArchiveDirectories(dirs).get('4utumn07')!;
expect(sources.map(s => s.kind)).toEqual(['posts', 'reels', 'stories', 'highlight', 'highlight']);
expect(sources.slice(3).map(s => s.title)).toEqual(['Drawheeing', 'Heestory']);
expect(sources.slice(3).map(s => s.title)).toEqual(['Sketching', 'Sunstory']);
});
it('still groups a sidecar whose base profile is missing', () => {
+4 -4
View File
@@ -18,10 +18,10 @@ export interface ArchiveSource {
/**
* Sidecar directories sit next to the profile directory they belong to:
*
* 0ct0ber19 -> posts (base)
* 0ct0ber19 - reels -> reels
* story - 0ct0ber19 -> stories
* story highlights - 0ct0ber19 - Heestory -> highlight "Heestory"
* 4utumn07 -> posts (base)
* 4utumn07 - reels -> reels
* story - 4utumn07 -> stories
* story highlights - 4utumn07 - Sunstory -> highlight "Sunstory"
*
* Instagram usernames cannot contain spaces, so matching the username as a
* run of non-space characters reliably separates it from a highlight title
+5 -5
View File
@@ -14,11 +14,11 @@ describe('isSystemDirectory', () => {
});
it.each([
'0ct0ber19',
'0ct0ber19 - reels',
'story - cher_ryppo',
'story highlights - official_artms - G.C.I',
'story highlights - theoldtaylorswiftinsta - 💙2014-1989 era',
'4utumn07',
'4utumn07 - reels',
'story - dawn_petal',
'story highlights - official_band - A.B.C',
'story highlights - theoldlyricmuseinsta - 💙1999-2005 era',
'Heejin_Bubble heejinmedia',
'gallery-dl',
'posts',
+26 -26
View File
@@ -3,10 +3,10 @@ import { canonicalItemId, parseArchiveFilename, scopedPostId } from './archive-p
describe('parseArchiveFilename — Instagram export format', () => {
it('parses a single-image post', () => {
expect(parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM.mp4')).toEqual({
expect(parseArchiveFilename('2023-04-19_4utumn07 - CrORBIcJJbM.mp4')).toEqual({
postId: 'CrORBIcJJbM',
date: '2023-04-19',
username: '0ct0ber19',
username: '4utumn07',
index: 1,
ext: 'mp4',
isStory: false,
@@ -15,7 +15,7 @@ describe('parseArchiveFilename — Instagram export format', () => {
});
it('parses a carousel slide index', () => {
const parsed = parseArchiveFilename('2023-04-12_0ct0ber19 - Cq8LrxSJAJE - 3.jpg');
const parsed = parseArchiveFilename('2023-04-12_4utumn07 - Cq8LrxSJAJE - 3.jpg');
expect(parsed).toMatchObject({ postId: 'Cq8LrxSJAJE', index: 3, ext: 'jpg' });
});
@@ -27,7 +27,7 @@ describe('parseArchiveFilename — Instagram export format', () => {
});
it('parses caption sidecar files', () => {
expect(parseArchiveFilename('2023-04-12_0ct0ber19 - Cq8LrxSJAJE.txt')).toMatchObject({
expect(parseArchiveFilename('2023-04-12_4utumn07 - Cq8LrxSJAJE.txt')).toMatchObject({
postId: 'Cq8LrxSJAJE',
ext: 'txt',
});
@@ -39,8 +39,8 @@ describe('parseArchiveFilename — Instagram export format', () => {
it('parses the story sidecar layout (date_user - N - shortcode)', () => {
// Files in `story - <user>` carry a per-day ordinal before the shortcode.
const parsed = parseArchiveFilename('2025-10-26_0ct0ber19 - 2 - DQRuDx9iW5Q.jpg', 'stories');
expect(parsed).toMatchObject({ date: '2025-10-26', username: '0ct0ber19', ext: 'jpg' });
const parsed = parseArchiveFilename('2025-10-26_4utumn07 - 2 - DQRuDx9iW5Q.jpg', 'stories');
expect(parsed).toMatchObject({ date: '2025-10-26', username: '4utumn07', ext: 'jpg' });
expect(parsed!.postId).toContain('DQRuDx9iW5Q');
});
@@ -71,9 +71,9 @@ describe('parseArchiveFilename — Instaloader format', () => {
describe('parseArchiveFilename — story highlights', () => {
it('parses the dateless highlight layout', () => {
expect(parseArchiveFilename('0ct0ber19 - C5dQPEYpd9W.mp4', 'highlight')).toMatchObject({
expect(parseArchiveFilename('4utumn07 - C5dQPEYpd9W.mp4', 'highlight')).toMatchObject({
postId: 'C5dQPEYpd9W',
username: '0ct0ber19',
username: '4utumn07',
ext: 'mp4',
isStory: false,
});
@@ -95,7 +95,7 @@ describe('parseArchiveFilename — story highlights', () => {
});
describe('parseArchiveFilename — non-matching files', () => {
it.each(['0ct0ber19.jpg', 'profile_pic.jpg', 'README.md', 'no-separator.png'])(
it.each(['4utumn07.jpg', 'profile_pic.jpg', 'README.md', 'no-separator.png'])(
'returns null for %s',
name => expect(parseArchiveFilename(name)).toBeNull(),
);
@@ -107,8 +107,8 @@ describe('scopedPostId', () => {
});
it('namespaces sidecar ids by directory', () => {
expect(scopedPostId('C5dQ', 'highlight', 'story highlights - u - Heestory'))
.toBe('story highlights - u - Heestory/C5dQ');
expect(scopedPostId('C5dQ', 'highlight', 'story highlights - u - Sunstory'))
.toBe('story highlights - u - Sunstory/C5dQ');
});
it('keeps the same shortcode distinct across sources', () => {
@@ -125,8 +125,8 @@ describe('scopedPostId', () => {
*/
describe('gallery-dl / JDownloader naming interop', () => {
it('treats a single-media post the same with or without an index', () => {
const jd2 = parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM.mp4')!;
const gdl = parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM - 1.mp4')!;
const jd2 = parseArchiveFilename('2023-04-19_4utumn07 - CrORBIcJJbM.mp4')!;
const gdl = parseArchiveFilename('2023-04-19_4utumn07 - CrORBIcJJbM - 1.mp4')!;
expect(jd2.postId).toBe(gdl.postId);
expect(jd2.index).toBe(gdl.index);
expect(jd2.index).toBe(1);
@@ -136,21 +136,21 @@ describe('gallery-dl / JDownloader naming interop', () => {
// JD2 pads to the width of the media count (10+ items -> "01"), and
// gallery-dl's count can be one higher, so the same post may be padded
// by one tool and not the other.
expect(parseArchiveFilename('2024-04-17_0ct0ber19 - C53YPQzp7Wj - 09.jpg')!.index).toBe(9);
expect(parseArchiveFilename('2024-04-17_0ct0ber19 - C53YPQzp7Wj - 9.jpg')!.index).toBe(9);
expect(parseArchiveFilename('2023-11-03_0ct0ber19 - CzM8Uf6B6H_ - 01.jpg')!.index).toBe(1);
expect(parseArchiveFilename('2024-04-17_4utumn07 - C53YPQzp7Wj - 09.jpg')!.index).toBe(9);
expect(parseArchiveFilename('2024-04-17_4utumn07 - C53YPQzp7Wj - 9.jpg')!.index).toBe(9);
expect(parseArchiveFilename('2023-11-03_4utumn07 - CzM8Uf6B6H_ - 01.jpg')!.index).toBe(1);
});
it('reads a gallery-dl story name, which carries a per-item shortcode', () => {
const p = parseArchiveFilename('2026-08-16_official_artms - DcF9OyhBJ1H.jpg', 'stories')!;
const p = parseArchiveFilename('2026-08-16_official_band - DcF9OyhBJ1H.jpg', 'stories')!;
expect(p.postId).toBe('DcF9OyhBJ1H');
expect(p.date).toBe('2026-08-16');
});
it('gives a dated highlight a real date instead of the mtime fallback', () => {
const mtime = Date.parse('2026-08-17T00:00:00Z');
const undated = parseArchiveFilename('0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
const dated = parseArchiveFilename('2024-08-04_0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
const undated = parseArchiveFilename('4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
const dated = parseArchiveFilename('2024-08-04_4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
// Same item either way, so re-fetching cannot split it into two posts.
expect(dated.postId).toBe(undated.postId);
expect(undated.date).toBe('2026-08-17');
@@ -167,13 +167,13 @@ describe('dateFromMtime', () => {
const mtime = Date.parse('2026-08-17T00:00:00Z');
it('flags an undated highlight name as mtime-dated', () => {
const p = parseArchiveFilename('0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
const p = parseArchiveFilename('4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
expect(p.date).toBe('2026-08-17');
expect(p.dateFromMtime).toBe(true);
});
it('does not flag a highlight that carries its own date', () => {
const p = parseArchiveFilename('2024-08-04_0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
const p = parseArchiveFilename('2024-08-04_4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
expect(p.date).toBe('2024-08-04');
expect(p.dateFromMtime).toBe(false);
});
@@ -184,7 +184,7 @@ describe('dateFromMtime', () => {
});
it('leaves the date empty rather than guessing when no mtime is given', () => {
const p = parseArchiveFilename('0ct0ber19 - C-IImhvpFuk.jpg', 'highlight')!;
const p = parseArchiveFilename('4utumn07 - C-IImhvpFuk.jpg', 'highlight')!;
expect(p.date).toBe('');
expect(p.dateFromMtime).toBe(false);
});
@@ -196,9 +196,9 @@ describe('dateFromMtime', () => {
*/
describe('canonicalItemId', () => {
it('collapses the two highlight naming conventions onto one id', () => {
const dir = 'story highlights - 0ct0ber19 - Heestory';
const undated = parseArchiveFilename('0ct0ber19 - C5dQPEYpd9W.mp4', 'highlight', 1)!;
const dated = parseArchiveFilename('2024-04-07_0ct0ber19 - 01 - C5dQPEYpd9W.mp4', 'highlight')!;
const dir = 'story highlights - 4utumn07 - Sunstory';
const undated = parseArchiveFilename('4utumn07 - C5dQPEYpd9W.mp4', 'highlight', 1)!;
const dated = parseArchiveFilename('2024-04-07_4utumn07 - 01 - C5dQPEYpd9W.mp4', 'highlight')!;
expect(scopedPostId(dated.postId, 'highlight', dir))
.toBe(scopedPostId(undated.postId, 'highlight', dir));
});
@@ -216,7 +216,7 @@ describe('canonicalItemId', () => {
});
it('leaves a shortcode that merely starts with digits alone', () => {
expect(canonicalItemId('0ct0ber19')).toBe('0ct0ber19');
expect(canonicalItemId('4utumn07')).toBe('4utumn07');
expect(canonicalItemId('C5dQPEYpd9W')).toBe('C5dQPEYpd9W');
expect(canonicalItemId('12345')).toBe('12345');
});
+2 -2
View File
@@ -7,13 +7,13 @@ import {
const REEL: GalleryDlSidecar = {
post_shortcode: 'Db-lNCoib9m', post_id: '3962768346034323302', type: 'reel',
date: '2026-08-13 11:00:44', post_date: '2026-08-13 11:00:44',
username: 'official_artms', fullname: 'Official ARTMS',
username: 'official_band', fullname: 'Official ARTMS',
description: 'Dancing in the spotlight', count: 1, likes: 22914,
};
const FEED_VIDEO: GalleryDlSidecar = { ...REEL, post_shortcode: 'DbdG9L9jU4m', type: 'post', count: 2 };
const HIGHLIGHT: GalleryDlSidecar = {
post_shortcode: 'BATVdRZi_3', post_id: '18099435932626935', type: 'highlight',
date: '2026-08-08 16:22:09', username: 'official_artms', count: 154,
date: '2026-08-08 16:22:09', username: 'official_band', count: 154,
};
describe('isGalleryDlSidecar', () => {
+1 -1
View File
@@ -16,7 +16,7 @@ import { Tab } from './routing';
/**
* The shortcode shared by every copy of a post, regardless of which source
* directory it came from. Sidecar ids are directory-scoped
* (`0ct0ber19 - reels/Cq8LrxSJAJE`); the trailing segment is the shortcode.
* (`4utumn07 - reels/Cq8LrxSJAJE`); the trailing segment is the shortcode.
*/
const shortcode = (post: Post): string => post.id.split('/').pop() ?? post.id;
+16 -16
View File
@@ -12,21 +12,21 @@ describe('parseRoute', () => {
});
it('reads a profile', () => {
expect(parseRoute('/0ct0ber19/')).toEqual({ archive: '0ct0ber19', tab: 'posts', post: null });
expect(parseRoute('/4utumn07/')).toEqual({ archive: '4utumn07', tab: 'posts', post: null });
});
it('reads a profile without a trailing slash', () => {
expect(parseRoute('/0ct0ber19')).toEqual({ archive: '0ct0ber19', tab: 'posts', post: null });
expect(parseRoute('/4utumn07')).toEqual({ archive: '4utumn07', tab: 'posts', post: null });
});
it('reads a tab', () => {
expect(parseRoute('/0ct0ber19/reels/').tab).toBe('reels');
expect(parseRoute('/0ct0ber19/saved/').tab).toBe('saved');
expect(parseRoute('/4utumn07/reels/').tab).toBe('reels');
expect(parseRoute('/4utumn07/saved/').tab).toBe('saved');
});
it('reads a post in Instagram form', () => {
expect(parseRoute('/0ct0ber19/p/Db5tIoRCcvm/')).toEqual({
archive: '0ct0ber19', tab: 'posts', post: 'Db5tIoRCcvm',
expect(parseRoute('/4utumn07/p/Db5tIoRCcvm/')).toEqual({
archive: '4utumn07', tab: 'posts', post: 'Db5tIoRCcvm',
});
});
@@ -41,8 +41,8 @@ describe('parseRoute', () => {
});
it('still understands the legacy query form', () => {
expect(parseRoute('/', '?a=0ct0ber19&t=reels&p=ABC')).toEqual({
archive: '0ct0ber19', tab: 'reels', post: 'ABC',
expect(parseRoute('/', '?a=4utumn07&t=reels&p=ABC')).toEqual({
archive: '4utumn07', tab: 'reels', post: 'ABC',
});
});
@@ -54,9 +54,9 @@ describe('parseRoute', () => {
describe('buildPath', () => {
it.each([
[{ archive: null, tab: 'posts', post: null }, '/'],
[{ archive: '0ct0ber19', tab: 'posts', post: null }, '/0ct0ber19/'],
[{ archive: '0ct0ber19', tab: 'reels', post: null }, '/0ct0ber19/reels/'],
[{ archive: '0ct0ber19', tab: 'posts', post: 'Db5tIoRCcvm' }, '/0ct0ber19/p/Db5tIoRCcvm/'],
[{ archive: '4utumn07', tab: 'posts', post: null }, '/4utumn07/'],
[{ archive: '4utumn07', tab: 'reels', post: null }, '/4utumn07/reels/'],
[{ archive: '4utumn07', tab: 'posts', post: 'Db5tIoRCcvm' }, '/4utumn07/p/Db5tIoRCcvm/'],
] as const)('builds %j', (route, expected) => {
expect(buildPath(route as any)).toBe(expected);
});
@@ -71,8 +71,8 @@ describe('buildPath', () => {
it('round-trips through parseRoute', () => {
for (const route of [
{ archive: '0ct0ber19', tab: 'posts' as const, post: null },
{ archive: '0ct0ber19', tab: 'reels' as const, post: null },
{ archive: '4utumn07', tab: 'posts' as const, post: null },
{ archive: '4utumn07', tab: 'reels' as const, post: null },
{ archive: 'Heejin_Bubble heejinmedia', tab: 'posts' as const, post: null },
]) {
expect(parseRoute(buildPath(route))).toEqual(route);
@@ -86,12 +86,12 @@ describe('postSlug / findPostBySlug', () => {
});
it('strips the sidecar directory from the slug', () => {
expect(postSlug(post('story highlights - u - Heestory/C5dQPEYpd9W'))).toBe('C5dQPEYpd9W');
expect(postSlug(post('story highlights - u - Sunstory/C5dQPEYpd9W'))).toBe('C5dQPEYpd9W');
});
it('resolves a slug back to its post', () => {
const posts = [post('AAA'), post('0ct0ber19 - reels/BBB', 'reels')];
expect(findPostBySlug(posts, 'BBB')?.id).toBe('0ct0ber19 - reels/BBB');
const posts = [post('AAA'), post('4utumn07 - reels/BBB', 'reels')];
expect(findPostBySlug(posts, 'BBB')?.id).toBe('4utumn07 - reels/BBB');
expect(findPostBySlug(posts, 'AAA')?.id).toBe('AAA');
});