The hand-run example had no pacing flags, so following it ran at gdl-sync.py's
defaults -- 6-10s and 1M -- which since yesterday is roughly half the caution
the wrapper applies. A documented manual procedure that is less careful than
the automation is backwards, and that is what this file said until now.
Adds the parts that were only ever in a transcript:
- how to detach a long run, and why `pkill -f <pattern>` over ssh kills your
own shell instead of the sync (the pattern matches the ssh command line --
this happened twice yesterday)
- that nothing reaches the archive until a run finishes, so killing one midway
is safe
- a "when a run fails" runbook in cost order: read the log, check the cookies,
then look at the browser. The first two are free.
The two results that look like failures and are not: rsync exit 23 is the
chown to rslsync failing because the ssh user is not root, and "No results"
just means a profile has no active story.
The cookie check is the one worth having written down -- exporting them and
looking at the shape of sessionid separates "Chrome decryption broke locally"
from "the account is in trouble" without spending a single request, and it is
what ruled out the wrong explanation yesterday.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
`full` was the misleading one: it is the abort-LIMITED run, the one that
deliberately stops enumerating a profile as soon as it reaches content already
held. Calling it "full" invited exactly the wrong assumption about coverage.
And `sweep` gave no hint that it was the exhaustive one.
full -> profiles every surface, --abort 50, ~40-60 requests
sweep -> full-sweep every surface, no abort, ~420 requests
The old names now exit 2 with a pointer to the new one rather than a bare
"unknown mode", since muscle memory and any stray crontab will still use them.
full-sweep's description now says what it costs. At ~420 requests it is the
same order as the run that preceded the 2026-08-21 scraping warning, spent to
catch a handful of retroactively edited posts, so the docs suggest running it
by hand when you mean to rather than leaving it on a timer. Its cadence was
never actually agreed.
Units renamed to match and re-verified with systemd-analyze; the old ones are
removed from the host. All three timers remain disabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
All three modes now use the pacing the 2026-08-22 runs used by hand after the
scraping warning -- 12-20s between requests, 5-10s between downloads, 500K --
against gdl-sync.py's defaults of 6-10 / 3-6 / 1M. Those values produced 0 400s
and 0 429s across 18 sources. An archive sync has no deadline; being slow is
free and being restricted is not.
Two things that made "scheduled" not mean much:
`stories` now uses --min-interval 8, not 20. At 20h the daily timer silently
did nothing whenever a manual run had happened the previous afternoon, which is
exactly what happened on 2026-08-21. The floor is there to stop an aborted
restart re-enumerating profiles -- minutes to hours -- and a stories fetch is
one request per profile, so 8h permits about twelve requests a day instead of
six. A run that skips every source now exits 75 and says so.
And the wrapper never reported failure at all. It ran the sync inside
`{ ...; exit $status; } | tee`, and the left side of a pipeline is a subshell,
so that exit set the subshell's status while the script returned tee's --
always 0. Every claim about cron mailing on failure was wrong. Fixed with
bash PIPESTATUS, hence the shebang change.
Verified with a stub sync across five cases: skipped-and-clean exits 75 with a
warning, a real failure propagates its own code, a normal run stays quiet, and
`full` skipping everything is not treated as an anomaly. Testing also caught
that the log name was only second-granular, so runs in the same second shared a
file and the skip check saw the previous run's output; the check now reads a
per-run temp file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Four claims went stale over two days and would each have misled someone
reading this cold:
- "the first unattended run has not happened yet" -- it has, and it did the
wrong thing: skipped every source on the 20h floor and reported success.
- "no daily stories run is scheduled" -- one exists, but is disabled after the
scraping warning, which is a materially different situation. Stories now
depend on someone remembering, and every day nobody runs it is a day gone.
- "--abort 50 is opt-in and nothing uses it yet" -- gdl-cron.sh full passes it
and both manual runs used it. sweep deliberately does not, which is the
point of sweep.
- the diff baseline in "Verifying a run" was pre-2026-08-22 and would have
made a correct run look like it had lost files.
Adds the gap that matters most for re-enabling: the scheduled modes still use
the default 6-10s pacing while the hand runs after the warning used 12-20s, so
turning the timers back on as they stand would make the automation less
careful than the humans were.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The pause section predicted official_artms and 0ct0ber19 stories would expire
uncollected. They did not: both surfaces were fetched by hand the next day at
roughly double the configured caution, with 0 400s and 0 429s. 16 story media
and 26 posts/reels media, 85 files into the archive.
That confirms the 400s were the challenge state rather than a block -- once
the interstitial was dismissed, the same endpoints served normally.
It does not retire the warning, and the section says so. Two hand-paced runs
are not evidence the old cadence was safe. It also flags the gap that matters
for whenever the timers go back on: the scheduled "full" mode still runs at
the default 6-10s pacing, so the automation would be less careful than the
manual runs that followed a warning.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Instagram flagged the account on 2026-08-21 -- not a suspension, an
interstitial at /accounts/scraping_warning/ saying it suspects automated
behaviour. Dismissed; account healthy. All three timers are disabled.
The sequence is worth keeping because each step misled differently. The daily
timer fired and exited 0 having fetched nothing: yesterday's manual run was
19h earlier, under the 20h floor, so a silent no-op on the one surface that
cannot be backfilled reported as success. Chrome was separately OOM-killed, so
no browser was running to notice. The recovery run then drew 400s from the
stories endpoint on all six profiles.
Cookies were exported and checked before assuming a block -- sessionid
well-formed and unexpired -- which cost nothing and ruled out the local
explanation. Only then did the browser reveal the interstitial.
Records the four things to fix before re-enabling: the floor defeats the daily
job, a skipped stories run must be loud rather than exit 0, the daily cadence
is itself the most machine-like pattern we run, and chrome-devtools.service
has Restart=no.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The first unattended run is tomorrow morning and nobody will be watching it,
so this writes down how to tell whether it worked -- entirely from the
journal, local logs and our own viewer's API, without spending a single
Instagram request to check.
Includes the per-profile file counts as of today so tomorrow's numbers have
something to diff against, and says plainly that a stories run adding zero
files is a normal result rather than a failure: profiles often have no active
story, and someone checking for the first time would reasonably read 0 as
broken.
Repeats the one rule that matters more than any tuning: a CDN 429 ends the
session. The warning order last time was 429 -> 400 on the highlights endpoint
-> suspension, so the runbook gives the disable command rather than leaving it
to judgement in the moment.
Also records what changed today and why, since none of it is recoverable from
the diffs: the run, the --abort measurement, the public/private split, and the
automation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
mattellite's key is now in the NAS's authorized_keys for agentapi, so
gdl-sync.py publishes straight to the archive. That was the last manual step:
the 2026-08-20 run published locally and rsynced by hand with sshpass, which
no scheduled job could do. Verified end to end -- key auth, then a dry-run
rsync from the fetch host into the archive root.
Three timers, installed and enabled: stories daily, full monthly, sweep
quarterly. A templated gdl-sync@.service takes the mode as its instance name.
Settings that are load-bearing rather than decoration:
RandomizedDelaySec=45m a job firing at exactly 09:00 daily is obviously a
machine; list-timers now shows 09:36, not 09:00
Persistent=true catch up a run the host slept through -- cron just
skips, and a skipped stories run is content gone
TimeoutStartSec=infinity a sweep runs for hours at this pacing; the 90s
default would kill it mid-fetch
Two things caught while installing, both documented: month names are invalid
in OnCalendar's date field (Jan,Apr,Jul,Oct-07 is rejected -- use numerics,
and check with systemd-analyze calendar), and `systemctl --user` over
non-interactive ssh needs XDG_RUNTIME_DIR or it cannot find the bus.
Smoke-tested by starting the stories service: it spent zero Instagram requests
because the 20h floor skipped all six sources, which is exactly what a job
firing twice should do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
gdl-sync.py always needed a fistful of paths on the command line, and the
2026-08-20 run published to a local directory and then rsynced to the NAS by
hand -- fine interactively, impossible from cron. gdl-cron.sh takes one
argument (stories|full|sweep) and does the whole thing.
Staging is wiped every run deliberately: what we already hold is decided by
the skip-archive, never by what happens to be sitting in staging, so starting
empty is correct and keeps the publish to just the new files.
Documents the three cadences and what each is for -- notably that only `sweep`
notices carousels edited after we archived them, and that only `stories` loses
content if it is skipped, since stories cannot be backfilled.
Cron notes the two things that would silently break it: cron runs /bin/sh so
$RANDOM does not exist (use shuf), and `%` in a crontab line means newline.
systemd is offered as the better option because RandomizedDelaySec and
Persistent=true are built in -- a host asleep at 09:00 silently skips a cron
job, and a skipped stories run is content gone for good.
The publish step is still blocked on mattellite having no key on the NAS; that
is recorded under Outstanding as the one thing between this and unattended.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Nothing here is broken — these are decisions not made and cleanups not done,
written down before the session's context is lost.
The two that can actually cost something: no daily stories run is scheduled,
and stories are the one surface that cannot be backfilled; and the `seeded`
flags in the state file were reconstructed by hand from a log rather than
derived from the archive DB, so losing artms.db without also clearing the
state file would leave those sources permanently unseeded and re-download
everything.
Also corrects "Scanner work (not done yet)", which shipped in 53b1f80 —
the three .json shapes are told apart structurally in gallery-dl-sidecar.ts
and isReel comes from the sidecar's type. That file lives on main: it parses
archives at display time and is viewer code, not tooling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Putting it at the top of README.md worked, but it diverged a shared file: a
probe showed main editing the adjacent line conflicts on every merge. The
warning survived the conflict, so nothing was ever silently lost, but a file
that only exists on this branch has no such cost at all.
TOOLING.md carries the warning plus what CLAUDE.md would have said if it could
— the commands, the remote policy, and the two guards, which are local and
unversioned and so are absent from every fresh clone.
Shared files are byte-identical to main again: CLAUDE.md, README.md and
package.json. The divergence is now only files main has never had.
Trade-off worth knowing: gitea renders README.md on the branch page and does
not render TOOLING.md, so this warning is one click less visible than it was.
The pre-push hook, not the documentation, remains the guard that actually
stops a mistake.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This branch is the only place the fetching tooling still exists, and it names
the fetch host's public IP, the cookie's browser profile, the NAS path and the
accounts being archived. main's history was rewritten, the GitHub repo deleted
and recreated, and 22 ghcr images pruned to get that material out of public
view — one push of this branch undoes all of it.
Deliberately at the top of the README rather than in a separate file: gitea
renders README.md on the branch page, which is where someone actually looks
before pushing. That does diverge from main, but only by an addition, which a
merge cannot silently drop — unlike the CLAUDE.md/package.json case, where
main was DELETING lines this branch kept and every merge re-applied the
deletion. Worst case here is a trivial conflict if main edits the same region,
and the block says to keep both.
Also records that the pre-push hook is local and unversioned, since a fresh
clone is the realistic way this goes wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
`tsc` keeps comments by default, so the doc comment in archive-grouping.ts
describing the sidecar layout was emitted into dist-server and copied into the
runtime image. Every published container image on ghcr carries it — verified by
pulling the dist-server layer of :latest and grepping it:
app/src/lib/archive-grouping.js:10: * <user> -> posts (base)
That comment names real archived accounts, which is exactly what main was
redacted to remove, so the redaction was incomplete while the build kept
re-emitting them. The frontend was never affected: Vite strips comments, and
the 432K dist layer greps clean.
--removeComments takes dist-server from 0 comment lines. docs/ was never at
risk; the multi-stage build copies only dist/ and dist-server/ into runtime.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
main's history was redacted on 2026-08-20: every archived account name was
replaced throughout all 65 commits, because main is published to GitHub and
the list of accounts identifies the archive as surely as a hostname does. The
viewer's test fixtures now use synthetic names chosen to preserve exactly what
each case tests — a leading digit, an underscore, a dotted username, a
bracketed or dotted highlight title, and the trailing U+2800 that case 12
exists for.
The redaction swept these three docs too, since they were in the same history.
This puts the real names back, on the branch that is not published. The
scripts needed nothing: they are generic and never named an account.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
A superset branch only stays cheap if the files it shares with main are
untouched. Both diverged: package.json carried a jd2 script and CLAUDE.md
described the tooling, so every merge from main would have conflicted on them
forever, and main's removal commit would have kept trying to delete the
tooling entries here.
The tooling files themselves do not exist on main, so they can hold whatever
they like. The jd2 invocation moves into docs/jdownloader.md as a direct npx
call instead of an npm script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The scripts and docs for fetching from Instagram now live here rather than on
main, which is the branch published to GitHub. They carry things that do not
belong in a public repo: the fetch host's public IP, the browser profile path
the cookie is read from, the NAS archive path, and the list of accounts being
archived.
This branch is a superset of main — the viewer plus the tooling — so it can
take main's changes by merging, and the npm script and CLAUDE.md entries that
reference the tooling live here where the files actually exist.
Restored with the sync work from the 2026-08-20 run already in place: the
--abort flag, the corrected yt-dlp install advice, and the measurements behind
both.
Note that main's history was rewritten to strip these paths, so the tooling's
own per-file history does not exist on this branch. It is preserved on gitea
as pre-rewrite-20260820 and pre-rewrite-tooling-20260820.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The fetching scripts and their docs now live on the `tooling` branch, which
is not published to GitHub. This removes the two references that would
otherwise dangle here: the `jd2` npm script and the CLAUDE.md bullet
describing it.
The viewer's own gallery-dl support is untouched and stays here —
src/lib/gallery-dl-sidecar.ts and friends parse sidecars at display time and
are app code, not tooling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Two .pyc files under scripts/__pycache__ were committed at some point and have
been churning ever since — merely importing gdl-sync.py to check a config
rewrites them and dirties the tree, which is how they surfaced.
.gitignore had no Python entries at all, only Node ones. The files stay on
disk; this just untracks them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Docker Build and Publish / build-and-push (push) Failing after 10s
Ships the gallery-dl sidecar work to the viewer. The visible change is
reel classification: official_artms' Reels tab drops from 781 items to
360, because the sidecars say the other 421 are ordinary feed videos the
clips endpoint returns via include_feed_video. Directory-based
classification counted them all as reels.
Also in this release: dates ranked by source rather than scan order, and
highlight items no longer appearing twice when the archive holds them
under both JDownloader naming conventions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JDownloader wrote story-shaped names for highlights during one period of
its life, so the same item exists on disk as both
0ct0ber19 - C5dQPEYpd9W.mp4
2024-04-07_0ct0ber19 - 01 - C5dQPEYpd9W.mp4
which parsed to the ids "C5dQPEYpd9W" and "01 - C5dQPEYpd9W" -- two posts
for one item. The leading ordinal is a position within a day's stories
and carries nothing the shortcode does not, so story and highlight ids
drop it. Post ids are untouched, since those are permalinks.
Measured on the two real files, same archive, cache cleared between:
without the fix the profile reads "Heestory - 2 items", with it
"Heestory - 1 item".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit had this backwards: the sidecar date was only
consulted when the existing date came from an mtime, so a filename date
silently outranked what Instagram itself reported.
The order is sidecar, then filename, then mtime -- metadata first,
mtime last, since mtime is when the file hit disk and says nothing about
when the post was made. Ties keep the incumbent so two equally
authoritative files cannot flip a post's date by scan order.
Extracted to src/lib/post-dates.ts rather than left inline, because the
rule is easy to state and easy to get wrong -- the tests include an
order-independence case that would have caught the original mistake.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The .json sidecars published with the ARTMS fetch were inert: the scanner
fed them through the Instaloader path, where `node.edge_media_to_caption`
and `checkIsStory`'s `product_type` are both absent, so nothing happened.
They are now recognised structurally -- flat, with post_shortcode and
type, and none of the markers the other two JSON shapes carry -- and used
for three things:
- `type` sets post.isReel, which post-tabs prefers over every fallback.
This is Instagram's own classification and it disagrees with ours a
lot: of 781 items in "official_artms - reels", the sidecars say only
360 are reels. The other 421 are feed videos the clips endpoint returns
via include_feed_video, and the directory-based rule counted them all.
- `description` fills the caption where no .txt exists.
- `date` dates a post whose filename could not.
Also fixes date precedence. Only JDownloader highlights lack a date in
the filename, so parseArchiveFilename now marks those as mtime-derived
and the scanner lets any real date replace them -- previously the date
depended on which file the scan reached first.
Verified against real published files: a directory of three type=post and
three type=reel renders 6 in the grid and exactly the 3 reels in the
Reels tab.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every claim in docs/gallery-dl.md was measured against the live site and
the archive rather than taken from documentation, because two of the
assumptions turned out to be wrong.
The safety model is the reason the config looks the way it does.
gallery-dl has two API backends: the graphql one issues a request PER
POST for every video and carousel -- the pattern that got this account
banned via Instaloader -- while the default rest one paginates listings
at 30-50 items and carries carousel_media, video_versions and
product_type inline. A 300-post profile costs ~10 requests.
Findings worth recording:
- JD2 stamped filenames in desktop LOCAL time (US Eastern), not UTC.
Across 212 comparable posts: UTC 19 mismatches, UTC-5 10, UTC-4 zero.
{date:Olocal/%Y-%m-%d} reproduces it; the trailing separator must be
omitted or it lands in the strftime format.
- A profile's reels tab returns collab reels owned by OTHER accounts, so
the directory must be forced with -D. JD2 did the same: chuuo3o and
official_artms filenames sit inside "0ct0ber19 - reels".
- Stories and highlights need per-item {shortcode}; {post_shortcode} is
the reel's id and is shared by every item. {date} is per-item, verified
on a 154-item highlight with distinct times.
- gallery-dl reproduces JD2's caption .txt exactly, including writing
nothing for an empty caption and omitting the trailing newline.
- The json sidecar needs `include`, not `fields`; `fields` silently does
nothing in mode:json and leaks audio_user blobs. It yields `type`
(post/reel) -- Instagram's own flag, which can retire the lone-video
heuristic once the scanner reads it.
Naming differences between the two tools are cosmetic: EXPORT_RE already
makes the index optional and parseInt normalises zero-padding, so a mixed
archive parses identically. Tests pin that down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Docker Build and Publish / build-and-push (push) Failing after 10s
The header rendered allPosts.length, which is pre-dedupe — 0ct0ber19
showed "303 posts" over a 300-tile grid. Instagram's counter equals its
grid, so count the grid.
Also correct CLAUDE.md. v1.7.0 claimed the grid holds everything "as on
Instagram"; Instagram actually includes a reel in the grid only when the
creator shared it to feed, per post. Measured live: official_artms has
21 reels in its first 34 grid tiles, 0ct0ber19 has 1 in 214. Archives
carry no such flag, so showing everything approximates the behaviour
rather than reproducing it.
Records the DOM trap that caused the wrong reading in the first place:
grid reels link to /reel/<code>/, not /<user>/p/<code>/.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Docker Build and Publish / build-and-push (push) Failing after 9s
The Posts tab filtered reels out, so the grid was not the archive — it
was the archive minus its videos. For `for.heejin` that hid 533 of 1225
posts; for `loonatheworld`, 1100 of 3813. On Instagram the grid holds
everything and the Reels tab is a filtered view of that same set.
Extract the tab logic to src/lib/post-tabs.ts so the reel heuristic is
testable outside the component, and add dedupePostCopies: the jd2 flow
crawls the profile URL and the /reels URL separately because the profile
page misses some reels, so the two overlap and a reel can land on disk
twice. Those are two posts with distinct directory-scoped ids, which the
grid would now render side by side; the reels-source copy wins so the
survivor is still recognised as a reel.
Deciding what *is* a reel stays a guess for most archives. Instagram
marks it with product_type ("clips" vs "feed" vs "igtv" — all three are
GraphVideo, and aspect ratio does not separate them), but only newer
Instaloader captures carry it: 1101 of gibiofficial's 5919 sidecars, and
only 2 marked clips. JDownloader archives carry none, so those still
fall back to treating a lone video as a reel.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two failures in this session were expensive because nothing recorded them:
- The CSP must keep 'wasm-unsafe-eval' and connect-src data:, because the xz
decompressor for Instaloader sidecars is WebAssembly embedded as a data: URL.
Removing either breaks decoding with a bare "Failed to fetch" and no stack,
and the visible symptom is silent metadata loss rather than an error.
- The service worker precaches index.html with its headers, so a server-only
change never reaches installed clients. The version compiled into the client
is what forces the precache to turn over each release; it is load-bearing,
not decoration.
Also notes that the Vite dev server sends none of these headers, so CSP and PWA
behaviour must be verified against a built dist/ served by server.js, and that
the live archive root is now the archives/ subdirectory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 10s
The service worker precaches index.html together with its response headers, so
a server-only change never reaches an installed client: the client build is
byte-identical, the precache manifest is unchanged, and the worker has no
reason to update. That is why the CSP fix in 1.6.1 did not reach a browser that
already had the app cached — it kept replaying a cached shell carrying the old,
broken CSP, indefinitely.
The release version is now compiled into the client, which makes every release
change the bundle hash, hence index.html, hence its precache revision, hence
sw.js itself — the bytes browsers compare to decide whether to update. Verified
by bumping only the version: index-DYufL2Fa.js -> index-60N_3d5j.js, with the
new name carried into the sw.js manifest.
It also surfaces in the footer, so the deployed version is visible without
digging through devtools.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 10s
The xz decompressor for Instaloader's .json.xz sidecars is WebAssembly,
embedded as a data: URL that it fetches at startup. The CSP added in 1.3.0
blocked both halves of that:
fetch('data:application/wasm;...') -> TypeError: Failed to fetch
WebAssembly.instantiate(...) -> CompileError: violates script-src 'self'
The first surfaces through new Response(stream).json() as a bare "Failed to
fetch" with no stack, which reads like a network fault and is why this was
mis-diagnosed twice. Vite's dev server never sends the CSP, so it reproduced
only in production — every Instaloader archive silently lost its captions,
story flags and profile metadata from 1.3.0 onward.
script-src now allows 'wasm-unsafe-eval', which permits WebAssembly compilation
without permitting eval() of JavaScript, and connect-src allows data: for the
embedded module.
Verified against the production bundle: rivvsofficial goes from 188 posts / 0
followers / no stories to 68 posts, 120 stories, 10,337 followers and its real
name, bio and link — with zero decode errors.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 10s
Instaloader metadata was silently lost
Every .json.xz failed with "Failed to fetch" during a scan, though the same URL
fetched fine on its own. RemoteArchiveFile.stream() started a fetch, piped the
body into a TransformStream and returned the readable immediately — nothing
caught a fetch rejection, and the decompressor stops reading at the end of the
xz member, so the response body was never drained or cancelled. Across ~190
sidecars that exhausted the connection pool.
Everything Instaloader archives carry lives in those files, so the failure was
invisible but total. rivvsofficial reported 188 posts, no stories, 0 followers
and a placeholder bio; it now reports 68 posts, 120 stories, 10,337 followers
and the real name, bio and link — 68 + 120 = 188, matching the sidecars exactly
(106 GraphStoryVideo + 14 GraphStoryImage = 120).
These sidecars are a few KB, so they are now read into memory before
decompressing. stream() was left unused by that change and is removed from the
interface and both implementations rather than kept as a trap.
Arrow keys page the carousel
They moved between posts, which contradicted the arrows drawn on the carousel
itself. Arrows now page slides; , and . move between posts, alongside the side
buttons.
Backdrop cross-fade
AnimatePresence had no exit variant, so the outgoing scan backdrop was removed
instantly while its replacement faded in over 1.5s, exposing the pale page
behind it as a white flash. Layers now stack: the outgoing image holds full
opacity until covered, and the 0.4 moved onto the group so overlapping layers
don't darken as they cross. Measured over a real scan: 152 cross-fades with a
layer always opaque, except the opening fade-in where nothing is underneath.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Covers why fetching goes through JDownloader rather than Instaloader (the
instagram.com vs CDN split, and what the metadata gap actually costs), the
settings that matter, cookie handling, the two-URL workflow, jd2-sync usage,
the expected on-disk layout, and what to do when something breaks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 11s
The archive root was filtered by prefix, but the recursive walk below it was
not, so anything inside a profile directory got indexed. NAS filesystems put
sidecar metadata *inside* every folder rather than only at the share root:
Synology writes @eaDir (thumbnails and indexing data), #recycle holds
deletions, .sync is Resilio state. On the live share those account for 12,516
of 123,023 files.
None currently sit inside a profile directory, so nothing was miscounted yet —
but the moment that share gets indexed for Photos, every generated thumbnail
would be counted as archive media and stat'd one by one over the network, which
is the cost the index exists to avoid.
One isSystemDirectory rule now applies at every level, and the root listing uses
it too instead of keeping a second copy of the pattern.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
The manual flow is: paste a profile URL into JDownloader, paste the /reels URL
separately (the profile page misses some reels), set the output folder by hand,
repeat per profile. scripts/jd2-sync.ts emits one crawljob per source with the
folder already pointed at the right directory, so folder-watch picks up the
whole batch at once.
Profiles and sidecars are derived with the same grouping logic the server uses,
so output folders always match what the viewer expects to find. --download-base
maps the path for a JDownloader running on another machine (Windows paths
included), since it typically runs on a desktop against the share.
Directories that aren't Instagram profiles are skipped: an archive root also
collects tool output and exports from other services, and pointing a crawl at
those spends requests on instagram.com to be told the profile doesn't exist —
exactly the traffic worth not spending. Filtering is by username shape, plus
--skip and a .jd2ignore file for names that look like usernames but aren't.
Defaults are conservative: chunks=1, because multi-chunk ranged requests are the
one CDN-side pattern that doesn't resemble a browser, and links park in the
LinkGrabber for review rather than auto-starting.
Only posts and reels are emitted; highlight URLs need a numeric id and story
URLs expire, so those stay manual.
Format verified against JDownloader's own explain.txt for the folderwatch
extension, read from the daily SVN mirror rather than one of the decade-stale
GitHub copies.
Also refreshes CLAUDE.md, whose URL-state section still described the query
parameters replaced in 1.4.0, and documents the mobile feed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 10s
Tapping a post on a phone now opens a real feed page — header, media, actions,
caption, next post peeking in below — scrolled with the browser's own vertical
scrolling rather than swipe gestures. Desktop keeps the modal, where a centred
sheet with side arrows suits a pointer.
Only a window of posts is mounted: a profile here holds up to 1129 posts and
mounting them all would mean as many full-size images. The window grows in both
directions as you scroll. Growing upwards shifts everything below it, so the
scroll offset is corrected in the same frame, before paint — measured against
the real archive, an anchored post moves exactly one screen per scroll with no
jump.
Only the post crossing the viewport centre plays its video; the rest stay
paused, so a feed of reels doesn't play ten at once. The URL tracks that same
post, so scrolling updates /<archive>/p/<shortcode>/ the way Instagram does,
and the back button returns to the grid with its scroll position intact.
Feed video sizes to the container width rather than its intrinsic size: a
<video> reports 300x150 until metadata loads, which made it render narrow and
then jump to full width. It also gets a taller height ceiling than the modal so
ordinary portrait media fills the width instead of sitting in side bars.
The carousel is extracted into a shared MediaCarousel used by both surfaces, so
horizontal paging behaves identically; touch-action keeps vertical scrolling
passing through to the feed. PostModal loses its now-dead mobile swipe branch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 10s
Navigation gestures
Horizontal swipe used to advance the carousel and then, on the last slide,
fling you into the next post — one gesture meaning two things. Horizontal is
now carousel-only. On touch, vertical swipe moves between posts (down on the
first post still dismisses, keeping drag-to-close where it can't mean
"previous"). Desktop keeps the arrows outside the modal.
URLs
Permalinks now mirror Instagram:
/<archive>/ profile
/<archive>/reels/ tab
/<archive>/p/<shortcode>/ post
A post URL carries no tab, as on Instagram; the tab is re-derived from the
post's source, so opening a reel link lands on the Reels tab with next/prev
paging through reels. Sidecar posts keep directory-scoped ids internally but
expose only the shortcode. The old ?a=&t=&p= form is still parsed so existing
links keep working, and reserved prefixes (api, archives, assets…) can never be
mistaken for a profile name.
Animations
Adds a shared motion vocabulary tuned to feel native: critically damped springs
rather than fixed-duration easing, and gestures hand their exit velocity to the
animation so a flick continues instead of restarting. Post transitions animate
along the axis the input implies — vertical for a swipe, horizontal for the
arrows. Modal and story viewer present/dismiss with a scale, tiles and
highlight circles get touch-down feedback, and prefers-reduced-motion is
honoured throughout.
Adds 22 routing tests (58 total).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 11s
The prev/next arrows are fixed to the viewport edges while the modal grows to
fill the available width, so below roughly 1200px the modal slid underneath
them and a white chevron landed on the white caption panel — invisible until
hovered. The overlay now reserves a horizontal gutter (md:px-16 lg:px-24) so
the arrows always sit outside the modal, and they get a solid white pill with a
dark chevron so they read against anything behind them. Verified clearing the
modal at 768, 1024, 1440 and 1920px.
The caption sidebar shrinks to w-80 at md so the narrower modal doesn't squeeze
the media pane.
Also adds overscroll-contain to the overlay: the wheel previously chained
through to the post grid behind it, scrolling the background while a post was
open.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 10s
Media in the post modal used w-full/h-auto, so a portrait video or image grew
taller than the screen (a 720x1280 reel rendered 768x1365 in a 786px viewport)
and forced the modal to scroll. Full view now caps height to the viewport minus
the modal's own padding. Video sizes to its own aspect within the cap so a
portrait clip isn't letterboxed edge to edge; images keep filling the modal
width and only gain a height ceiling.
Opening the modal or a story reel is a user gesture, so playback now starts
unmuted and only falls back to muted if the browser actually refuses the
play() promise — previously it always started muted, and the earlier
muted-by-default fix meant a blocked video could stall the story progress bar.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
os.userInfo() throws ERR_SYSTEM_ERROR (uv_os_get_passwd) for a UID that has no
/etc/passwd entry, which is exactly what `docker run --user 1234:1234` produces
— the very workaround the README recommends. Combined with the switch to a
non-root image user, this crashed the server on startup for any deployment that
needed a custom UID to read its archives.
Also document the non-root default and the /cache index volume.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Security
- Fix path traversal in GET /api/archives/:name/files. Express decodes route
params after segment matching, so `..%2f..%2fetc` escaped ARCHIVES_DIR and
returned a recursive listing of arbitrary directories.
- Add CSP and baseline security headers; disable x-powered-by.
- Stop baking GEMINI_API_KEY into the client bundle (the SDK was unused).
- Run the container as `node` instead of root.
Performance
- Add a directory-mtime-keyed archive index, warmed in the background and
persisted. Listing 110k files went from ~52s to ~0.1s; the largest archive
(24k files) serves in ~0.3s. Per-file stat over CIFS costs ~1.4ms and does
not parallelise, so it is now done once rather than per request.
- Build media URLs from the File directly instead of
`new Blob([await file.arrayBuffer()])`, which read every media file fully
into memory (a 20GB archive tried to become 20GB of resident blobs).
- Track and revoke object URLs; previously none were ever revoked.
- Give `requestThumbnail` a stable identity so a completed thumbnail stops
re-running the effect in every mounted thumbnail.
- Namespace IndexedDB keys so listing archives no longer deserializes every
cached thumbnail blob, and thumbnails no longer collide across archives.
- Serve real file sizes: RemoteArchiveFile was constructed with size 0, which
silently disabled high-res thumbnailing for every server archive.
Correctness
- Local archives cached media as blob: URLs, which die with the document, so
a cached local archive restored as an archive of broken images. Media now
carries a stable path and is rehydrated from a persisted directory handle
(File System Access API), falling back to re-prompting for the folder.
- Fix permalinks: the URL-writing effect erased ?a= on mount before the
archive list arrived to consume it, so deep links never resolved.
- Make cache invalidation detect nested changes via a directory signature.
- Add an error boundary and tolerate unparseable dates, which previously
threw a RangeError and blanked the app.
- Default video to muted so autoplay is not blocked by Safari/Firefox.
Features
- Fold sidecar directories into their base profile: `<user> - reels`,
`story - <user>` and `story highlights - <user> - <title>` now appear as
reels, the story ring and Instagram-style highlight circles rather than as
separate archives.
Housekeeping
- Add @types/react; React was previously type-checked against its JavaScript
source, so `npm run lint` gave almost no type safety on components.
- Vendor fonts and PWA icons locally; the app made third-party CDN requests
despite advertising offline support and local-only processing.
- Drop unused better-sqlite3 (a native module that broke `npm install`).
- Add vitest with 36 tests over the filename and directory-naming rules.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
Docker Build and Publish / build-and-push (push) Failing after 9s
Key changes:
- Updated README.md and GEMINI.md with details on background thumbnailing and inter-post preloading.
- Documented persistent local archive caching and smart profile fallback features.
- Added dist-server/ to .gitignore.
- Restored missing feature descriptions and troubleshooting tips in README.
Key changes:
- Enabled full metadata caching for local folder archives, allowing them to load instantly from IndexedDB without re-uploading.
- Implemented oldest-image fallback for profiles missing an explicit profile picture.
- Restored folder-name-to-username detection for local archive uploads.
- Optimized scan indexing to track all image files for fallback use.
Key changes:
- Corrected logic in PostThumbnail to prevent blur effects on images smaller than 1MiB.
- Restored the white glass aesthetic to the scanning dashboard with improved contrast and transparency.
- Optimized scanning background transitions to ensure a smooth, flicker-free crossfade.
Key changes:
- Added Web Worker for background image thumbnailing with a 1MiB threshold to optimize CPU/memory usage.
- Implemented a serial task queue for memory-safe high-res image processing, preventing OOM crashes.
- Added inter-post preloading in the modal for seamless 'Previous/Next' navigation.
- Refined scanning UI with double-buffering and a dark background to completely eliminate white flashes.
- Renamed project to 'instaarchive-viewer' in package.json.
- Fixed 'Open image in new tab' by denylisting /archives and /api in PWA config.
Summary of changes:
- Extracted archive scanning logic into a modular 'useArchiveScanner' hook for better maintainability and performance.
- Refined PostModal carousel with intelligent media preloading and smoother, jitter-free transitions.
- Optimized image rendering with 'decoding=async' and removed 'black flashes' between slide changes.
- Updated PWA configuration to 'autoUpdate' with hourly periodic checks for fresh content.
- Fixed several bugs including stories sorting, permalink parameter cleanup, and profile metadata cache restoration.
- Comprehensive updates to documentation (README.md and GEMINI.md) reflecting the new architecture.
ergosteurandClaude Opus 5