Two additions to the metadata sidecars, following up on the 2026-09-01
collab/dedup investigation already documented here:
- coauthors added to the post-level JSON's include list -- a direct,
native signal for collab detection instead of inferring it from the
filename/directory identity mismatch also documented that day.
- width, height, width_original, height_original and tagged_users get
their own per-file sidecar (event: "file", one JSON per downloaded
media item, named "<filename>.json"), since a carousel's items can
each have different dimensions and tags -- data the existing
post-level JSON has no way to represent. owner is deliberately left
out, same reasoning as audio_user's existing exclusion.
Verified against a live re-fetch of an already-archived carousel:
correct per-item data came back, zero media re-downloaded (skip-archive
still applies; only the new sidecars are new files). 3 new tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
Three findings from the 2026-09-01 collab-detection/VP9 investigation in
Cosmo-Live-Downloads, all rooted in gallery-dl's own scraping output:
- Collab posts' JSON is filed under the true owner's identity but the
media file under the scraped account's — plus coauthors is a native,
uncaptured signal that already excludes the owner for us.
- The shared archive-db dedups by media_id across all profiles, not per
profile, so a post reachable from two profiles' listings only gets its
media written into whichever profile's crawl reached it first (510 such
posts found archive-wide).
- Some video_versions entries are VP9; format selection is codec-blind,
and Safari's VP9 decode only wires into its WebM demuxer, never MP4.
Also commits the two saved raw API response examples and the missing-
captions CSV generated during that investigation.
Was only stated as supporting detail inside the dedup-fix narrative --
someone looking specifically for "where does a shared reel end up?"
had to read through the whole bug story to find it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
reels-sync.sh is the single-command version of the two-step pipeline:
scrape a profile's reels tab, then fetch and publish whatever's new,
with the same hand-paced settings gdl-cron.sh uses. Takes a bare
username or a full profile URL. Exits clean without touching
gdl-sync.py at all when a profile has nothing new.
Also fixes a real inefficiency in reels-scrape.py's dedup, found by
running the new script twice in a row: checking only the scraped
profile's own directories missed that a shortcode already existed
under its true owner elsewhere in the archive (reposts/collabs by
other tracked accounts), so 9 already-held reels got re-fetched for no
reason. Shortcodes are globally unique, so dedup now checks every
archived profile's listing -- all local requests to the viewer's own
API, never instagram.com, so this costs nothing on the budget that
actually matters. See TOOLING.md for the full story.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
Two additions:
- A "Reels" section in TOOLING.md with the exact two-step command
sequence for scraping and fetching reels by hand, since it's a
separate manual tool, not wired into gdl-cron.sh.
- ~/.config/gallery-dl/config.json on mattellite, so a plain
`gallery-dl <url>` typed by hand -- outside gdl-sync.py entirely --
still gets the hand-paced sleep/rate/api settings instead of
gallery-dl's own faster defaults. Documented in TOOLING.md since the
file itself isn't tracked (host-local, like artms.db).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
gallery-dl's dedicated reels extractor POSTs to /api/v1/clips/user/,
which now 302-redirects for this account -- confirmed across multiple
profiles, hours apart, with a freshly-warmed session and a correct
X-IG-WWW-Claim header (ruled out as the cause). The reels tab itself
loads fine in a real, already-signed-in browser, so reels-scrape.py
drives that same Chrome via its loopback CDP port, scrolls the reels
tab like a person would, and scrapes /reel/<code>/ links out of the
rendered page instead of calling the blocked endpoint at all.
It only finds shortcodes -- deduped against the archive via the same
--index gdl-sync.py already uses -- and prints new post URLs. Feeding
many of those into gdl-sync.py needed two small additions: a
--post-urls-file so the list doesn't have to become a giant argv, and
inter-item pacing in run_post_urls (each --post-url was its own
subprocess with nothing pacing the gap between them).
Verified end to end against zindoriyam: 26 reels found, 16 already
archived, 10 new ones fetched and published cleanly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
One example per way of selecting what to fetch (urls-file, --profile,
--all, --post-url), plus the routine/stories/full-sweep pacing choices,
so the flags don't have to be reverse-engineered from the arg list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
Lets an out-of-band link (shared by someone, not one of the tracked
profiles) be pulled in directly by URL, filed under its owner's account
like any other post. Bypasses profile planning, archive-db seeding, and
the --min-interval floor entirely, since it's a single request rather
than a recurring surface to budget against.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
Two commands to get new content onto the site, what the two alarming-but-fine
messages mean, and the two that mean stop. Everything else in this file is
reference material for when something is wrong; this is the part needed when
nothing is.
Says plainly that profiles covers everything but is not a substitute for
stories, because a story posted and expired between two profiles runs is gone
regardless of how thorough the later run is.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The hand-run example had no pacing flags, so following it ran at gdl-sync.py's
defaults -- 6-10s and 1M -- which since yesterday is roughly half the caution
the wrapper applies. A documented manual procedure that is less careful than
the automation is backwards, and that is what this file said until now.
Adds the parts that were only ever in a transcript:
- how to detach a long run, and why `pkill -f <pattern>` over ssh kills your
own shell instead of the sync (the pattern matches the ssh command line --
this happened twice yesterday)
- that nothing reaches the archive until a run finishes, so killing one midway
is safe
- a "when a run fails" runbook in cost order: read the log, check the cookies,
then look at the browser. The first two are free.
The two results that look like failures and are not: rsync exit 23 is the
chown to rslsync failing because the ssh user is not root, and "No results"
just means a profile has no active story.
The cookie check is the one worth having written down -- exporting them and
looking at the shape of sessionid separates "Chrome decryption broke locally"
from "the account is in trouble" without spending a single request, and it is
what ruled out the wrong explanation yesterday.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
`full` was the misleading one: it is the abort-LIMITED run, the one that
deliberately stops enumerating a profile as soon as it reaches content already
held. Calling it "full" invited exactly the wrong assumption about coverage.
And `sweep` gave no hint that it was the exhaustive one.
full -> profiles every surface, --abort 50, ~40-60 requests
sweep -> full-sweep every surface, no abort, ~420 requests
The old names now exit 2 with a pointer to the new one rather than a bare
"unknown mode", since muscle memory and any stray crontab will still use them.
full-sweep's description now says what it costs. At ~420 requests it is the
same order as the run that preceded the 2026-08-21 scraping warning, spent to
catch a handful of retroactively edited posts, so the docs suggest running it
by hand when you mean to rather than leaving it on a timer. Its cadence was
never actually agreed.
Units renamed to match and re-verified with systemd-analyze; the old ones are
removed from the host. All three timers remain disabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
All three modes now use the pacing the 2026-08-22 runs used by hand after the
scraping warning -- 12-20s between requests, 5-10s between downloads, 500K --
against gdl-sync.py's defaults of 6-10 / 3-6 / 1M. Those values produced 0 400s
and 0 429s across 18 sources. An archive sync has no deadline; being slow is
free and being restricted is not.
Two things that made "scheduled" not mean much:
`stories` now uses --min-interval 8, not 20. At 20h the daily timer silently
did nothing whenever a manual run had happened the previous afternoon, which is
exactly what happened on 2026-08-21. The floor is there to stop an aborted
restart re-enumerating profiles -- minutes to hours -- and a stories fetch is
one request per profile, so 8h permits about twelve requests a day instead of
six. A run that skips every source now exits 75 and says so.
And the wrapper never reported failure at all. It ran the sync inside
`{ ...; exit $status; } | tee`, and the left side of a pipeline is a subshell,
so that exit set the subshell's status while the script returned tee's --
always 0. Every claim about cron mailing on failure was wrong. Fixed with
bash PIPESTATUS, hence the shebang change.
Verified with a stub sync across five cases: skipped-and-clean exits 75 with a
warning, a real failure propagates its own code, a normal run stays quiet, and
`full` skipping everything is not treated as an anomaly. Testing also caught
that the log name was only second-granular, so runs in the same second shared a
file and the skip check saw the previous run's output; the check now reads a
per-run temp file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Four claims went stale over two days and would each have misled someone
reading this cold:
- "the first unattended run has not happened yet" -- it has, and it did the
wrong thing: skipped every source on the 20h floor and reported success.
- "no daily stories run is scheduled" -- one exists, but is disabled after the
scraping warning, which is a materially different situation. Stories now
depend on someone remembering, and every day nobody runs it is a day gone.
- "--abort 50 is opt-in and nothing uses it yet" -- gdl-cron.sh full passes it
and both manual runs used it. sweep deliberately does not, which is the
point of sweep.
- the diff baseline in "Verifying a run" was pre-2026-08-22 and would have
made a correct run look like it had lost files.
Adds the gap that matters most for re-enabling: the scheduled modes still use
the default 6-10s pacing while the hand runs after the warning used 12-20s, so
turning the timers back on as they stand would make the automation less
careful than the humans were.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The pause section predicted official_artms and 0ct0ber19 stories would expire
uncollected. They did not: both surfaces were fetched by hand the next day at
roughly double the configured caution, with 0 400s and 0 429s. 16 story media
and 26 posts/reels media, 85 files into the archive.
That confirms the 400s were the challenge state rather than a block -- once
the interstitial was dismissed, the same endpoints served normally.
It does not retire the warning, and the section says so. Two hand-paced runs
are not evidence the old cadence was safe. It also flags the gap that matters
for whenever the timers go back on: the scheduled "full" mode still runs at
the default 6-10s pacing, so the automation would be less careful than the
manual runs that followed a warning.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Instagram flagged the account on 2026-08-21 -- not a suspension, an
interstitial at /accounts/scraping_warning/ saying it suspects automated
behaviour. Dismissed; account healthy. All three timers are disabled.
The sequence is worth keeping because each step misled differently. The daily
timer fired and exited 0 having fetched nothing: yesterday's manual run was
19h earlier, under the 20h floor, so a silent no-op on the one surface that
cannot be backfilled reported as success. Chrome was separately OOM-killed, so
no browser was running to notice. The recovery run then drew 400s from the
stories endpoint on all six profiles.
Cookies were exported and checked before assuming a block -- sessionid
well-formed and unexpired -- which cost nothing and ruled out the local
explanation. Only then did the browser reveal the interstitial.
Records the four things to fix before re-enabling: the floor defeats the daily
job, a skipped stories run must be loud rather than exit 0, the daily cadence
is itself the most machine-like pattern we run, and chrome-devtools.service
has Restart=no.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The first unattended run is tomorrow morning and nobody will be watching it,
so this writes down how to tell whether it worked -- entirely from the
journal, local logs and our own viewer's API, without spending a single
Instagram request to check.
Includes the per-profile file counts as of today so tomorrow's numbers have
something to diff against, and says plainly that a stories run adding zero
files is a normal result rather than a failure: profiles often have no active
story, and someone checking for the first time would reasonably read 0 as
broken.
Repeats the one rule that matters more than any tuning: a CDN 429 ends the
session. The warning order last time was 429 -> 400 on the highlights endpoint
-> suspension, so the runbook gives the disable command rather than leaving it
to judgement in the moment.
Also records what changed today and why, since none of it is recoverable from
the diffs: the run, the --abort measurement, the public/private split, and the
automation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
mattellite's key is now in the NAS's authorized_keys for agentapi, so
gdl-sync.py publishes straight to the archive. That was the last manual step:
the 2026-08-20 run published locally and rsynced by hand with sshpass, which
no scheduled job could do. Verified end to end -- key auth, then a dry-run
rsync from the fetch host into the archive root.
Three timers, installed and enabled: stories daily, full monthly, sweep
quarterly. A templated gdl-sync@.service takes the mode as its instance name.
Settings that are load-bearing rather than decoration:
RandomizedDelaySec=45m a job firing at exactly 09:00 daily is obviously a
machine; list-timers now shows 09:36, not 09:00
Persistent=true catch up a run the host slept through -- cron just
skips, and a skipped stories run is content gone
TimeoutStartSec=infinity a sweep runs for hours at this pacing; the 90s
default would kill it mid-fetch
Two things caught while installing, both documented: month names are invalid
in OnCalendar's date field (Jan,Apr,Jul,Oct-07 is rejected -- use numerics,
and check with systemd-analyze calendar), and `systemctl --user` over
non-interactive ssh needs XDG_RUNTIME_DIR or it cannot find the bus.
Smoke-tested by starting the stories service: it spent zero Instagram requests
because the 20h floor skipped all six sources, which is exactly what a job
firing twice should do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
gdl-sync.py always needed a fistful of paths on the command line, and the
2026-08-20 run published to a local directory and then rsynced to the NAS by
hand -- fine interactively, impossible from cron. gdl-cron.sh takes one
argument (stories|full|sweep) and does the whole thing.
Staging is wiped every run deliberately: what we already hold is decided by
the skip-archive, never by what happens to be sitting in staging, so starting
empty is correct and keeps the publish to just the new files.
Documents the three cadences and what each is for -- notably that only `sweep`
notices carousels edited after we archived them, and that only `stories` loses
content if it is skipped, since stories cannot be backfilled.
Cron notes the two things that would silently break it: cron runs /bin/sh so
$RANDOM does not exist (use shuf), and `%` in a crontab line means newline.
systemd is offered as the better option because RandomizedDelaySec and
Persistent=true are built in -- a host asleep at 09:00 silently skips a cron
job, and a skipped stories run is content gone for good.
The publish step is still blocked on mattellite having no key on the NAS; that
is recorded under Outstanding as the one thing between this and unattended.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Nothing here is broken — these are decisions not made and cleanups not done,
written down before the session's context is lost.
The two that can actually cost something: no daily stories run is scheduled,
and stories are the one surface that cannot be backfilled; and the `seeded`
flags in the state file were reconstructed by hand from a log rather than
derived from the archive DB, so losing artms.db without also clearing the
state file would leave those sources permanently unseeded and re-download
everything.
Also corrects "Scanner work (not done yet)", which shipped in 53b1f80 —
the three .json shapes are told apart structurally in gallery-dl-sidecar.ts
and isReel comes from the sidecar's type. That file lives on main: it parses
archives at display time and is viewer code, not tooling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Putting it at the top of README.md worked, but it diverged a shared file: a
probe showed main editing the adjacent line conflicts on every merge. The
warning survived the conflict, so nothing was ever silently lost, but a file
that only exists on this branch has no such cost at all.
TOOLING.md carries the warning plus what CLAUDE.md would have said if it could
— the commands, the remote policy, and the two guards, which are local and
unversioned and so are absent from every fresh clone.
Shared files are byte-identical to main again: CLAUDE.md, README.md and
package.json. The divergence is now only files main has never had.
Trade-off worth knowing: gitea renders README.md on the branch page and does
not render TOOLING.md, so this warning is one click less visible than it was.
The pre-push hook, not the documentation, remains the guard that actually
stops a mistake.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This branch is the only place the fetching tooling still exists, and it names
the fetch host's public IP, the cookie's browser profile, the NAS path and the
accounts being archived. main's history was rewritten, the GitHub repo deleted
and recreated, and 22 ghcr images pruned to get that material out of public
view — one push of this branch undoes all of it.
Deliberately at the top of the README rather than in a separate file: gitea
renders README.md on the branch page, which is where someone actually looks
before pushing. That does diverge from main, but only by an addition, which a
merge cannot silently drop — unlike the CLAUDE.md/package.json case, where
main was DELETING lines this branch kept and every merge re-applied the
deletion. Worst case here is a trivial conflict if main edits the same region,
and the block says to keep both.
Also records that the pre-push hook is local and unversioned, since a fresh
clone is the realistic way this goes wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
`tsc` keeps comments by default, so the doc comment in archive-grouping.ts
describing the sidecar layout was emitted into dist-server and copied into the
runtime image. Every published container image on ghcr carries it — verified by
pulling the dist-server layer of :latest and grepping it:
app/src/lib/archive-grouping.js:10: * <user> -> posts (base)
That comment names real archived accounts, which is exactly what main was
redacted to remove, so the redaction was incomplete while the build kept
re-emitting them. The frontend was never affected: Vite strips comments, and
the 432K dist layer greps clean.
--removeComments takes dist-server from 0 comment lines. docs/ was never at
risk; the multi-stage build copies only dist/ and dist-server/ into runtime.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
main's history was redacted on 2026-08-20: every archived account name was
replaced throughout all 65 commits, because main is published to GitHub and
the list of accounts identifies the archive as surely as a hostname does. The
viewer's test fixtures now use synthetic names chosen to preserve exactly what
each case tests — a leading digit, an underscore, a dotted username, a
bracketed or dotted highlight title, and the trailing U+2800 that case 12
exists for.
The redaction swept these three docs too, since they were in the same history.
This puts the real names back, on the branch that is not published. The
scripts needed nothing: they are generic and never named an account.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
A superset branch only stays cheap if the files it shares with main are
untouched. Both diverged: package.json carried a jd2 script and CLAUDE.md
described the tooling, so every merge from main would have conflicted on them
forever, and main's removal commit would have kept trying to delete the
tooling entries here.
The tooling files themselves do not exist on main, so they can hold whatever
they like. The jd2 invocation moves into docs/jdownloader.md as a direct npx
call instead of an npm script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The scripts and docs for fetching from Instagram now live here rather than on
main, which is the branch published to GitHub. They carry things that do not
belong in a public repo: the fetch host's public IP, the browser profile path
the cookie is read from, the NAS archive path, and the list of accounts being
archived.
This branch is a superset of main — the viewer plus the tooling — so it can
take main's changes by merging, and the npm script and CLAUDE.md entries that
reference the tooling live here where the files actually exist.
Restored with the sync work from the 2026-08-20 run already in place: the
--abort flag, the corrected yt-dlp install advice, and the measurements behind
both.
Note that main's history was rewritten to strip these paths, so the tooling's
own per-file history does not exist on this branch. It is preserved on gitea
as pre-rewrite-20260820 and pre-rewrite-tooling-20260820.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
The fetching scripts and their docs now live on the `tooling` branch, which
is not published to GitHub. This removes the two references that would
otherwise dangle here: the `jd2` npm script and the CLAUDE.md bullet
describing it.
The viewer's own gallery-dl support is untouched and stays here —
src/lib/gallery-dl-sidecar.ts and friends parse sidecars at display time and
are app code, not tooling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Two .pyc files under scripts/__pycache__ were committed at some point and have
been churning ever since — merely importing gdl-sync.py to check a config
rewrites them and dirties the tree, which is how they surfaced.
.gitignore had no Python entries at all, only Node ones. The files stay on
disk; this just untracks them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
Docker Build and Publish / build-and-push (push) Failing after 10s
Ships the gallery-dl sidecar work to the viewer. The visible change is
reel classification: official_artms' Reels tab drops from 781 items to
360, because the sidecars say the other 421 are ordinary feed videos the
clips endpoint returns via include_feed_video. Directory-based
classification counted them all as reels.
Also in this release: dates ranked by source rather than scan order, and
highlight items no longer appearing twice when the archive holds them
under both JDownloader naming conventions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JDownloader wrote story-shaped names for highlights during one period of
its life, so the same item exists on disk as both
0ct0ber19 - C5dQPEYpd9W.mp4
2024-04-07_0ct0ber19 - 01 - C5dQPEYpd9W.mp4
which parsed to the ids "C5dQPEYpd9W" and "01 - C5dQPEYpd9W" -- two posts
for one item. The leading ordinal is a position within a day's stories
and carries nothing the shortcode does not, so story and highlight ids
drop it. Post ids are untouched, since those are permalinks.
Measured on the two real files, same archive, cache cleared between:
without the fix the profile reads "Heestory - 2 items", with it
"Heestory - 1 item".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit had this backwards: the sidecar date was only
consulted when the existing date came from an mtime, so a filename date
silently outranked what Instagram itself reported.
The order is sidecar, then filename, then mtime -- metadata first,
mtime last, since mtime is when the file hit disk and says nothing about
when the post was made. Ties keep the incumbent so two equally
authoritative files cannot flip a post's date by scan order.
Extracted to src/lib/post-dates.ts rather than left inline, because the
rule is easy to state and easy to get wrong -- the tests include an
order-independence case that would have caught the original mistake.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The .json sidecars published with the ARTMS fetch were inert: the scanner
fed them through the Instaloader path, where `node.edge_media_to_caption`
and `checkIsStory`'s `product_type` are both absent, so nothing happened.
They are now recognised structurally -- flat, with post_shortcode and
type, and none of the markers the other two JSON shapes carry -- and used
for three things:
- `type` sets post.isReel, which post-tabs prefers over every fallback.
This is Instagram's own classification and it disagrees with ours a
lot: of 781 items in "official_artms - reels", the sidecars say only
360 are reels. The other 421 are feed videos the clips endpoint returns
via include_feed_video, and the directory-based rule counted them all.
- `description` fills the caption where no .txt exists.
- `date` dates a post whose filename could not.
Also fixes date precedence. Only JDownloader highlights lack a date in
the filename, so parseArchiveFilename now marks those as mtime-derived
and the scanner lets any real date replace them -- previously the date
depended on which file the scan reached first.
Verified against real published files: a directory of three type=post and
three type=reel renders 6 in the grid and exactly the 3 reels in the
Reels tab.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every claim in docs/gallery-dl.md was measured against the live site and
the archive rather than taken from documentation, because two of the
assumptions turned out to be wrong.
The safety model is the reason the config looks the way it does.
gallery-dl has two API backends: the graphql one issues a request PER
POST for every video and carousel -- the pattern that got this account
banned via Instaloader -- while the default rest one paginates listings
at 30-50 items and carries carousel_media, video_versions and
product_type inline. A 300-post profile costs ~10 requests.
Findings worth recording:
- JD2 stamped filenames in desktop LOCAL time (US Eastern), not UTC.
Across 212 comparable posts: UTC 19 mismatches, UTC-5 10, UTC-4 zero.
{date:Olocal/%Y-%m-%d} reproduces it; the trailing separator must be
omitted or it lands in the strftime format.
- A profile's reels tab returns collab reels owned by OTHER accounts, so
the directory must be forced with -D. JD2 did the same: chuuo3o and
official_artms filenames sit inside "0ct0ber19 - reels".
- Stories and highlights need per-item {shortcode}; {post_shortcode} is
the reel's id and is shared by every item. {date} is per-item, verified
on a 154-item highlight with distinct times.
- gallery-dl reproduces JD2's caption .txt exactly, including writing
nothing for an empty caption and omitting the trailing newline.
- The json sidecar needs `include`, not `fields`; `fields` silently does
nothing in mode:json and leaks audio_user blobs. It yields `type`
(post/reel) -- Instagram's own flag, which can retire the lone-video
heuristic once the scanner reads it.
Naming differences between the two tools are cosmetic: EXPORT_RE already
makes the index optional and parseInt normalises zero-padding, so a mixed
archive parses identically. Tests pin that down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Docker Build and Publish / build-and-push (push) Failing after 10s
The header rendered allPosts.length, which is pre-dedupe — 0ct0ber19
showed "303 posts" over a 300-tile grid. Instagram's counter equals its
grid, so count the grid.
Also correct CLAUDE.md. v1.7.0 claimed the grid holds everything "as on
Instagram"; Instagram actually includes a reel in the grid only when the
creator shared it to feed, per post. Measured live: official_artms has
21 reels in its first 34 grid tiles, 0ct0ber19 has 1 in 214. Archives
carry no such flag, so showing everything approximates the behaviour
rather than reproducing it.
Records the DOM trap that caused the wrong reading in the first place:
grid reels link to /reel/<code>/, not /<user>/p/<code>/.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Docker Build and Publish / build-and-push (push) Failing after 9s
The Posts tab filtered reels out, so the grid was not the archive — it
was the archive minus its videos. For `for.heejin` that hid 533 of 1225
posts; for `loonatheworld`, 1100 of 3813. On Instagram the grid holds
everything and the Reels tab is a filtered view of that same set.
Extract the tab logic to src/lib/post-tabs.ts so the reel heuristic is
testable outside the component, and add dedupePostCopies: the jd2 flow
crawls the profile URL and the /reels URL separately because the profile
page misses some reels, so the two overlap and a reel can land on disk
twice. Those are two posts with distinct directory-scoped ids, which the
grid would now render side by side; the reels-source copy wins so the
survivor is still recognised as a reel.
Deciding what *is* a reel stays a guess for most archives. Instagram
marks it with product_type ("clips" vs "feed" vs "igtv" — all three are
GraphVideo, and aspect ratio does not separate them), but only newer
Instaloader captures carry it: 1101 of gibiofficial's 5919 sidecars, and
only 2 marked clips. JDownloader archives carry none, so those still
fall back to treating a lone video as a reel.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ergosteurandClaude Sonnet 5