Compare commits

..
Author SHA1 Message Date
ergosteurandClaude Opus 5 71cfd29f36 docs: record the first full incremental sync, and what made it cheap
All six ARTMS profiles, four days after the previous run: 184 new media, 299
files published, 0 failures and 0 CDN 429s. The header still claimed the
script was a skeleton with nothing migrated, which stopped being true a while
ago.

Notes the two things that made it cheap — an already-seeded archive DB, so no
probe passes ran at all, and --abort 50 — and that the run was deliberately
stopped and resumed midway to pick up the new flag. That is only safe because
the state file had already marked the finished sources as fetched, so the 20h
floor skipped them. Stopping a run used to mean repeating it.

Also records that rsync exit 23 is the NORMAL outcome of a publish: the SSH
user cannot chown to rslsync, so attrs fail while data arrives intact. The
check is a --dry-run re-run returning an empty file list, which is what
verified this one (+299 files, matching rsync exactly).

The account was restored; noting that without pretending it licenses more
traffic.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
2026-08-20 14:36:32 -04:00
ergosteurandClaude Opus 5 a83da461c1 feat: stop enumerating a profile once it reaches what we already hold
The skip-archive suppresses downloads, which spends the CDN. It does nothing
about the listing pass, which spends `instagram.com` — the surface that
actually bans accounts — and that cost scales with how BIG a profile is, not
with how much of it is new. A 2275-post profile paid ~76 pages every run to
discover three new posts. Seeding saved the second full pass, never the first.

Measured from sidecar write times during today's run, free because the run was
paying for the listing anyway: three new posts took ~100s each, and the other
2272 were written in a single second — enumeration with nothing to show for it.

`--abort N` passes gallery-dl's `skip: abort:N`, stopping the extractor after N
consecutive already-archived files. Resuming a stopped run with `--abort 50`
enumerated 7 posts of cher_ryppo's 2151 and still caught every new one.

Three things make this safe, and all of them are load-bearing:

- N counts FILES, not posts, so it has to clear the largest already-held
  carousel — one post in this archive is 22 media. A reels tab needs 50 actual
  reels for the same threshold, since those are single-media.
- It applies to posts and reels only. Stories are always new, and highlight
  items are not ordered in a way that makes early abort safe.
- The REST listing is strictly reverse-chronological. Test case 16 claimed
  0ct0ber19 returns its 3 pinned posts out of date order; that is true of the
  web grid but not of this endpoint, measured today. Front-loaded old posts are
  the one thing that would trip abort before it reached anything new, so the
  correction is what licenses the feature rather than a footnote to it.

Default is 0 — walk everything — because aborting early stops noticing edited
carousels (test case 15), which only a full enumeration finds. Routine runs
want 50; a full sweep is still worth running occasionally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
2026-08-20 14:31:33 -04:00
ergosteurandClaude Opus 5 01702ea24b docs: fix the yt-dlp advice — a separate venv is not an importable one
`pipx install yt-dlp` was the instruction here, and it never worked. It
gives yt-dlp its own venv, so the binary lands on PATH while gallery-dl,
living in a different venv, still cannot `import yt_dlp`. Everything looks
installed and the log keeps saying `Cannot import yt-dlp or youtube-dl` —
which is exactly what the first run logged, and why it fell back to
progressive URLs for DASH video, the path the CDN 429s hit hardest.

`pipx inject gallery-dl yt-dlp` is the fix. Verify it by asking gallery-dl's
own interpreter rather than the shell, since `which yt-dlp` succeeds either
way and is what made the original advice look correct.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
2026-08-20 14:31:09 -04:00
ergosteurandClaude Opus 5 07df53acde chore: release 1.8.0
Docker Build and Publish / build-and-push (push) Failing after 13s
Ships the gallery-dl sidecar work to the viewer. The visible change is
reel classification: official_artms' Reels tab drops from 781 items to
360, because the sidecars say the other 421 are ordinary feed videos the
clips endpoint returns via include_feed_video. Directory-based
classification counted them all as reels.

Also in this release: dates ranked by source rather than scan order, and
highlight items no longer appearing twice when the archive holds them
under both JDownloader naming conventions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 13:58:15 -04:00
ergosteurandClaude Opus 5 c81f18276f fix: stop showing a highlight item twice under two naming conventions
JDownloader wrote story-shaped names for highlights during one period of
its life, so the same item exists on disk as both

    0ct0ber19 - C5dQPEYpd9W.mp4
    2024-04-07_0ct0ber19 - 01 - C5dQPEYpd9W.mp4

which parsed to the ids "C5dQPEYpd9W" and "01 - C5dQPEYpd9W" -- two posts
for one item. The leading ordinal is a position within a day's stories
and carries nothing the shortcode does not, so story and highlight ids
drop it. Post ids are untouched, since those are permalinks.

Measured on the two real files, same archive, cache cleared between:
without the fix the profile reads "Heestory - 2 items", with it
"Heestory - 1 item".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 13:57:42 -04:00
ergosteurandClaude Opus 5 4ce5cf048e feat: give the sync a memory, so it stops paying for the same listing twice
Nothing in this tool had any memory: every invocation started from zero
and would happily re-enumerate a profile it had listed minutes earlier.
That is what suspended the account -- the listing passes, not the
downloads -- and an aborted run re-enumerating five profiles on restart
was a large part of the bill.

Three changes, in order of how much they save:

- Seeding is now a one-time bootstrap per source. After the first
  successful sync the archive DB records everything gallery-dl has seen,
  so the source is never probed again. A second full sync costs roughly
  half what the first did.
- Stories never seed at all. A story cannot be in the archive before it
  is fetched, so there is nothing to seed from, and probing would double
  the cost of the cheapest surface we have.
- A source fetched within --min-interval (20h) is refused, and listing
  results are cached for --probe-ttl (24h), so a restart mid-run is free
  rather than a repeat. --force overrides both.

--only replaces --no-stories and takes any subset of the surfaces, which
is what makes a daily stories-only run possible: one source per profile,
no seeding, a handful of requests. Everything else stays monthly.

Tested with stdlib unittest -- no new dependencies, and it runs anywhere
the sync does. The cases include the aborted-restart scenario, which now
plans zero work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 13:53:18 -04:00
ergosteurandClaude Opus 5 7a085d2272 docs: record why the account was suspended — verification, not fetching
The account was suspended on 2026-08-17 for "spam", during the session
that built this tooling. Both fetching docs were confidently wrong about
what the risk was, so both now carry the correction.

docs/jdownloader.md said the ban vector is instagram.com requests, which
is right, and implied that meant per-post metadata fetching, which is
only part of it. The suspension came from read-only verification:
automated browser scrolling to enumerate profile grids (~18 paginated
loads per profile, done twice on one after a selector bug), repeated
--simulate and -j passes over the same profiles, per-post /p/ fetches
while testing filename formats, and an aborted sync that re-ran every
listing pass before dying. None of that produced a file, and together it
rivalled the real sync for request count.

The rules that follow are in docs/gallery-dl.md: verify against the
archive rather than the live site, count read-only work against the same
budget, treat the first CDN 429 as the end of the session rather than a
pacing knob, and cache probe_live so a restart does not re-enumerate
everything. The warning order was CDN 429, then 400 on the highlights
tray, then suspension.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 12:56:00 -04:00
ergosteurandClaude Opus 5 087304243e fix: rank date sources instead of letting scan order decide
The previous commit had this backwards: the sidecar date was only
consulted when the existing date came from an mtime, so a filename date
silently outranked what Instagram itself reported.

The order is sidecar, then filename, then mtime -- metadata first,
mtime last, since mtime is when the file hit disk and says nothing about
when the post was made. Ties keep the incumbent so two equally
authoritative files cannot flip a post's date by scan order.

Extracted to src/lib/post-dates.ts rather than left inline, because the
rule is easy to state and easy to get wrong -- the tests include an
order-independence case that would have caught the original mistake.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 12:50:48 -04:00
ergosteurandClaude Opus 5 e14dbf6ec8 feat: read gallery-dl sidecars for reel type and post dates
The .json sidecars published with the ARTMS fetch were inert: the scanner
fed them through the Instaloader path, where `node.edge_media_to_caption`
and `checkIsStory`'s `product_type` are both absent, so nothing happened.

They are now recognised structurally -- flat, with post_shortcode and
type, and none of the markers the other two JSON shapes carry -- and used
for three things:

- `type` sets post.isReel, which post-tabs prefers over every fallback.
  This is Instagram's own classification and it disagrees with ours a
  lot: of 781 items in "official_artms - reels", the sidecars say only
  360 are reels. The other 421 are feed videos the clips endpoint returns
  via include_feed_video, and the directory-based rule counted them all.
- `description` fills the caption where no .txt exists.
- `date` dates a post whose filename could not.

Also fixes date precedence. Only JDownloader highlights lack a date in
the filename, so parseArchiveFilename now marks those as mtime-derived
and the scanner lets any real date replace them -- previously the date
depended on which file the scan reached first.

Verified against real published files: a directory of three type=post and
three type=reel renders 6 in the grid and exactly the 3 reels in the
Reels tab.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 12:42:54 -04:00
ergosteurandClaude Opus 5 3e29976866 fix: line-buffer sync output so a redirected log shows progress live
A sync runs for hours and is normally watched through a redirected log,
where Python's block buffering withheld the per-source progress lines
until they happened to flush. The gallery-dl subprocesses write to the
same descriptor unbuffered, so the log also interleaved out of order.

Reconfiguring the streams in-process rather than relying on `python3 -u`
means it holds however the script is invoked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 22:48:40 -04:00
ergosteurandClaude Opus 5 0ac1ef951f docs: move project status out of the sync script into the docs
Script comments should describe the script. Where a run has got to is
project state, so it belongs in docs/gallery-dl.md, which now records the
live withaseul publish, the file-ownership caveat and where the profile
list lives.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 22:33:38 -04:00
ergosteurandClaude Opus 5 ee66b87bff feat: drive gdl-sync from a file of profile URLs
Adds --urls-file so a run can reference a hand-maintained list rather
than repeating --profile, which is how this actually gets used: the
ARTMS accounts now live in artms_account_links.txt at the archive root.

The parser takes what a person would paste. Full URLs, scheme-less URLs
and bare usernames all work; blank lines and # comments are ignored and
duplicates dropped, so the list can be appended to carelessly. Lines that
are not profiles are rejected loudly rather than silently syncing
nothing: an Instagram post URL yields the segment "p", which would
otherwise be treated as a username and create a directory called "p".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 22:32:06 -04:00
ergosteurandClaude Opus 5 68568ef855 fix: keep the generated config out of the published tree
A dry-run publish against the real archive caught gdl-sync.config.json
being created in the archive root: it was written into the staging
directory, and staging is rsynced wholesale. It now lives as a sibling of
staging instead, with rsync excludes as a second line of defence.

The dry run is otherwise clean -- 322 files added, 0 deleted, no new
directories -- and confirms the property that matters most: of 74 new
media files, zero duplicate media already held under a different name.
The JD2 and gallery-dl naming really do converge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 21:43:12 -04:00
ergosteurandClaude Opus 5 3a34e4359e feat: publish fetched media by rsync, and back off from the CDN's 429s
Wires up the staging -> rsync step and replaces --archives with --index
(a listing source: local path or the viewer's API), --staging and
--publish, so the fetch host needs no copy of the archive.

rsync runs --ignore-existing with no --delete. That is a safety property
rather than an optimisation: the archive deliberately outlives Instagram,
so publishing must only ever add. It runs once at the end so a profile
that fails midway never reaches the archive half-written.

Exercised end-to-end against withaseul across all four surfaces,
publishing to a scratch directory. Seeding worked as designed (915 of 984
post items and 28 of 34 reel items already held), stories and highlights
returned no results cleanly, and the collab-reel case landed correctly:
"withaseul - reels" holds files owned by cher_ryppo, 0ct0ber19 and
official_artms, each with the owner in the filename and the crawl scope
as the directory.

The first run drew '429 Too Many Requests' from the CDN at 3M with 1-3s
sleeps and lost two videos. That is the tolerant surface complaining, so
the defaults are now 1M, 6-10s between requests, 3-6s between downloads,
sleep-429 of 120s and 8 retries. Re-running recovered both videos with
zero failures and zero 429s. Installing yt-dlp on the fetch host also
matters: without it DASH videos fall back to a progressive URL, which is
what the rate limiting hit hardest.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 21:37:38 -04:00
ergosteurandClaude Opus 5 4fee8b1dfe feat: seed gallery-dl's skip-archive so the fetcher needs no archive copy
gallery-dl skips already-held media either by file existence -- which
requires the archive mounted where it writes -- or by a sqlite
skip-archive, which requires nothing on disk. Using the latter lets the
fetch host write to local disk and rsync afterwards, avoiding tens of
thousands of small writes over CIFS and keeping a mid-sync failure from
leaving partial files on the live Resilio share.

The key is archive_prefix + archive_fmt: the literal "instagram" plus the
per-media numeric pk. Verified against a real run -- a 3-image carousel
produced 3 rows and a re-run skipped every media file.

media_id is absent from our filenames, so the DB cannot be built from
names alone, but the listing pass we already make maps every live item to
its media_id, and a file listing says which we hold. Seeding therefore
costs no extra Instagram requests and no archive content -- the listing
GET /api/archives/:name/files already serves is enough.

Measured on 0ct0ber19: 2275 live items, 2248 seeded, 27 left to fetch --
exactly the media of the two posts added since the last crawl.

The trap worth the comment it carries: posts and reels are filed under
post_shortcode, while stories and highlights use the per-item shortcode
(post_shortcode there is the containing reel's id, shared by every item).
Matching on the wrong field seeded 5 of 2275 rather than failing loudly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 21:10:37 -04:00
ergosteurandClaude Opus 5 7b63ba6a76 docs: design a gallery-dl replacement for the JDownloader fetcher
Every claim in docs/gallery-dl.md was measured against the live site and
the archive rather than taken from documentation, because two of the
assumptions turned out to be wrong.

The safety model is the reason the config looks the way it does.
gallery-dl has two API backends: the graphql one issues a request PER
POST for every video and carousel -- the pattern that got this account
banned via Instaloader -- while the default rest one paginates listings
at 30-50 items and carries carousel_media, video_versions and
product_type inline. A 300-post profile costs ~10 requests.

Findings worth recording:

- JD2 stamped filenames in desktop LOCAL time (US Eastern), not UTC.
  Across 212 comparable posts: UTC 19 mismatches, UTC-5 10, UTC-4 zero.
  {date:Olocal/%Y-%m-%d} reproduces it; the trailing separator must be
  omitted or it lands in the strftime format.
- A profile's reels tab returns collab reels owned by OTHER accounts, so
  the directory must be forced with -D. JD2 did the same: chuuo3o and
  official_artms filenames sit inside "0ct0ber19 - reels".
- Stories and highlights need per-item {shortcode}; {post_shortcode} is
  the reel's id and is shared by every item. {date} is per-item, verified
  on a 154-item highlight with distinct times.
- gallery-dl reproduces JD2's caption .txt exactly, including writing
  nothing for an empty caption and omitting the trailing newline.
- The json sidecar needs `include`, not `fields`; `fields` silently does
  nothing in mode:json and leaks audio_user blobs. It yields `type`
  (post/reel) -- Instagram's own flag, which can retire the lone-video
  heuristic once the scanner reads it.

Naming differences between the two tools are cosmetic: EXPORT_RE already
makes the index optional and parseInt normalises zero-padding, so a mixed
archive parses identically. Tests pin that down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 21:04:39 -04:00
ergosteurandClaude Opus 5 b84113af18 fix: count the grid in the post header, and qualify the Instagram claim
Docker Build and Publish / build-and-push (push) Failing after 10s
The header rendered allPosts.length, which is pre-dedupe — 0ct0ber19
showed "303 posts" over a 300-tile grid. Instagram's counter equals its
grid, so count the grid.

Also correct CLAUDE.md. v1.7.0 claimed the grid holds everything "as on
Instagram"; Instagram actually includes a reel in the grid only when the
creator shared it to feed, per post. Measured live: official_artms has
21 reels in its first 34 grid tiles, 0ct0ber19 has 1 in 214. Archives
carry no such flag, so showing everything approximates the behaviour
rather than reproducing it.

Records the DOM trap that caused the wrong reading in the first place:
grid reels link to /reel/<code>/, not /<user>/p/<code>/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 20:00:11 -04:00
ergosteurandClaude Opus 5 abf22eb50f feat: show reels in the profile grid, as Instagram does
Docker Build and Publish / build-and-push (push) Failing after 10s
The Posts tab filtered reels out, so the grid was not the archive — it
was the archive minus its videos. For `for.heejin` that hid 533 of 1225
posts; for `loonatheworld`, 1100 of 3813. On Instagram the grid holds
everything and the Reels tab is a filtered view of that same set.

Extract the tab logic to src/lib/post-tabs.ts so the reel heuristic is
testable outside the component, and add dedupePostCopies: the jd2 flow
crawls the profile URL and the /reels URL separately because the profile
page misses some reels, so the two overlap and a reel can land on disk
twice. Those are two posts with distinct directory-scoped ids, which the
grid would now render side by side; the reels-source copy wins so the
survivor is still recognised as a reel.

Deciding what *is* a reel stays a guess for most archives. Instagram
marks it with product_type ("clips" vs "feed" vs "igtv" — all three are
GraphVideo, and aspect ratio does not separate them), but only newer
Instaloader captures carry it: 1101 of gibiofficial's 5919 sidecars, and
only 2 marked clips. JDownloader archives carry none, so those still
fall back to treating a lone video as a reel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 19:06:53 -04:00
ergosteurandClaude Opus 5 e6d874a5d9 docs: record the CSP/wasm and PWA-precache traps in CLAUDE.md
Two failures in this session were expensive because nothing recorded them:

- The CSP must keep 'wasm-unsafe-eval' and connect-src data:, because the xz
  decompressor for Instaloader sidecars is WebAssembly embedded as a data: URL.
  Removing either breaks decoding with a bare "Failed to fetch" and no stack,
  and the visible symptom is silent metadata loss rather than an error.
- The service worker precaches index.html with its headers, so a server-only
  change never reaches installed clients. The version compiled into the client
  is what forces the precache to turn over each release; it is load-bearing,
  not decoration.

Also notes that the Vite dev server sends none of these headers, so CSP and PWA
behaviour must be verified against a built dist/ served by server.js, and that
the live archive root is now the archives/ subdirectory.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 18:36:09 -04:00
ergosteurandClaude Opus 5 ff45b6d898 fix: make server header changes reach installed PWA clients
Docker Build and Publish / build-and-push (push) Failing after 10s
The service worker precaches index.html together with its response headers, so
a server-only change never reaches an installed client: the client build is
byte-identical, the precache manifest is unchanged, and the worker has no
reason to update. That is why the CSP fix in 1.6.1 did not reach a browser that
already had the app cached — it kept replaying a cached shell carrying the old,
broken CSP, indefinitely.

The release version is now compiled into the client, which makes every release
change the bundle hash, hence index.html, hence its precache revision, hence
sw.js itself — the bytes browsers compare to decide whether to update. Verified
by bumping only the version: index-DYufL2Fa.js -> index-60N_3d5j.js, with the
new name carried into the sw.js manifest.

It also surfaces in the footer, so the deployed version is visible without
digging through devtools.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 13:16:25 -04:00
ergosteurandClaude Opus 5 30ff8de1e8 fix: allow WebAssembly in the CSP so xz sidecars can be decoded
Docker Build and Publish / build-and-push (push) Failing after 9s
The xz decompressor for Instaloader's .json.xz sidecars is WebAssembly,
embedded as a data: URL that it fetches at startup. The CSP added in 1.3.0
blocked both halves of that:

  fetch('data:application/wasm;...')  -> TypeError: Failed to fetch
  WebAssembly.instantiate(...)        -> CompileError: violates script-src 'self'

The first surfaces through new Response(stream).json() as a bare "Failed to
fetch" with no stack, which reads like a network fault and is why this was
mis-diagnosed twice. Vite's dev server never sends the CSP, so it reproduced
only in production — every Instaloader archive silently lost its captions,
story flags and profile metadata from 1.3.0 onward.

script-src now allows 'wasm-unsafe-eval', which permits WebAssembly compilation
without permitting eval() of JavaScript, and connect-src allows data: for the
embedded module.

Verified against the production bundle: rivvsofficial goes from 188 posts / 0
followers / no stories to 68 posts, 120 stories, 10,337 followers and its real
name, bio and link — with zero decode errors.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 12:51:42 -04:00
ergosteurandClaude Opus 5 9560f85515 fix: read xz sidecars by buffer, page carousel with arrows, stop backdrop flash
Docker Build and Publish / build-and-push (push) Failing after 9s
Instaloader metadata was silently lost
Every .json.xz failed with "Failed to fetch" during a scan, though the same URL
fetched fine on its own. RemoteArchiveFile.stream() started a fetch, piped the
body into a TransformStream and returned the readable immediately — nothing
caught a fetch rejection, and the decompressor stops reading at the end of the
xz member, so the response body was never drained or cancelled. Across ~190
sidecars that exhausted the connection pool.

Everything Instaloader archives carry lives in those files, so the failure was
invisible but total. rivvsofficial reported 188 posts, no stories, 0 followers
and a placeholder bio; it now reports 68 posts, 120 stories, 10,337 followers
and the real name, bio and link — 68 + 120 = 188, matching the sidecars exactly
(106 GraphStoryVideo + 14 GraphStoryImage = 120).

These sidecars are a few KB, so they are now read into memory before
decompressing. stream() was left unused by that change and is removed from the
interface and both implementations rather than kept as a trap.

Arrow keys page the carousel
They moved between posts, which contradicted the arrows drawn on the carousel
itself. Arrows now page slides; , and . move between posts, alongside the side
buttons.

Backdrop cross-fade
AnimatePresence had no exit variant, so the outgoing scan backdrop was removed
instantly while its replacement faded in over 1.5s, exposing the pale page
behind it as a white flash. Layers now stack: the outgoing image holds full
opacity until covered, and the 0.4 moved onto the group so overlapping layers
don't darken as they cross. Measured over a real scan: 152 cross-fades with a
layer always opaque, except the opening fade-in where nothing is underneath.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 12:28:26 -04:00
ergosteurandClaude Opus 5 f0c054946f docs: add a JDownloader quick reference
Covers why fetching goes through JDownloader rather than Instaloader (the
instagram.com vs CDN split, and what the metadata gap actually costs), the
settings that matter, cookie handling, the two-URL workflow, jd2-sync usage,
the expected on-disk layout, and what to do when something breaks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 12:02:41 -04:00
ergosteurandClaude Opus 5 42eb598d1c fix: never descend into NAS metadata directories when indexing
Docker Build and Publish / build-and-push (push) Failing after 9s
The archive root was filtered by prefix, but the recursive walk below it was
not, so anything inside a profile directory got indexed. NAS filesystems put
sidecar metadata *inside* every folder rather than only at the share root:
Synology writes @eaDir (thumbnails and indexing data), #recycle holds
deletions, .sync is Resilio state. On the live share those account for 12,516
of 123,023 files.

None currently sit inside a profile directory, so nothing was miscounted yet —
but the moment that share gets indexed for Photos, every generated thumbnail
would be counted as archive media and stat'd one by one over the network, which
is the cost the index exists to avoid.

One isSystemDirectory rule now applies at every level, and the root listing uses
it too instead of keeping a second copy of the pattern.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 11:57:29 -04:00
ergosteurandClaude Opus 5 aac753ced9 feat: generate JDownloader crawljobs from the archives on disk
The manual flow is: paste a profile URL into JDownloader, paste the /reels URL
separately (the profile page misses some reels), set the output folder by hand,
repeat per profile. scripts/jd2-sync.ts emits one crawljob per source with the
folder already pointed at the right directory, so folder-watch picks up the
whole batch at once.

Profiles and sidecars are derived with the same grouping logic the server uses,
so output folders always match what the viewer expects to find. --download-base
maps the path for a JDownloader running on another machine (Windows paths
included), since it typically runs on a desktop against the share.

Directories that aren't Instagram profiles are skipped: an archive root also
collects tool output and exports from other services, and pointing a crawl at
those spends requests on instagram.com to be told the profile doesn't exist —
exactly the traffic worth not spending. Filtering is by username shape, plus
--skip and a .jd2ignore file for names that look like usernames but aren't.

Defaults are conservative: chunks=1, because multi-chunk ranged requests are the
one CDN-side pattern that doesn't resemble a browser, and links park in the
LinkGrabber for review rather than auto-starting.

Only posts and reels are emitted; highlight URLs need a numeric id and story
URLs expire, so those stay manual.

Format verified against JDownloader's own explain.txt for the folderwatch
extension, read from the daily SVN mirror rather than one of the decade-stale
GitHub copies.

Also refreshes CLAUDE.md, whose URL-state section still described the query
parameters replaced in 1.4.0, and documents the mobile feed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 11:41:23 -04:00
ergosteurandClaude Opus 5 9173504190 feat: mobile opens posts as a scrolling feed instead of a modal
Docker Build and Publish / build-and-push (push) Failing after 9s
Tapping a post on a phone now opens a real feed page — header, media, actions,
caption, next post peeking in below — scrolled with the browser's own vertical
scrolling rather than swipe gestures. Desktop keeps the modal, where a centred
sheet with side arrows suits a pointer.

Only a window of posts is mounted: a profile here holds up to 1129 posts and
mounting them all would mean as many full-size images. The window grows in both
directions as you scroll. Growing upwards shifts everything below it, so the
scroll offset is corrected in the same frame, before paint — measured against
the real archive, an anchored post moves exactly one screen per scroll with no
jump.

Only the post crossing the viewport centre plays its video; the rest stay
paused, so a feed of reels doesn't play ten at once. The URL tracks that same
post, so scrolling updates /<archive>/p/<shortcode>/ the way Instagram does,
and the back button returns to the grid with its scroll position intact.

Feed video sizes to the container width rather than its intrinsic size: a
<video> reports 300x150 until metadata loads, which made it render narrow and
then jump to full width. It also gets a taller height ceiling than the modal so
ordinary portrait media fills the width instead of sitting in side bars.

The carousel is extracted into a shared MediaCarousel used by both surfaces, so
horizontal paging behaves identically; touch-action keeps vertical scrolling
passing through to the feed. PostModal loses its now-dead mobile swipe branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 10:23:33 -04:00
ergosteurandClaude Opus 5 1340d85945 feat: Instagram-shaped URLs, Instagram-shaped gestures, iOS-feel animations
Docker Build and Publish / build-and-push (push) Failing after 9s
Navigation gestures
Horizontal swipe used to advance the carousel and then, on the last slide,
fling you into the next post — one gesture meaning two things. Horizontal is
now carousel-only. On touch, vertical swipe moves between posts (down on the
first post still dismisses, keeping drag-to-close where it can't mean
"previous"). Desktop keeps the arrows outside the modal.

URLs
Permalinks now mirror Instagram:

  /<archive>/                 profile
  /<archive>/reels/           tab
  /<archive>/p/<shortcode>/   post

A post URL carries no tab, as on Instagram; the tab is re-derived from the
post's source, so opening a reel link lands on the Reels tab with next/prev
paging through reels. Sidecar posts keep directory-scoped ids internally but
expose only the shortcode. The old ?a=&t=&p= form is still parsed so existing
links keep working, and reserved prefixes (api, archives, assets…) can never be
mistaken for a profile name.

Animations
Adds a shared motion vocabulary tuned to feel native: critically damped springs
rather than fixed-duration easing, and gestures hand their exit velocity to the
animation so a flick continues instead of restarting. Post transitions animate
along the axis the input implies — vertical for a swipe, horizontal for the
arrows. Modal and story viewer present/dismiss with a scale, tiles and
highlight circles get touch-down feedback, and prefers-reduced-motion is
honoured throughout.

Adds 22 routing tests (58 total).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 10:01:04 -04:00
ergosteurandClaude Opus 5 c5b0a5cb5f fix: keep post nav arrows outside the modal and stop scroll chaining
Docker Build and Publish / build-and-push (push) Failing after 10s
The prev/next arrows are fixed to the viewport edges while the modal grows to
fill the available width, so below roughly 1200px the modal slid underneath
them and a white chevron landed on the white caption panel — invisible until
hovered. The overlay now reserves a horizontal gutter (md:px-16 lg:px-24) so
the arrows always sit outside the modal, and they get a solid white pill with a
dark chevron so they read against anything behind them. Verified clearing the
modal at 768, 1024, 1440 and 1920px.

The caption sidebar shrinks to w-80 at md so the narrower modal doesn't squeeze
the media pane.

Also adds overscroll-contain to the overlay: the wheel previously chained
through to the post grid behind it, scrolling the background while a post was
open.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 09:05:24 -04:00
ergosteurandClaude Opus 5 ae0f075855 feat: fit full-view media to the viewport and play with sound
Docker Build and Publish / build-and-push (push) Failing after 9s
Media in the post modal used w-full/h-auto, so a portrait video or image grew
taller than the screen (a 720x1280 reel rendered 768x1365 in a 786px viewport)
and forced the modal to scroll. Full view now caps height to the viewport minus
the modal's own padding. Video sizes to its own aspect within the cap so a
portrait clip isn't letterboxed edge to edge; images keep filling the modal
width and only gain a height ceiling.

Opening the modal or a story reel is a user gesture, so playback now starts
unmuted and only falls back to muted if the browser actually refuses the
play() promise — previously it always started muted, and the earlier
muted-by-default fix meant a blocked video could stall the story progress bar.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 08:50:49 -04:00
ergosteurandClaude Opus 5 3146f896c3 docs: update CLAUDE.md for the archive index, sidecars and cache rehydration
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 02:14:11 -04:00
ergosteurandClaude Opus 5 ed078c7b46 chore: bump version to 1.3.1
Docker Build and Publish / build-and-push (push) Failing after 9s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 02:08:35 -04:00
ergosteurandClaude Opus 5 55b0752b05 fix: don't crash at boot when running under a UID with no passwd entry
os.userInfo() throws ERR_SYSTEM_ERROR (uv_os_get_passwd) for a UID that has no
/etc/passwd entry, which is exactly what `docker run --user 1234:1234` produces
— the very workaround the README recommends. Combined with the switch to a
non-root image user, this crashed the server on startup for any deployment that
needed a custom UID to read its archives.

Also document the non-root default and the /cache index volume.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 02:08:35 -04:00
ergosteurandClaude Opus 5 53703cd7cd Merge branch 'review-fixes': security, performance and sidecar archive support
Docker Build and Publish / build-and-push (push) Failing after 9s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 01:59:41 -04:00
ergosteurandClaude Opus 5 97f5d19ce4 chore: bump version to 1.3.0
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 01:59:41 -04:00
ergosteurandClaude Opus 5 1b4aba54d6 fix: security, performance and correctness pass; add sidecar archive support
Security
- Fix path traversal in GET /api/archives/:name/files. Express decodes route
  params after segment matching, so `..%2f..%2fetc` escaped ARCHIVES_DIR and
  returned a recursive listing of arbitrary directories.
- Add CSP and baseline security headers; disable x-powered-by.
- Stop baking GEMINI_API_KEY into the client bundle (the SDK was unused).
- Run the container as `node` instead of root.

Performance
- Add a directory-mtime-keyed archive index, warmed in the background and
  persisted. Listing 110k files went from ~52s to ~0.1s; the largest archive
  (24k files) serves in ~0.3s. Per-file stat over CIFS costs ~1.4ms and does
  not parallelise, so it is now done once rather than per request.
- Build media URLs from the File directly instead of
  `new Blob([await file.arrayBuffer()])`, which read every media file fully
  into memory (a 20GB archive tried to become 20GB of resident blobs).
- Track and revoke object URLs; previously none were ever revoked.
- Give `requestThumbnail` a stable identity so a completed thumbnail stops
  re-running the effect in every mounted thumbnail.
- Namespace IndexedDB keys so listing archives no longer deserializes every
  cached thumbnail blob, and thumbnails no longer collide across archives.
- Serve real file sizes: RemoteArchiveFile was constructed with size 0, which
  silently disabled high-res thumbnailing for every server archive.

Correctness
- Local archives cached media as blob: URLs, which die with the document, so
  a cached local archive restored as an archive of broken images. Media now
  carries a stable path and is rehydrated from a persisted directory handle
  (File System Access API), falling back to re-prompting for the folder.
- Fix permalinks: the URL-writing effect erased ?a= on mount before the
  archive list arrived to consume it, so deep links never resolved.
- Make cache invalidation detect nested changes via a directory signature.
- Add an error boundary and tolerate unparseable dates, which previously
  threw a RangeError and blanked the app.
- Default video to muted so autoplay is not blocked by Safari/Firefox.

Features
- Fold sidecar directories into their base profile: `<user> - reels`,
  `story - <user>` and `story highlights - <user> - <title>` now appear as
  reels, the story ring and Instagram-style highlight circles rather than as
  separate archives.

Housekeeping
- Add @types/react; React was previously type-checked against its JavaScript
  source, so `npm run lint` gave almost no type safety on components.
- Vendor fonts and PWA icons locally; the app made third-party CDN requests
  despite advertising offline support and local-only processing.
- Drop unused better-sqlite3 (a native module that broke `npm install`).
- Add vitest with 36 tests over the filename and directory-naming rules.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uBWhwV3wFQ5MBCcMHHem7
2026-08-14 01:59:19 -04:00
ergosteur b30285fe70 docs: update documentation for high-res performance and local persistence
Docker Build and Publish / build-and-push (push) Failing after 10s
Key changes:
- Updated README.md and GEMINI.md with details on background thumbnailing and inter-post preloading.
- Documented persistent local archive caching and smart profile fallback features.
- Added dist-server/ to .gitignore.
- Restored missing feature descriptions and troubleshooting tips in README.
2026-03-07 21:59:43 -05:00
ergosteur 20209bcad5 feat: enable persistent local archives and smart profile fallback
Key changes:
- Enabled full metadata caching for local folder archives, allowing them to load instantly from IndexedDB without re-uploading.
- Implemented oldest-image fallback for profiles missing an explicit profile picture.
- Restored folder-name-to-username detection for local archive uploads.
- Optimized scan indexing to track all image files for fallback use.
2026-03-07 21:56:25 -05:00
ergosteur 74902234b3 fix: restore white glass scanning UI and resolve small image blur bug
Key changes:
- Corrected logic in PostThumbnail to prevent blur effects on images smaller than 1MiB.
- Restored the white glass aesthetic to the scanning dashboard with improved contrast and transparency.
- Optimized scanning background transitions to ensure a smooth, flicker-free crossfade.
2026-03-07 21:46:11 -05:00
ergosteur d62bddc3aa perf: implement background thumbnail generation and inter-post preloading
Key changes:
- Added Web Worker for background image thumbnailing with a 1MiB threshold to optimize CPU/memory usage.
- Implemented a serial task queue for memory-safe high-res image processing, preventing OOM crashes.
- Added inter-post preloading in the modal for seamless 'Previous/Next' navigation.
- Refined scanning UI with double-buffering and a dark background to completely eliminate white flashes.
- Renamed project to 'instaarchive-viewer' in package.json.
- Fixed 'Open image in new tab' by denylisting /archives and /api in PWA config.
2026-03-07 21:42:56 -05:00
ergosteur 42c13ea106 chore: bump version to 1.2.0
Docker Build and Publish / build-and-push (push) Failing after 9s
2026-03-07 21:16:39 -05:00
ergosteur a4e9ce16a7 feat: modularize scanner, enhance carousel preloading, and improve PWA updates
Summary of changes:
- Extracted archive scanning logic into a modular 'useArchiveScanner' hook for better maintainability and performance.
- Refined PostModal carousel with intelligent media preloading and smoother, jitter-free transitions.
- Optimized image rendering with 'decoding=async' and removed 'black flashes' between slide changes.
- Updated PWA configuration to 'autoUpdate' with hourly periodic checks for fresh content.
- Fixed several bugs including stories sorting, permalink parameter cleanup, and profile metadata cache restoration.
- Comprehensive updates to documentation (README.md and GEMINI.md) reflecting the new architecture.
2026-03-07 21:16:28 -05:00
ergosteur 4f89a69ee3 fix: optimize scanning performance and resolve zero-post bug
Docker Build and Publish / build-and-push (push) Failing after 10s
2026-03-07 20:25:23 -05:00
ergosteur 147dcdf2f1 fix: restore missing UI handlers and finalize generic parser
Docker Build and Publish / build-and-push (push) Failing after 10s
2026-03-07 20:18:33 -05:00
ergosteur d4e20d9b98 fix: refine dockerignore and bump version to v1.1.4
Docker Build and Publish / build-and-push (push) Failing after 10s
2026-03-07 20:10:40 -05:00
ergosteur ec8c771733 feat: implement permalinks and document PWA cache troubleshooting
Docker Build and Publish / build-and-push (push) Failing after 10s
2026-03-07 05:20:09 -05:00
ergosteur 3784e8729b debug: add verbose logging to permalink synchronization 2026-03-07 05:12:14 -05:00
ergosteur 69d62eaa5c fix: improve permalink comparison and add debug logging 2026-03-07 05:10:43 -05:00
ergosteur 767f9c508b feat: implement permalinks for archives, tabs, and posts 2026-03-07 05:07:04 -05:00
ergosteur 103ce6f207 fix: exhaustive generic parser and implement local archive history 2026-03-07 05:05:08 -05:00
ergosteur 5267dab236 fix: address Docker EACCES errors with better logging and SELinux hints
Docker Build and Publish / build-and-push (push) Failing after 9s
2026-03-07 03:02:02 -05:00
ergosteur ebf2bf660a fix: improve Docker archive discovery and switch to compiled server
Docker Build and Publish / build-and-push (push) Failing after 1m6s
2026-03-07 02:57:19 -05:00
ergosteur c0f3523a9c docs: update README and GEMINI with Docker usage and new features 2026-03-07 02:50:18 -05:00
ergosteur 6f5021638c feat: add Dockerfile and GitHub Actions workflow for GHCR deployment
Docker Build and Publish / build-and-push (push) Canceled after 11s
2026-03-07 02:45:17 -05:00
ergosteur 67f7750157 feat: refine navigation protection to only warn when leaving the app 2026-03-07 02:41:16 -05:00
ergosteur 9e306eb85e feat: add explicit confirmation for back button and refresh in archives 2026-03-07 02:39:39 -05:00
ergosteur b2da08d52d feat: add navigation protection and refine cached badge visibility 2026-03-07 02:37:17 -05:00
ergosteur d7c13ecc19 fix: resolve Firefox media warnings by improving video cleanup 2026-03-07 02:30:46 -05:00
ergosteur d396b356be feat: implement persistent caching, glassy scanning UI, and UI refinements 2026-03-07 02:28:22 -05:00
ergosteur e23dfe4474 feat: implement self-hostable mode with server-side directory scanning 2026-03-07 00:59:31 -05:00
ergosteur 41e7c5e206 docs: update README and GEMINI.md, remove AI Studio boilerplate and .env.example 2026-03-07 00:36:46 -05:00
ergosteur f685eaebd7 feat: enhance story viewer and media playback experience 2026-03-07 00:31:04 -05:00
ergosteur 47e44ec5e9 feat: improve archive parsing, add .json.xz support, and fix profile pic display 2026-03-07 00:03:54 -05:00
ergosteur cd7dc5f981 feat: Initialize InstaArchive PWA project
Sets up a new React PWA project with Vite, Tailwind CSS, and basic PWA features. Includes essential files like README, .gitignore, package.json, and initial app structure.
2026-03-06 22:41:48 -05:00
ergosteurandGitHub a724e5bc87 Initial commit 2026-03-06 22:41:33 -05:00
29 changed files with 115 additions and 2386 deletions
-2
View File
@@ -9,5 +9,3 @@ coverage/
!.env.example
_sample-archives
_gemini-plans
__pycache__/
*.pyc
+11 -8
View File
@@ -15,6 +15,9 @@ InstaArchive Viewer is a React 19 + Vite 6 PWA for browsing archived Instagram d
- `npm run lint` — type-check only (`tsc --noEmit`)
- `npm test` / `npm run test:watch` — vitest
- `npx vitest run src/lib/archive-patterns.test.ts` — a single test file
- `npm run jd2 -- --archives <dir> --dry-run` — generate JDownloader `.crawljob`
files for every profile on disk (see `scripts/jd2-sync.ts` and
`docs/jdownloader.md`)
Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not.
@@ -34,10 +37,10 @@ Loading is unified behind the `ArchiveFile` interface (`src/types/index.ts`, imp
An archive root holds one directory per profile plus *sidecars* that belong to it:
```
4utumn07 -> posts (base)
4utumn07 - reels -> reels
story - 4utumn07 -> stories
story highlights - 4utumn07 - Sunstory -> highlight "Sunstory"
0ct0ber19 -> posts (base)
0ct0ber19 - reels -> reels
story - 0ct0ber19 -> stories
story highlights - 0ct0ber19 - Heestory -> highlight "Heestory"
```
`src/lib/archive-grouping.ts` (shared by server and tests) folds these into a single profile with a `sources` list. Sidecars never appear as standalone archives. Each file the server returns carries its `kind`, so the client routes posts / reels / story ring / highlight circles without re-deriving naming rules.
@@ -69,7 +72,7 @@ Three different JSON shapes turn up as `.json`, so they are told apart structura
| Instaloader `.json.xz` | GraphQL node under `node` / `__typename` |
| gallery-dl sidecar | flat, `post_shortcode` + `type`, none of the above |
The gallery-dl sidecar is the only source that states what a post *is*: its `type` (`post` / `reel` / `story` / `highlight`) is Instagram's own classification, so `post.isReel` set from it beats every fallback in `post-tabs.ts`. This matters — of the 781 items in `official_band - reels`, the sidecars say only **360 are reels**; the other 421 are ordinary feed videos the clips endpoint returns via `include_feed_video`. Directory-based classification counted all 781.
The gallery-dl sidecar is the only source that states what a post *is*: its `type` (`post` / `reel` / `story` / `highlight`) is Instagram's own classification, so `post.isReel` set from it beats every fallback in `post-tabs.ts`. This matters — of the 781 items in `official_artms - reels`, the sidecars say only **360 are reels**; the other 421 are ordinary feed videos the clips endpoint returns via `include_feed_video`. Directory-based classification counted all 781.
**Dates are ranked, not last-write-wins** (`src/lib/post-dates.ts`): sidecar (what Instagram reported) beats filename (what the fetcher wrote) beats mtime (when the file hit disk, and unrelated to when it was posted). Ties keep the incumbent. Several files describe one post and they are scanned in directory order, not in order of trustworthiness, so without the ranking the date was decided by whichever file came first. Only JDownloader highlights fall to mtime at all — `parseArchiveFilename` flags those via `dateFromMtime`.
@@ -85,15 +88,15 @@ Images over 1MiB are downscaled in a Web Worker via `OffscreenCanvas`. The queue
### Profile tabs (`src/lib/post-tabs.ts`)
The grid holds **everything**, reels included, and the Reels tab is a *filtered view* of that same set. Only the Reels tab filters. The tabs were mutually exclusive until v1.7.0, which hid a lot: 1100 of `groupfandom`'s 3813 posts and 533 of `for.member`'s 1225 never appeared in the grid at all.
The grid holds **everything**, reels included, and the Reels tab is a *filtered view* of that same set. Only the Reels tab filters. The tabs were mutually exclusive until v1.7.0, which hid a lot: 1100 of `loonatheworld`'s 3813 posts and 533 of `for.heejin`'s 1225 never appeared in the grid at all.
This *approximates* Instagram rather than matching it. Instagram's grid includes a reel only if the creator shared it to feed — a per-post choice, measured live on 2026-08-16: `official_band` had 21 reels in its first 34 grid tiles, `4utumn07` just 1 in 214. That flag appears nowhere in an archive (JD2 stores no metadata, and Instaloader's `product_type` says what a post *is*, not whether it was shared to feed), so showing everything is the closest reachable behaviour. Instagram's "N posts" counter equals its grid, which is why the header counts `postsForTab(allPosts, 'posts')` and not `allPosts` — the raw list still holds both copies of a double-fetched post.
This *approximates* Instagram rather than matching it. Instagram's grid includes a reel only if the creator shared it to feed — a per-post choice, measured live on 2026-08-16: `official_artms` had 21 reels in its first 34 grid tiles, `0ct0ber19` just 1 in 214. That flag appears nowhere in an archive (JD2 stores no metadata, and Instaloader's `product_type` says what a post *is*, not whether it was shared to feed), so showing everything is the closest reachable behaviour. Instagram's "N posts" counter equals its grid, which is why the header counts `postsForTab(allPosts, 'posts')` and not `allPosts` — the raw list still holds both copies of a double-fetched post.
When checking the live site, note that grid reels link to `/reel/<code>/` while ordinary posts link to `/<user>/p/<code>/`. Matching only `/p/` silently drops every reel, which once produced a confident and completely wrong conclusion that Instagram never shows reels in the grid.
Deciding *what is a reel* has no good answer for most archives. Instagram's own marker is `product_type` on the post's GraphQL node (`clips` = reel, `feed` = ordinary feed video, `igtv`, `story`) — `__typename` is `GraphVideo` for all three, and aspect ratio does not separate them either. But:
- Only Instaloader archives carry that metadata, and only newer captures. A survey of `hazelofficial` found `product_type` on 1101 of 5919 sidecars, and just **2** posts marked `clips`.
- Only Instaloader archives carry that metadata, and only newer captures. A survey of `gibiofficial` found `product_type` on 1101 of 5919 sidecars, and just **2** posts marked `clips`.
- JDownloader archives carry none at all — media plus a `.txt` holding the bare caption.
So the viewer believes a `- reels` sidecar directory when one exists, and otherwise falls back to treating a lone video as a reel. **The fallback is a guess**: it cannot tell a reel from a feed video or an old IGTV upload, and it misses videos inside carousels.
-751
View File
@@ -1,751 +0,0 @@
# Tooling branch
> [!CAUTION]
> ## This branch must not be pushed to GitHub
>
> `tooling` is the only branch that still contains the archive-fetching
> scripts and their docs, and those name things `main` was rewritten to
> remove:
>
> - the fetch host's **public IP** (`docs/gallery-dl.md`)
> - the browser profile the session cookie is read from
> - the NAS archive path
> - the **list of Instagram accounts being archived**
>
> On 2026-08-20 `main`'s entire history was rewritten with `git filter-repo`,
> the GitHub repo was deleted and recreated, and 22 container images were
> pruned from ghcr — all to get exactly this material out of public view.
> **One push of this branch to GitHub undoes all of it.**
>
> A second cleanup would be harder than the first: after a force-push the old
> commits stayed reachable by raw SHA, and only deleting the repository
> outright removed them.
## Quick reference — getting new content onto the site
Timers are **disabled**, so this is manual today. Everything runs on
`mattellite`; it fetches, publishes to the NAS itself, and the site picks it up
with no deploy or restart.
```sh
ssh mattellite
~/gdl/gdl-cron.sh stories # ~1 min. DO THIS OFTEN - stories die in 24h
~/gdl/gdl-cron.sh profiles # ~5 min. posts, reels, highlights, stories
```
`profiles` covers everything, so it is the one to run if you only run one — but
it is no substitute for `stories`, because a story posted and expired between
two `profiles` runs is simply gone. Both are safe to run back-to-back; the
`--min-interval` floor makes a repeat a no-op rather than a re-fetch.
Then check <https://instaarchive.ergosteur.com> — new posts appear without a
redeploy. The archive index re-reads a directory when its mtime changes.
**Reading the output.** These two look alarming and are fine:
- `rsync error: ... (code 23)` / `done; 1 step(s) failed` — the `chown` to
`rslsync` failing because the ssh user is not root. Files landed.
- `No results` for a profile — it has no active story right now.
**These mean stop**, and are covered in "When a run fails":
- `429` from `scontent-*.cdninstagram.com`
- `400 Bad Request` on `/api/v1/feed/reels_media/` — the account is behind a
scraping-warning interstitial; check the browser before anything else.
Long runs: `setsid nohup ~/gdl/gdl-cron.sh profiles >/dev/null 2>&1 &` and
`tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`. Nothing reaches the
archive until a run finishes, so killing one midway is safe.
## Safety net: a global gallery-dl config
`~/.config/gallery-dl/config.json` on `mattellite` exists so that a **plain
`gallery-dl <url>` typed by hand** — for a quick manual check, outside
`gdl-sync.py` entirely — still gets the hand-paced caution settings instead of
gallery-dl's own faster defaults. It is loaded automatically; nothing needs to
reference it. `gdl-sync.py`'s own `--sleep-request`/`--sleep`/`--rate` flags
still override it as normal — this is only a floor for when nobody passed any.
```json
{
"extractor": {
"instagram": {
"api": "rest",
"cookies": ["chrome", "/home/matt/.config/google-chrome-devtools"],
"sleep-request": [12.0, 20.0],
"sleep": [5.0, 10.0],
"sleep-429": 120.0,
"retries": 8,
"videos": true
}
},
"downloader": {
"http": {
"rate": "500K",
"retries": 8
}
}
}
```
Every value here mirrors `gdl-cron.sh`'s own hand-paced defaults (see its
`SLEEP_REQUEST`/`SLEEP`/`RATE` comments) and `build_config()` in
`gdl-sync.py``api: rest` matters most: the graphql backend issues one
request PER POST for every video and carousel, the pattern that got this
account banned once already. `cookies` here is the config-file equivalent of
`--cookies-from-browser`, so a bare `gallery-dl <url>` is already
authenticated as the archive account, not anonymous.
Verified with `gallery_dl.config.load()` + `config.get(...)` (zero live
requests) — every value above loads correctly with no `--config` flag passed.
This file is **not tracked in the repo** — like `artms.db` and the state
files, it is host-local runtime config, and it embeds the same real
Chrome-profile path already documented above. Recreate it by hand (or from
this section) after a fresh `mattellite` setup.
## Remotes
| remote | what goes there |
|---|---|
| `origin` → gitea | everything: `main`, `tooling`, tags, backups |
| `github` | **`main` and the current release tag only** — it exists to run the CI/CD image build |
The other 22 release tags stay on gitea. Pushing them all to GitHub triggers
one container build per tag, because each tag carries its own workflow file.
## Guards — recreate these after a fresh clone
Neither guard is versioned, so a new clone has **no protection at all**:
```sh
git config remote.github.push refs/heads/main:refs/heads/main
cat > .git/hooks/pre-push <<'HOOK'
#!/bin/sh
remote_url="$2"
case "$remote_url" in *github.com*) ;; *) exit 0 ;; esac
while read -r _ _ remote_ref _; do
[ -z "$remote_ref" ] && continue
case "$remote_ref" in
refs/heads/main|refs/tags/*) ;;
*) echo "pre-push: refusing to push '$remote_ref' to GitHub." >&2; exit 1 ;;
esac
done
exit 0
HOOK
chmod +x .git/hooks/pre-push
```
Decide on the **remote** ref, not the local one: a delete push sends
`(delete)` as the local ref, and an earlier version of this hook rejected
every deletion because of it.
## What lives here
| path | what it is |
|---|---|
| `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
| `scripts/systemd/` | the timers actually installed on the fetch host |
| `scripts/test_gdl_sync.py` | its tests |
| `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
| `docs/gallery-dl.md` | the measurements behind every option in the fetcher — **read before changing pacing** |
| `docs/jdownloader.md` | the older JD2 flow |
| `docs/artms-instagram-accounts.txt` | the profile list passed to `--urls-file` |
## Commands
```sh
# fetch: always --dry-run first; it prints the plan and the publish step
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \
--staging <dir> --publish <user>@<nas>:<archives> \
--archive-db <db> --urls-file artms_account_links.txt --abort 50 --dry-run
# crawljobs (no npm script — package.json is kept identical to main)
npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run
```
Run the fetcher from the host whose public IP matches the browser the cookie
came from. `--abort 50` is the routine setting; omit it for a full sweep that
also catches edited carousels.
## Why there is no CLAUDE.md entry for any of this
`CLAUDE.md`, `README.md` and `package.json` are kept **byte-identical** to
`main` so that merging `main` into `tooling` never conflicts. The earlier
attempt put tooling notes in `CLAUDE.md` and a `jd2` script in `package.json`;
because `main` had *deleted* those lines, every merge re-applied the deletion.
Keep branch-specific documentation in this file, which `main` does not have.
## Running it by hand
Everything happens on **`mattellite`** — the fetch host whose public IP matches
the browser the cookie came from. Running it anywhere else is what
session-hijack detection looks for.
```sh
ssh mattellite
~/gdl/gdl-cron.sh profiles # or: stories | full-sweep
```
That is the whole thing: it wipes staging, fetches, and publishes straight to
the NAS. To drive `gdl-sync.py` directly instead — always `--dry-run` first,
which prints the plan and the exact rsync that would touch the archive:
```sh
cd ~/gdl
PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
--index https://instaarchive.ergosteur.com \
--staging ~/gdl/staging-manual \
--publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
--archive-db ~/gdl/artms.db \
--urls-file ~/gdl/artms_account_links.txt \
--sleep-request 12 20 --sleep 5 10 --rate 500K \
--abort 50 --dry-run # swap for --execute when the plan looks right
```
**Do not omit the pacing flags.** `gdl-sync.py`'s own defaults are 6-10s / 3-6s
/ 1M, which is roughly half the caution the wrapper applies. A hand-run that
leaves them off is *less* careful than the automation — which is backwards, and
was true of this very example until 2026-08-22. Prefer `gdl-cron.sh`; it
carries them for you.
`PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not
on cron's PATH and not on a non-login shell's either.
Useful variations:
```sh
--profile 0ct0ber19 --profile kimxxlip # instead of --urls-file
--only stories # one surface
--max-sources 4 # hard ceiling on what one run touches
--force # ignore --min-interval AND the probe
# cache; almost never what you want
```
### Long runs
A sync runs for minutes to hours at this pacing, so detach it rather than
holding an ssh session open:
```sh
cd ~/gdl && setsid nohup ./gdl-cron.sh profiles > /dev/null 2>&1 &
tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)
```
**Do not kill it with `pkill -f <pattern>` over ssh.** The pattern matches your
own `ssh` command line, so you kill your own shell and the sync survives — this
happened twice on 2026-08-22. Use `pkill -x chrome` style exact-name matches,
or kill the pid: `pgrep -f 'only posts,reels' | tail -1`.
Nothing reaches the archive until the run finishes: `gdl-sync.py` publishes
once, at the end, so a run killed midway leaves the archive untouched.
### The three modes
Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run —
and `sweep` did not convey that it is the exhaustive one. The wrapper rejects
the old names with a pointer rather than a bare error.
| mode | cadence | cost | why |
|---|---|---|---|
| `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped |
| `profiles` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating a profile once it reaches content already held. Catches everything **new** |
| `full-sweep` | rarely, by hand | **~420 requests** | no abort; walks every profile to the end. The only run that notices posts **edited** after we archived them (test case 15) |
The skip-archive means an infrequent `profiles` run costs barely more than a frequent
one — it only fetches what is new. Frequency buys freshness, not completeness,
except for stories.
**Pacing is deliberately slower than `gdl-sync.py`'s own defaults.** All three
modes run at `--sleep-request 12 20 --sleep 5 10 --rate 500K`, against defaults
of 6-10 / 3-6 / 1M. These are the values the 2026-08-22 runs used by hand after
the scraping warning, and they produced 0 400s and 0 429s. An archive sync has
no deadline: being slow is free, being restricted is not. Override with
`GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE` — to raise them, not lower them.
`stories` also runs at `--min-interval 8` rather than the 20h default, because
at 20h the daily timer silently did nothing whenever a manual run had happened
the previous afternoon. The floor exists to stop an *aborted restart*
re-enumerating profiles, which is a minutes-to-hours concern; a stories fetch
is one request per profile, so 8h permits at worst about twelve requests in a
day instead of six. **A stories run that skips every source now exits 75 and
prints a warning** rather than reporting success.
## Scheduling — installed on `mattellite`
systemd **user** timers, running as `matt`, with lingering enabled so they fire
without a login session:
```sh
loginctl show-user matt --property=Linger # Linger=yes
systemctl --user list-timers 'gdl-sync@*'
```
| unit | schedule | next fire (as installed) |
|---|---|---|
| `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working |
| `gdl-sync@profiles.timer` | 3rd of each month, 04:00 | 04:37:44 |
| `gdl-sync@full-sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
Unit files are version-controlled in `scripts/systemd/` and installed to
`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
the mode as its instance name and runs `gdl-cron.sh %i`.
Three settings are load-bearing:
- **`RandomizedDelaySec=45m`** — a job firing at exactly 09:00 daily is
obviously a machine, and the entire safety model is about not looking like
one. This is why the table above shows 09:36 rather than 09:00.
- **`Persistent=true`** — catch up a run missed because the host was off.
cron silently skips, and a skipped `stories` run is content gone for good.
- **`TimeoutStartSec=infinity`** — a sweep can run for hours at this pacing.
The default 90s would kill it mid-fetch.
Operating them:
```sh
export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed over non-interactive ssh
systemctl --user start gdl-sync@stories.service # run one now
systemctl --user status gdl-sync@profiles.timer
journalctl --user -u 'gdl-sync@*' -n 50
systemctl --user disable --now gdl-sync@full-sweep.timer # stop one
```
`systemctl --user` fails with "Failed to connect to bus" over ssh unless
`XDG_RUNTIME_DIR` is set. Note also that **month names are not valid in
`OnCalendar`'s date field** — `Jan,Apr,Jul,Oct-07` is rejected outright, hence
`*-01,04,07,10-07`. Check any change with `systemd-analyze calendar '<expr>'`
before installing it.
### cron, if you ever prefer it
```cron
17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh profiles
11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full-sweep
```
cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a
crontab line means newline unless escaped, so avoid it entirely. cron has no
equivalent of `Persistent=true`.
## Verifying a run
The first unattended run is **2026-08-21, around 09:36** (09:00 plus the
randomised delay). Nothing below costs an Instagram request — every check is
against the journal, local logs, or our own viewer's API.
```sh
# 1. did it run, and did it exit 0?
ssh mattellite
export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user list-timers 'gdl-sync@*' # LAST/PASSED columns
journalctl --user -u 'gdl-sync@stories.service' --since yesterday --no-pager
# 2. what did it actually fetch? (sidecars vastly outnumber media -- count media)
ls -1t ~/gdl/logs | head -3
grep -E '^(==>| FAILED|done;)' ~/gdl/logs/stories-*.log | tail -20
grep -ic 429 ~/gdl/logs/stories-*.log # MUST be 0 -- see below
```
**A `429` from `scontent-*.cdninstagram.com` ends the session, it is not a
pacing knob to tune.** The warning order last time was CDN 429 → `400` on the
highlights endpoint → suspension. If a run logs one, disable the timers and
stop for the day:
```sh
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
```
Then confirm the archive actually grew, from the workstation:
```sh
# 3. did the publish land? compare against yesterday's counts
for u in 0ct0ber19 kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
printf '%-18s %s\n' "$u" "$n"
done
```
Counts after the 2026-08-22 runs, to diff against:
| profile | files |
|---|---:|
| 0ct0ber19 | 3160 |
| official_artms | 6760 |
| cher_ryppo | 3056 |
| kimxxlip | 3021 |
| zindoriyam | 2256 |
| withaseul | 1760 |
A stories-only run adds few files and often **none** — profiles frequently have
no active story. "0 new" is a normal result, not a failure. `fileCount` in
`/api/archives` is stale by design; use the per-profile `/files` listing.
## 2026-08-21 — scraping warning, automation stopped
**Instagram flagged the account.** Not suspended: an interstitial at
`/accounts/scraping_warning/` reading *"We suspect automated behaviour on your
account"*. It was dismissed in the browser and the account is healthy — feed
loads, still signed in. **All three timers are disabled.** Do not re-enable
them without deciding the cadence question below.
How it unfolded, because each step misled in a different way:
1. **09:13** the daily timer fired, exited 0 in one second, logged
`done; 0 step(s) failed` — and fetched nothing. Yesterday's manual run was
18.919.2h earlier, just under the `--min-interval 20` floor, so all six
sources were skipped. **A silent no-op on the one surface that cannot be
backfilled, reported as success.**
2. **20:28** `chrome-devtools.service` was OOM-killed (5.1 GB peak, ~1w3d CPU).
Unrelated to the above, and it does not break fetching — gallery-dl reads
the cookie *file*, not a live browser — but it meant no browser was running
to notice anything was wrong.
3. **23:19** a manual recovery run passed the floor (33h) and every source
failed with `400 Bad Request` on
`/api/v1/feed/reels_media/?reel_ids=…`. Six identical failures across six
profiles is not a per-profile fault.
4. Cookies were **exported and checked before assuming a block**:
`sessionid` 77 chars, printable, colon-delimited, 360 days to expiry;
`ds_user_id` present. Decryption was fine, so the fault was server-side.
This check costs no Instagram requests and should always come first.
5. The browser then showed the interstitial. The 400s were the challenge
state, not a ban.
### What has to change before automation is re-enabled
- **The `--min-interval` floor silently defeats the daily job.** Any manual run
in the preceding 20h makes the scheduled one a no-op. The floor exists to
stop an *aborted restart* re-enumerating profiles — a minutes-to-hours
concern — and a stories fetch is one request per profile. `stories` should
use something like `--min-interval 8`, not 20.
- **A skipped stories run must be loud.** `0 to sync, 6 skipped` currently
exits 0 and looks identical to success. On this surface a skip is a real
loss, and it should be visible in the journal without reading the log.
- **Reconsider the daily cadence itself.** A job hitting story endpoints for
six profiles every morning is the most machine-like thing here, randomised
delay or not, and it is what was flagged. Every-few-days, or on-demand, may
be the honest answer even though stories will be missed.
- **`chrome-devtools.service` has `Restart=no`** and died silently for three
hours. It needs `Restart=on-failure` and probably a `MemoryMax=`, or it will
be dead the next time the cookie needs refreshing.
### 2026-08-22 — caught up by hand, cleanly
Both surfaces were fetched manually the next day, on the owner's call, at
**roughly double the configured caution**: `--sleep-request 12 20`,
`--sleep 5 10`, `--rate 500K`, versus the defaults of 6-10 / 3-6 / 1M.
| run | result |
|---|---|
| stories, 6 profiles | 16 media, +21 files, **0 400s, 0 429s** |
| posts+reels, 12 sources, `--abort 50` | 26 media, +64 files, **0 400s, 0 429s** |
So the 400s really were the challenge state and nothing more: once the
interstitial was dismissed in the browser, the same endpoints served normally.
The stories that looked lost — `official_artms`, `0ct0ber19`, `cher_ryppo`,
`kimxxlip`, `zindoriyam` — were all captured before expiry.
**This does not retire the warning.** Two hand-paced runs a day later are not
evidence that the previous cadence was safe; they are evidence that the
account still works. What actually changed the request cost is `--abort 50`:
twelve sources across six profiles, `official_artms` included at 1829 posts
and 781 reels, finished in minutes for a few dozen requests where the old
behaviour would have spent ~400.
Note the gap this leaves: **the scheduled `profiles` mode still uses the default
6-10s pacing**, not the 12-20s used here. Reconcile that before re-enabling
the timers, or the automation will be less careful than the hand runs that
followed a warning.
## When a run fails
Worked through on 2026-08-22. Do these **in order** — the first two cost no
Instagram requests, and the third costs one page view.
### 1. Read the error, not the exit code
```sh
L=~/gdl/logs/$(ls -1t ~/gdl/logs | head -1); grep -vE '^(profiles|surfaces|sources|pacing|staging|publish|read )' "$L" | head -40
```
Two results are **not** failures, despite how they look:
| looks like | actually |
|---|---|
| `rsync error: some files/attrs were not transferred (code 23)` and `done; 1 step(s) failed` | the `chown` to `rslsync` failing because the ssh user is not root. Data landed. Confirm by re-running the rsync with `--dry-run`: an empty file list means everything arrived |
| a source reporting `No results` | that profile simply has no active story / no highlights |
### 2. Check the cookies before concluding you are blocked
Free, and it separates a local fault from a server-side one:
```sh
PATH=$HOME/.local/bin:$PATH gallery-dl \
--cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools" \
--cookies-export /tmp/ck.txt
grep instagram /tmp/ck.txt | awk '{print $6, length($7)}' # names + value lengths
rm -f /tmp/ck.txt
```
A healthy `sessionid` is ~77 chars, printable ASCII, colon-delimited, and
unexpired; `ds_user_id` should be an 11-digit number. Garbage or non-printable
values mean Chrome's cookie decryption failed locally — not that the account is
in trouble. **Never print the values into a transcript or a bug report.**
### 3. Look at the account in the browser
The session lives in a dedicated Chrome on `mattellite`, on display `:1` with
CDP on 9222, run by a systemd unit:
```sh
export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user status chrome-devtools.service
systemctl --user start chrome-devtools.service # it has Restart=no
```
Then VNC to `:1` and open `instagram.com`. What you are looking for:
- **`/accounts/scraping_warning/`** — *"We suspect automated behaviour on your
account"*. This is what `400 Bad Request` on `/api/v1/feed/reels_media/`
actually means; the session is in a challenge state, not banned. Dismissing
it in the browser restores API access immediately, verified 2026-08-22.
- a checkpoint or login page — the session is gone; re-log in the browser.
- a normal feed — the fault is elsewhere.
**Do not dismiss a warning and immediately resume.** The dismissal fixes the
symptom. The behaviour that caused it is the thing to change.
### 4. Stop, if it was a 429 or a repeated 400
```sh
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
```
The documented escalation is CDN 429 → 400 on a stories/highlights endpoint →
suspension. It has now run twice, and both times the 400 was the last warning
before something worse.
## Reels: the API is blocked, scrape by scrolling instead
As of 2026-08-26/27, `gallery-dl`'s dedicated reels extractor fails on every
profile with `HTTP redirect to home page` — confirmed hours apart, with a
freshly-warmed session and a correct `X-IG-WWW-Claim` header (that was the
first suspect; ruled out by tracing the raw HTTP exchange). It is not a
scraping-warning interstitial — the reels tab loads completely normally in a
real, already-signed-in browser — so this is Meta blocking the specific
`/api/v1/clips/user/` endpoint gallery-dl calls, not an account-health issue.
Posts, highlights and stories are unaffected; only reels breaks.
`scripts/reels-scrape.py` works around it by never calling that endpoint:
it drives the same signed-in Chrome via its loopback CDP port (`:9222`,
already exposed for MCP automation), scrolls the reels tab like a person
would, and scrapes `/reel/<code>/` links out of the rendered page. It only
finds shortcodes — nothing is downloaded until the fetch step — and dedupes
against the **whole archive**, not just the profile being scraped (see why
below), so re-running it costs nothing for reels already held anywhere.
### The easy way: `reels-sync.sh`
```sh
ssh mattellite
~/gdl/reels-sync.sh zindoriyam
# or a full URL: ~/gdl/reels-sync.sh https://www.instagram.com/zindoriyam/
```
Runs both steps (scrape, then fetch + publish whatever's new) with the same
hand-paced settings `gdl-cron.sh` uses, logs to `~/gdl/logs/reels-<profile>-*`,
and exits 0 with "no new reels" printed when a profile is already caught up
`gdl-sync.py` never even gets invoked in that case. Override pacing the
same way as `gdl-cron.sh` (`GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE`), plus
`GDL_SCROLL_PAUSE` and `GDL_MAX_IDLE_ROUNDS` for the scrape step. Staging
(`~/gdl/staging-reels-<profile>`) and the scraped URL list
(`~/gdl/<profile>-reels.txt`) are wiped at the START of the next run, not
after — left behind for inspection, same as `gdl-cron.sh`'s `staging-*`.
Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the
page, 10 new ones fetched and published cleanly on the first run.
**Not wired into `gdl-cron.sh` or the timers.** Both scripts drive your
actual browser session rather than firing a background request, and are
slower by design (real scrolling, not an API call) — both good reasons not
to run this unattended without deciding that deliberately. Today it is a
per-profile, by-hand tool only.
### Where a shared reel lands
A reel found on one profile's `/reels/` page is not necessarily *owned* by
that profile — reposts and collabs between tracked accounts show up there
too. It always gets filed under its **true owner**, per Instagram's own
metadata on the post, never under whichever profile's page you happened to
scrape it from — `gdl-sync.py`'s directory template for a scraped reel is
`{username}` filled in from that metadata, the same mechanism highlights
already used for their own directory. So:
- Scrape order doesn't matter. Run `reels-sync.sh` on `0ct0ber19` or
`zindoriyam` first, whichever — a reel they share lands in the same place
either way, and running it on the other one afterward just sees that
shortcode as already archived (dedup checks every profile, not only the
one being scraped) and skips it.
- It never gets duplicated into both accounts' directories, and it never
gets misattributed to the profile you scraped instead of who actually
posted it.
This is also literally why the whole-archive dedup fix above exists: a
zindoriyam-scraped shortcode that turned out to belong to `0ct0ber19` was
filed under `0ct0ber19/`, not `zindoriyam/` — dedup that only checked
`zindoriyam`'s own directory would never have found it there.
### What it's doing, if you want to run the two steps separately
```sh
PROFILE=someuser
# 1. Scrape by scrolling; dedupe against the archive; write new URLs to a file.
# Needs gallery-dl's own pipx venv python -- that's where websocket-client
# (the one extra dependency this needs) got injected.
~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/reels-scrape.py \
--profile "$PROFILE" \
--index https://instaarchive.ergosteur.com \
--out ~/gdl/"$PROFILE"-reels.txt
# 2. Fetch whatever's new -- dry run first, same as any other gdl-sync.py call.
cd ~/gdl && PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
--publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
--staging ~/gdl/staging-reels-scraped \
--sleep-request 12 20 --sleep 5 10 --rate 500K \
--post-urls-file ~/gdl/"$PROFILE"-reels.txt --dry-run
# ...then swap --dry-run for --execute once the plan looks right.
```
Notes:
- `reels-scrape.py` fails fast if Chrome/CDP isn't up
(`curl -s http://localhost:9222/json/version` to check first).
- If it finds nothing new, `gdl-sync.py --post-urls-file` refuses to run
("no usable URLs") rather than doing nothing quietly — expected when a
profile is already caught up.
- `--max-idle-rounds` (default 3) and `--scroll-pause MIN MAX` (default
`2.0 3.5`) are worth raising for an unusually large or slow-loading reels
tab; the default stops once 3 consecutive scrolls find nothing new.
### Why dedup checks the whole archive, not just the scraped profile
First version deduped only against the scraped profile's own directories.
Re-running it on `zindoriyam` minutes after a successful fetch found "9 new"
reels again — all reposts/collabs originally by `0ct0ber19` and
`official_artms` (also tracked profiles). The *old*, broken direct-reels-tab
fetch had left orphaned sidecar-only remnants (`.json`/`.txt`, no media) for
them, misfiled under `zindoriyam - reels/` with the true owner's name baked
into the filename stem — `index_existing()` correctly does not count a
sidecar-only entry as "held", so they looked new. `gdl-sync.py` re-fetched
them, filed them correctly under `{username}` (the post's *true* owner, per
its own metadata) — where they already existed from that profile's own
regular sync — and `rsync --ignore-existing` silently skipped every one, so
nothing was lost or duplicated. But 9 Instagram requests were spent finding
that out. Since a shortcode is globally unique, `reels-scrape.py` now checks
every archived profile's listing, not just the one being scraped — all local
requests to the viewer's own API, never to `instagram.com`, so checking all
16 profiles costs nothing on the budget that actually matters. Confirmed
fixed the same day: re-running against `zindoriyam` immediately afterward
found "26 already archived, 0 new" and exited clean.
## What changed on 2026-08-20
One session, three separate pieces of work. Recorded because the reasons are
not recoverable from the diffs.
**The sync run.** First incremental fetch in four days: 184 new media, 299
files published, 0 failures, 0 CDN 429s. 20 story items, which are the part
that could not have been recovered later. Cost about half what it would have,
because the archive DB was already seeded and the state file was primed by hand
so no probe passes ran.
**`--abort 50`.** The skip-archive suppresses *downloads*, which spends the
CDN; it does nothing about the *listing pass*, which spends `instagram.com` and
scales with how big a profile is rather than how much is new. Measured from
sidecar write times mid-run: 3 new posts took ~100s each, the other 2272 were
written in one second. Enumerating `cher_ryppo` fell from 2151 posts to 7.
**The repo split.** `main` is public and now carries none of the fetching
tooling, no host details, and no real account names — its entire history was
rewritten, the GitHub repo deleted and recreated to clear force-push residue,
and 22 container images pruned from ghcr because the server bundle had been
shipping source comments naming real accounts. This branch holds everything
that was removed. See the caution at the top.
**Automation.** mattellite got a key on the NAS, closing the last manual step,
and three systemd timers now run the sync unattended.
## Outstanding
State as of 2026-08-20, after the sync run and the repo split. Nothing here is
broken; these are decisions not yet made and cleanups not yet done.
### Fetching
- **The timers are DISABLED** after the 2026-08-21 scraping warning; see that
section. Their one unattended firing did the wrong thing — it skipped every
source on the 20h floor and reported success — so re-enabling should wait on
the four fixes listed there, not just on the account settling.
- `mattellite`'s `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys` for
`agentapi` (added 2026-08-20, alongside the workstation's existing key), so
the fetch host publishes straight to the archive and no `sshpass` step is
needed. **That key is what makes the timers work** — remove it and every
scheduled run will fetch successfully and then fail at publish.
- **5.4 GB of stale staging on `mattellite`** (`~/gdl`, 44 GB free): `staging`
and `out` at 2.2 GB each from 2026-08-17, `staging-0820` / `out-0820` at
446 MB each, plus `staging-full` (78 MB) and `staging-stories` (66 MB) from
2026-08-22. All of it was verified published, so all of it is safe to delete.
Staging is wiped per-run by `gdl-cron.sh`, but the `out-*` publish targets and
anything created by a direct `gdl-sync.py` call are never cleaned up.
- **Stories currently depend on someone remembering.** The daily timer exists
but is disabled, so the one surface that cannot be backfilled has no
automation behind it. Every day nobody runs `gdl-cron.sh stories` is a day
of stories gone. That is the central unresolved tension: the cadence that
protects stories is also the most machine-like pattern here.
- ~~The scheduled modes have not been reconciled with the pacing used by
hand.~~ **Done 2026-08-22**: all modes now pass `--sleep-request 12 20
--sleep 5 10 --rate 500K`, `stories` uses `--min-interval 8`, and a
fully-skipped stories run exits 75 with a warning instead of looking like a
success. The timers are still **disabled** — enabling them is a separate
decision about cadence, not about pacing.
- **`--abort 50` is opt-in.** `gdl-cron.sh profiles` passes it and the manual
runs used it; `full-sweep` deliberately does not. It stops noticing **edited
carousels** (test case 15), which only a full enumeration finds — which is
what `full-sweep` is for. No cadence for it was ever agreed, and at ~420
requests it is the riskiest thing on the schedule; prefer running it by hand.
- **The `seeded` flags in `<db>.state.json` were hand-written**, reconstructed
from the 2026-08-17 log rather than derived from the archive DB. They assert
"the skip-archive already knows this source". If `artms.db` is ever rebuilt,
moved or lost, **clear the state file too** — otherwise those sources will
never re-seed and a fetch into empty staging re-downloads everything.
- **`~/gdl/gdl-sync.py` on the fetch host is a copy, not a checkout.** It
currently matches this branch (`96e5694e…`), but nothing keeps them in sync;
`scp` it after any change and re-check the hash.
- The 2026-08-20 run is split across two logs — `artms-run3.log` (12 sources,
no abort) and `artms-run4.log` (12 sources, `--abort 50`) — because it was
stopped midway to pick up the new flag.
### Repo and infrastructure
- **The `pre-rewrite-*` branches on gitea hold the unredacted history** — real
account names, the fetch host's IP, and the tooling, as it was before the
rewrite. They are deliberate backups. Decide whether they expire; the
`pre-push` hook does cover them (it allows only `main` and tags to GitHub).
- **The `pre-rewrite-full.bundle` backup is in a session scratchpad** and will
be deleted with it. If a durable backup outside gitea is wanted, move it now.
- **Only one container image exists.** 22 versions were pruned, so rolling back
to an older release means checking out its tag from gitea and pushing that
tag to GitHub to rebuild it — the old images are gone, not archived.
- **CI warns that the Node 20 actions are deprecated.** `actions/checkout@v4`,
`docker/login-action@v3`, `docker/metadata-action@v5` and
`docker/build-push-action@v5` are being forced onto Node 24. They work today;
bump when convenient.
- **`review-fixes`** on gitea is a stale v1.3.0-era branch, never merged,
published only because the whole local repo was pushed. Probably deletable.
- GitHub Actions run history was lost when the repo was recreated. Cosmetic.
-221
View File
@@ -1,221 +0,0 @@
[
[
2,
{
"category": "instagram",
"coauthors": [
{
"full_name": "\ucd5c\ub9ac \ucd5c\uc608\ub9bc Choerry",
"id": "57668363710",
"username": "cher_ryppo"
},
{
"full_name": "\uc9c4\uc194 \ud835\udc09\ud835\udc08\ud835\udc0d\ud835\udc12\ud835\udc0e\ud835\udc14\ud835\udc0b",
"id": "55465048543",
"username": "zindoriyam"
},
{
"full_name": "Official ARTMS",
"id": "58524253183",
"username": "official_artms"
},
{
"full_name": "HEEJIN",
"id": "57723176039",
"username": "0ct0ber19"
},
{
"full_name": "\uae40\ub9bd KIMLIP",
"id": "57506288936",
"username": "kimxxlip"
}
],
"count": 1,
"date": "2026-08-18 07:18:54",
"description": "#\uc81c\uc791\uc9c0\uc6d0 ARTMS(\uc544\ub974\ud14c\ubbf8\uc2a4) \u2018Blue Blood Night\u2019 \ub9ac\uc2a4\ub2dd \ud30c\ud2f0 \ud83c\udf19\n\n\uc131\uc218\ub3d9\uc5d0\uc11c \ud3bc\uccd0\uc9c4 ARTMS\uc640 300\uba85 \ud32c\ub4e4\uc758 \uc5ed\ub300\uae09 \ubc24!\n\nARTMS\uac00 \uc57d 300\uba85\uc758 \ud32c\ub4e4\uacfc \ud568\uaed8 \ud074\ub7fd \ud615\ud0dc\uc758\n\ub3c5\ud2b9\ud558\uace0 \ubabd\ud658\uc801\uc778 \uacf5\uac04\uc5d0\uc11c \uc624\ud504\ub77c\uc778 \ud32c \uc774\ubca4\ud2b8\n\u2018Blue Blood Night\u2019 \ub9ac\uc2a4\ub2dd \ud30c\ud2f0\ub97c \uac1c\ucd5c\ud588\uc2b5\ub2c8\ub2e4 \u2728\n\n\ud558\uc774 \ub370\ud328\ub274\ub3c4 \uc678\uce58\uace0 ARTMS \uc0ac\ub791\ud574\uc694\ub3c4 \uc678\ucce4\ub358\n\ub728\uac70\uc6e0\ub358 \ud604\uc7a5\uc5d0 \ub370\ud328\ub274\uac00 \ub2e4\ub140\uc654\uc2b5\ub2c8\ub2e4 \ud83d\ude0e\n\n\ud83d\udccc\uc544\ub974\ud14c\ubbf8\uc2a4 (ARTMS) \n@official_artms \n\n\u25aa\ufe0f2024\ub144\uc5d0 \ub370\ubdd4\ud55c \ubaa8\ub4dc\ud558\uc6b0\uc2a4(MODHAUS) \uc18c\uc18d\uc758 5\uc778\uc870 \uac78\uadf8\ub8f9 \n\u25aa\ufe0f\uba64\ubc84: \ud76c\uc9c4, \ud558\uc2ac, \uae40\ub9bd, \uc9c4\uc194, \ucd5c\ub9ac \n\u25aa\ufe0f\uc774\ub2ec\uc758 \uc18c\ub140 \ucd9c\uc2e0 \uba64\ubc84\ub4e4\uc774 \ub73b\uc744 \ubaa8\uc544 \uacb0\uc131\ud55c \ud504\ub85c\uc81d\ud2b8\uc774\uc790 \uc815\uc2dd \uac78\uadf8\ub8f9 \n\u25aa\ufe0f\uccab \uc815\uaddc \uc568\ubc94\uacfc \ud0c0\uc774\ud2c0\uace1 \u2018Virtual Angel\u2019 \ub4f1\uc744 \uc120\ubcf4\uc784\n\n#ARTMS #\uc544\ub974\ud14c\ubbf8\uc2a4 #BlueBloodNight\n\n\ud83c\udfa5 @dailyfashion_news",
"fullname": "\u270d\ud83c\udffb\ud83e\udd13\ud328\uc158 \ub274\uc2a4 \uc694\uc57d \u2b50\ufe0f \ub370\ud328\ub274\ud83d\ude0e\ud83d\udc85\ud83c\udffb",
"liked": false,
"likes": 7219,
"owner_id": "38980901318",
"pinned": [],
"post_date": "2026-08-18 07:18:54",
"post_id": "3966279735525149864",
"post_shortcode": "DcLDme7zoyo",
"post_url": "https://www.instagram.com/reel/DcLDme7zoyo/",
"subcategory": "reel",
"tags": [
"#ARTMS",
"#BlueBloodNight",
"#\uc544\ub974\ud14c\ubbf8\uc2a4",
"#\uc81c\uc791\uc9c0\uc6d0"
],
"type": "reel",
"username": "dailyfashion_news"
}
],
[
3,
"ytdl:https://www.instagram.com/reel/DcLDme7zoyo/1.mp4",
{
"category": "instagram",
"coauthors": [
{
"full_name": "\ucd5c\ub9ac \ucd5c\uc608\ub9bc Choerry",
"id": "57668363710",
"username": "cher_ryppo"
},
{
"full_name": "\uc9c4\uc194 \ud835\udc09\ud835\udc08\ud835\udc0d\ud835\udc12\ud835\udc0e\ud835\udc14\ud835\udc0b",
"id": "55465048543",
"username": "zindoriyam"
},
{
"full_name": "Official ARTMS",
"id": "58524253183",
"username": "official_artms"
},
{
"full_name": "HEEJIN",
"id": "57723176039",
"username": "0ct0ber19"
},
{
"full_name": "\uae40\ub9bd KIMLIP",
"id": "57506288936",
"username": "kimxxlip"
}
],
"count": 1,
"date": "2026-08-18 07:18:54",
"description": "#\uc81c\uc791\uc9c0\uc6d0 ARTMS(\uc544\ub974\ud14c\ubbf8\uc2a4) \u2018Blue Blood Night\u2019 \ub9ac\uc2a4\ub2dd \ud30c\ud2f0 \ud83c\udf19\n\n\uc131\uc218\ub3d9\uc5d0\uc11c \ud3bc\uccd0\uc9c4 ARTMS\uc640 300\uba85 \ud32c\ub4e4\uc758 \uc5ed\ub300\uae09 \ubc24!\n\nARTMS\uac00 \uc57d 300\uba85\uc758 \ud32c\ub4e4\uacfc \ud568\uaed8 \ud074\ub7fd \ud615\ud0dc\uc758\n\ub3c5\ud2b9\ud558\uace0 \ubabd\ud658\uc801\uc778 \uacf5\uac04\uc5d0\uc11c \uc624\ud504\ub77c\uc778 \ud32c \uc774\ubca4\ud2b8\n\u2018Blue Blood Night\u2019 \ub9ac\uc2a4\ub2dd \ud30c\ud2f0\ub97c \uac1c\ucd5c\ud588\uc2b5\ub2c8\ub2e4 \u2728\n\n\ud558\uc774 \ub370\ud328\ub274\ub3c4 \uc678\uce58\uace0 ARTMS \uc0ac\ub791\ud574\uc694\ub3c4 \uc678\ucce4\ub358\n\ub728\uac70\uc6e0\ub358 \ud604\uc7a5\uc5d0 \ub370\ud328\ub274\uac00 \ub2e4\ub140\uc654\uc2b5\ub2c8\ub2e4 \ud83d\ude0e\n\n\ud83d\udccc\uc544\ub974\ud14c\ubbf8\uc2a4 (ARTMS) \n@official_artms \n\n\u25aa\ufe0f2024\ub144\uc5d0 \ub370\ubdd4\ud55c \ubaa8\ub4dc\ud558\uc6b0\uc2a4(MODHAUS) \uc18c\uc18d\uc758 5\uc778\uc870 \uac78\uadf8\ub8f9 \n\u25aa\ufe0f\uba64\ubc84: \ud76c\uc9c4, \ud558\uc2ac, \uae40\ub9bd, \uc9c4\uc194, \ucd5c\ub9ac \n\u25aa\ufe0f\uc774\ub2ec\uc758 \uc18c\ub140 \ucd9c\uc2e0 \uba64\ubc84\ub4e4\uc774 \ub73b\uc744 \ubaa8\uc544 \uacb0\uc131\ud55c \ud504\ub85c\uc81d\ud2b8\uc774\uc790 \uc815\uc2dd \uac78\uadf8\ub8f9 \n\u25aa\ufe0f\uccab \uc815\uaddc \uc568\ubc94\uacfc \ud0c0\uc774\ud2c0\uace1 \u2018Virtual Angel\u2019 \ub4f1\uc744 \uc120\ubcf4\uc784\n\n#ARTMS #\uc544\ub974\ud14c\ubbf8\uc2a4 #BlueBloodNight\n\n\ud83c\udfa5 @dailyfashion_news",
"display_url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.82787-15/777982919_18124593284301319_6754878398807305029_n.jpg?stp=dst-jpg_e15_tt6&_nc_cat=107&ig_cache_key=Mzk2NjI3OTczNTUyNTE0OTg2NDE4MTI0NTkzMjgxMzAxMzE5.3-ccb7-5&ccb=7-5&_nc_sid=58cdad&efg=eyJ2ZW5jb2RlX3RhZyI6IkNMSVBTLnhwaWRzLjEzMjAuc2RyLnZpZGVvX2RlZmF1bHRfY292ZXJfZnJhbWUuQzMifQ%3D%3D&_nc_ohc=VTiIkvHCoBMQ7kNvwFnD08V&_nc_oc=Adp2K1e-pF18avq4GpMbXRgiZv_T6ySFEqTFdXPRtemyWnFeG8p6cgvxaXxZtyDDw-U&_nc_ad=z-m&_nc_cid=0&_nc_zt=23&_nc_ht=instagram.fyto3-1.fna&_nc_gid=-Kd8mdFfght8nFvwkJmRPA&_nc_ss=7a22e&oh=00_AQIKaJEGeU5iyrLIjnfXozkpbabHpemHWlDoSEEx11SMFQ&oe=6A9D5761",
"extension": "mp4",
"filename": "AQNbHAasCyJwuQWdYi7vWw3w6iCiMdBAXZQ58cnOKhanFZgw_IAhVC_f380GVo23sNeeCD559rVhmUSr0bmruolOjAgXMVwrmoRu88A",
"fullname": "\u270d\ud83c\udffb\ud83e\udd13\ud328\uc158 \ub274\uc2a4 \uc694\uc57d \u2b50\ufe0f \ub370\ud328\ub274\ud83d\ude0e\ud83d\udc85\ud83c\udffb",
"height": 1920,
"height_original": 1920,
"liked": false,
"likes": 7219,
"media_id": "3966279735525149864",
"num": 1,
"owner": {
"account_badges": [],
"account_type": 3,
"can_see_quiet_post_attribution": true,
"eligible_for_text_app_activation_badge": false,
"fan_club_info": {
"autosave_to_exclusive_highlight": null,
"connected_member_count": null,
"fan_club_id": null,
"fan_club_name": null,
"fan_consideration_page_revamp_eligiblity": null,
"has_created_ssc": null,
"has_enough_subscribers_for_ssc": null,
"is_fan_club_gifting_eligible": null,
"is_fan_club_referral_eligible": null,
"is_free_trial_eligible": null,
"largest_public_bc_id": null,
"should_show_playlists_in_profile_tab": null,
"subscriber_count": null
},
"fbid_v2": "17841439039552612",
"feed_post_reshare_disabled": false,
"friendship_status": {
"followed_by": false,
"following": false,
"is_bestie": false,
"is_feed_favorite": false,
"is_muting_reel": false,
"is_private": false,
"is_restricted": false
},
"full_name": "\u270d\ud83c\udffb\ud83e\udd13\ud328\uc158 \ub274\uc2a4 \uc694\uc57d \u2b50\ufe0f \ub370\ud328\ub274\ud83d\ude0e\ud83d\udc85\ud83c\udffb",
"has_anonymous_profile_picture": false,
"hd_profile_pic_url_info": {
"height": 1080,
"url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.2885-19/283399158_1201757910664534_1998021056617019217_n.jpg?efg=eyJ2ZW5jb2RlX3RhZyI6InByb2ZpbGVfcGljLmRqYW5nby4xMDgwLmMyIn0&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_cat=1&_nc_oc=Q6cZ2gE8DUM4skVKFnTxJ_6S8QN3RWaHsgKT9sigiWMRE8sqvl9r488ADsJFU6ErFHln2eg&_nc_ohc=LjOAkQaNPtMQ7kNvwHhtSrv&_nc_gid=-Kd8mdFfght8nFvwkJmRPA&edm=ALQROFkBAAAA&ccb=7-5&oh=00_AQLdX6t5krhp725dzvNOvQbGOFpVdvwgZWqDkFvv1AeQPA&oe=6A9D6555&_nc_sid=fc8dfb",
"width": 1080
},
"hd_profile_pic_versions": [
{
"height": 320,
"url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.2885-19/283399158_1201757910664534_1998021056617019217_n.jpg?stp=dst-jpg_s320x320_tt6&efg=eyJ2ZW5jb2RlX3RhZyI6InByb2ZpbGVfcGljLmRqYW5nby4xMDgwLmMyIn0&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_cat=1&_nc_oc=Q6cZ2gE8DUM4skVKFnTxJ_6S8QN3RWaHsgKT9sigiWMRE8sqvl9r488ADsJFU6ErFHln2eg&_nc_ohc=LjOAkQaNPtMQ7kNvwHhtSrv&_nc_gid=-Kd8mdFfght8nFvwkJmRPA&edm=ALQROFkBAAAA&ccb=7-5&oh=00_AQKux7g7JpZsP_Q7z6rX3bFlcdVCUCwwDAQfmRRWbB6ePQ&oe=6A9D6555&_nc_sid=fc8dfb",
"width": 320
},
{
"height": 640,
"url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.2885-19/283399158_1201757910664534_1998021056617019217_n.jpg?stp=dst-jpg_s640x640_tt6&efg=eyJ2ZW5jb2RlX3RhZyI6InByb2ZpbGVfcGljLmRqYW5nby4xMDgwLmMyIn0&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_cat=1&_nc_oc=Q6cZ2gE8DUM4skVKFnTxJ_6S8QN3RWaHsgKT9sigiWMRE8sqvl9r488ADsJFU6ErFHln2eg&_nc_ohc=LjOAkQaNPtMQ7kNvwHhtSrv&_nc_gid=-Kd8mdFfght8nFvwkJmRPA&edm=ALQROFkBAAAA&ccb=7-5&oh=00_AQJJz0fXTbLsfMqkiG7KHQtZbaHwHAU8nS1eTzB0sQqRSA&oe=6A9D6555&_nc_sid=fc8dfb",
"width": 640
}
],
"id": "38980901318",
"is_active_on_text_post_app": true,
"is_embeds_disabled": false,
"is_favorite": false,
"is_private": false,
"is_ring_creator": false,
"is_unpublished": false,
"is_verified": true,
"latest_reel_media": 1788306389,
"pk": "38980901318",
"pk_id": "38980901318",
"profile_pic_id": "2844231369895585868_38980901318",
"profile_pic_url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.2885-19/283399158_1201757910664534_1998021056617019217_n.jpg?stp=dst-jpg_s150x150_tt6&efg=eyJ2ZW5jb2RlX3RhZyI6InByb2ZpbGVfcGljLmRqYW5nby4xMDgwLmMyIn0&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_cat=1&_nc_oc=Q6cZ2gE8DUM4skVKFnTxJ_6S8QN3RWaHsgKT9sigiWMRE8sqvl9r488ADsJFU6ErFHln2eg&_nc_ohc=LjOAkQaNPtMQ7kNvwHhtSrv&_nc_gid=-Kd8mdFfght8nFvwkJmRPA&edm=ALQROFkBAAAA&ccb=7-5&oh=00_AQI28hrlvg9XAWGFkTjCyv96HLuMEZb5q6AlNbUUqw2WVQ&oe=6A9D6555&_nc_sid=fc8dfb",
"show_account_transparency_details": true,
"show_ring_award": false,
"strong_id__": "38980901318",
"text_post_app_is_private": false,
"third_party_downloads_enabled": 2,
"transparency_product_enabled": false,
"user_activation_info": {},
"username": "dailyfashion_news"
},
"owner_id": "38980901318",
"pinned": [],
"post_date": "2026-08-18 07:18:54",
"post_id": "3966279735525149864",
"post_shortcode": "DcLDme7zoyo",
"post_url": "https://www.instagram.com/reel/DcLDme7zoyo/",
"shortcode": "DcLDme7zoyo",
"subcategory": "reel",
"tagged_users": [
{
"full_name": "\uc9c4\uc194 \ud835\udc09\ud835\udc08\ud835\udc0d\ud835\udc12\ud835\udc0e\ud835\udc14\ud835\udc0b",
"id": "55465048543",
"username": "zindoriyam"
},
{
"full_name": "\uae40\ub9bd KIMLIP",
"id": "57506288936",
"username": "kimxxlip"
},
{
"full_name": "\ucd5c\ub9ac \ucd5c\uc608\ub9bc Choerry",
"id": "57668363710",
"username": "cher_ryppo"
},
{
"full_name": "HEEJIN",
"id": "57723176039",
"username": "0ct0ber19"
},
{
"full_name": "Official ARTMS",
"id": "58524253183",
"username": "official_artms"
}
],
"tags": [
"#ARTMS",
"#BlueBloodNight",
"#\uc544\ub974\ud14c\ubbf8\uc2a4",
"#\uc81c\uc791\uc9c0\uc6d0"
],
"type": "reel",
"username": "dailyfashion_news",
"video_url": "https://instagram.fyto3-1.fna.fbcdn.net/o1/v/t2/f2/m86/AQNbHAasCyJwuQWdYi7vWw3w6iCiMdBAXZQ58cnOKhanFZgw_IAhVC_f380GVo23sNeeCD559rVhmUSr0bmruolOjAgXMVwrmoRu88A.mp4?_nc_cat=105&_nc_oc=AdomTMuitkVGoe5U3qTaa9WVmrQNaUmC-hpxQMgtcKNDrT2Htj3cu6iPiBDfh9M0Hyk&_nc_sid=5e9851&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_ohc=O3edipmWdLQQ7kNvwFP9r4s&efg=eyJ2ZW5jb2RlX3RhZyI6Inhwdl9wcm9ncmVzc2l2ZS5JTlNUQUdSQU0uQ0xJUFMuQzMuNzIwLmRhc2hfYmFzZWxpbmVfMV92MSIsInhwdl9hc3NldF9pZCI6MTgxMTkyNTgxMzEyMzA0OSwiYXNzZXRfYWdlX2RheXMiOjE0LCJ2aV91c2VjYXNlX2lkIjoxMDA5OSwiZHVyYXRpb25fcyI6NTQsInVybGdlbl9zb3VyY2UiOiJ3d3cifQ%3D%3D&ccb=17-1&vs=69de96aad46e550e&_nc_vs=HBksFQIYUmlnX3hwdl9yZWVsc19wZXJtYW5lbnRfc3JfcHJvZC83RTRFMjc3MDhFMjVCMzUxNjA5MURDQUM2QTQ3MTlBQV92aWRlb19kYXNoaW5pdC5tcDQVAALIARIAFQIYUWlnX3hwdl9wbGFjZW1lbnRfcGVybWFuZW50X3YyL0I3NDdFMUM4NTRCODk2NDRENDM2NTIwQzVBODBCOTk5X2F1ZGlvX2Rhc2hpbml0Lm1wNBUCAsgBEgAoABgAGwKIB3VzZV9vaWwBMRJwcm9ncmVzc2l2ZV9yZWNpcGUBMRUAACbS35qn-_u3BhUCKAJDMywXQEtRBiTdLxsYEmRhc2hfYmFzZWxpbmVfMV92MREAdf4HZeadAQA&_nc_gid=-Kd8mdFfght8nFvwkJmRPA&_nc_zt=28&_nc_ss=7a22e&oh=00_AQLoXUFyQixJHvIcp3UE-6sEsjKuOlmhiPWmQphL7U7o5g&oe=6A996A42",
"width": 1080,
"width_original": 1080
}
]
]
-181
View File
@@ -1,181 +0,0 @@
[
[
2,
{
"audio_artist": null,
"audio_duration": 15.0,
"audio_timestamps": null,
"audio_title": "Original audio",
"audio_user": {
"full_name": "\uc2e0\ubaa8\ucc0c \ud835\udde0\ud835\uddfc\ud835\uddf0\ud835\uddf5\ud835\uddf6 \ud83c\udf58",
"id": "61396137942",
"is_private": false,
"is_verified": false,
"pk": "61396137942",
"pk_id": "61396137942",
"profile_pic_id": "3506506412954824214_61396137942",
"profile_pic_url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.2885-19/468037362_1017459960091671_5806837525855946557_n.jpg?stp=dst-jpg_s150x150_tt6&efg=eyJ2ZW5jb2RlX3RhZyI6InByb2ZpbGVfcGljLmRqYW5nby42MDAuYzIifQ&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_cat=103&_nc_oc=Q6cZ2gEBkafhMKkLe-YbSz6FVQiMLEw4juIKWvM8PNmZ2nTcMpNyjdl0kV_-fWna0tytdLA&_nc_ohc=sPpx_bESe4cQ7kNvwFvI70J&_nc_gid=RGZW24yVvqRHKLuQvnRkKQ&edm=ALQROFkBAAAA&ccb=7-5&oh=00_AQIS_52Qhu7n6Fo7XNtQaIKXGEhkwhgz-C9vpWo4yb6dnw&oe=6A9D5107&_nc_sid=fc8dfb",
"strong_id__": "61396137942",
"username": "mochi.053"
},
"category": "instagram",
"coauthors": [
{
"full_name": "HEEJIN",
"id": "57723176039",
"username": "0ct0ber19"
}
],
"count": 2,
"date": "2026-08-22 02:48:05",
"description": "\u3060\u3044\u3058\u3087\u270c\ufe0f\u3067\u3057\u3087",
"fullname": "\uce04",
"liked": true,
"likes": 178797,
"owner_id": "54301371254",
"pinned": [],
"post_date": "2026-08-22 02:48:05",
"post_id": "3969041070976751617",
"post_shortcode": "DcU3dM-gVgB",
"post_url": "https://www.instagram.com/p/DcU3dM-gVgB/",
"subcategory": "post",
"type": "post",
"username": "chuuo3o"
}
],
[
3,
"ytdl:https://www.instagram.com/p/DcU3dM-gVgB/1.mp4",
{
"audio_artist": null,
"audio_duration": 15.0,
"audio_timestamps": null,
"audio_title": "Original audio",
"audio_user": {
"full_name": "\uc2e0\ubaa8\ucc0c \ud835\udde0\ud835\uddfc\ud835\uddf0\ud835\uddf5\ud835\uddf6 \ud83c\udf58",
"id": "61396137942",
"is_private": false,
"is_verified": false,
"pk": "61396137942",
"pk_id": "61396137942",
"profile_pic_id": "3506506412954824214_61396137942",
"profile_pic_url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.2885-19/468037362_1017459960091671_5806837525855946557_n.jpg?stp=dst-jpg_s150x150_tt6&efg=eyJ2ZW5jb2RlX3RhZyI6InByb2ZpbGVfcGljLmRqYW5nby42MDAuYzIifQ&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_cat=103&_nc_oc=Q6cZ2gEBkafhMKkLe-YbSz6FVQiMLEw4juIKWvM8PNmZ2nTcMpNyjdl0kV_-fWna0tytdLA&_nc_ohc=sPpx_bESe4cQ7kNvwFvI70J&_nc_gid=RGZW24yVvqRHKLuQvnRkKQ&edm=ALQROFkBAAAA&ccb=7-5&oh=00_AQIS_52Qhu7n6Fo7XNtQaIKXGEhkwhgz-C9vpWo4yb6dnw&oe=6A9D5107&_nc_sid=fc8dfb",
"strong_id__": "61396137942",
"username": "mochi.053"
},
"category": "instagram",
"coauthors": [
{
"full_name": "HEEJIN",
"id": "57723176039",
"username": "0ct0ber19"
}
],
"count": 2,
"date": "2026-08-22 02:48:05",
"description": "\u3060\u3044\u3058\u3087\u270c\ufe0f\u3067\u3057\u3087",
"display_url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.82787-15/783856831_18043412798811255_1724949997749570795_n.jpg?stp=dst-jpg_e15_tt6&_nc_cat=108&ig_cache_key=Mzk2OTA0MTA3MDk3Njc1MTYxNzE4MDQzNDEyNzkyODExMjU1.3-ccb7-5&ccb=7-5&_nc_sid=58cdad&efg=eyJ2ZW5jb2RlX3RhZyI6IkNMSVBTLnhwaWRzLjcyMC5zZHIudmlkZW9fZGVmYXVsdF9jb3Zlcl9mcmFtZS5DMyJ9&_nc_ohc=JWzEND3gUxAQ7kNvwGDGCLf&_nc_oc=AdpwpFkh8Wj84hyF4zWcsaNB5zUnYJh2bJijNe9AxrOR1WKa08Uy35khb5BpFnG9XhM&_nc_ad=z-m&_nc_cid=0&_nc_zt=23&_nc_ht=instagram.fyto3-1.fna&_nc_gid=RGZW24yVvqRHKLuQvnRkKQ&_nc_ss=7a22e&oh=00_AQLQlu6EItD2G6QrlOmQSas2IHDOpucLIhxVWcCPFE1BOw&oe=6A9D4B4F",
"extension": "mp4",
"filename": "AQN8nhc_L3hdDSQVVRF2OXm2gKwDpsdh_jw81ZnA9Uh4nqOwukKke8QXrGCe1jrMtOXzPT5Bkuipu8RShT651VV7q6EVbjvxC-fxgUE",
"fullname": "\uce04",
"height": 1920,
"height_original": 1920,
"liked": true,
"likes": 178797,
"media_id": "3969041070976751617",
"num": 1,
"owner": {
"account_badges": [],
"account_type": 3,
"can_see_quiet_post_attribution": true,
"eligible_for_text_app_activation_badge": false,
"fan_club_info": {
"autosave_to_exclusive_highlight": null,
"connected_member_count": null,
"fan_club_id": null,
"fan_club_name": null,
"fan_consideration_page_revamp_eligiblity": null,
"has_created_ssc": null,
"has_enough_subscribers_for_ssc": null,
"is_fan_club_gifting_eligible": null,
"is_fan_club_referral_eligible": null,
"is_free_trial_eligible": null,
"largest_public_bc_id": null,
"should_show_playlists_in_profile_tab": null,
"subscriber_count": null
},
"fbid_v2": "17841454337627671",
"feed_post_reshare_disabled": false,
"friendship_status": {
"followed_by": false,
"following": true,
"is_bestie": false,
"is_feed_favorite": false,
"is_muting_reel": false,
"is_private": false,
"is_restricted": false
},
"full_name": "\uce04",
"has_anonymous_profile_picture": false,
"hd_profile_pic_url_info": {
"height": 1080,
"url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.82787-19/714930835_18032224961811255_6816644959905468206_n.jpg?efg=eyJ2ZW5jb2RlX3RhZyI6InByb2ZpbGVfcGljLmRqYW5nby4xMDgwLmMyIn0&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_cat=1&_nc_oc=Q6cZ2gEBkafhMKkLe-YbSz6FVQiMLEw4juIKWvM8PNmZ2nTcMpNyjdl0kV_-fWna0tytdLA&_nc_ohc=hpoiFipmAqsQ7kNvwE3y5Ow&_nc_gid=RGZW24yVvqRHKLuQvnRkKQ&edm=ALQROFkBAAAA&ccb=7-5&oh=00_AQJiPho8ymPf1bC2B5dAlKCbdXW8d4KK-9qqsVrcOytVGQ&oe=6A9D4CB4&_nc_sid=fc8dfb",
"width": 1080
},
"hd_profile_pic_versions": [
{
"height": 320,
"url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.82787-19/714930835_18032224961811255_6816644959905468206_n.jpg?stp=dst-jpg_s320x320_tt6&efg=eyJ2ZW5jb2RlX3RhZyI6InByb2ZpbGVfcGljLmRqYW5nby4xMDgwLmMyIn0&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_cat=1&_nc_oc=Q6cZ2gEBkafhMKkLe-YbSz6FVQiMLEw4juIKWvM8PNmZ2nTcMpNyjdl0kV_-fWna0tytdLA&_nc_ohc=hpoiFipmAqsQ7kNvwE3y5Ow&_nc_gid=RGZW24yVvqRHKLuQvnRkKQ&edm=ALQROFkBAAAA&ccb=7-5&oh=00_AQIn43F6SprbhopBuJxmswE2wd4C39mDZoL9Fbqs8Ih-cw&oe=6A9D4CB4&_nc_sid=fc8dfb",
"width": 320
},
{
"height": 640,
"url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.82787-19/714930835_18032224961811255_6816644959905468206_n.jpg?stp=dst-jpg_s640x640_tt6&efg=eyJ2ZW5jb2RlX3RhZyI6InByb2ZpbGVfcGljLmRqYW5nby4xMDgwLmMyIn0&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_cat=1&_nc_oc=Q6cZ2gEBkafhMKkLe-YbSz6FVQiMLEw4juIKWvM8PNmZ2nTcMpNyjdl0kV_-fWna0tytdLA&_nc_ohc=hpoiFipmAqsQ7kNvwE3y5Ow&_nc_gid=RGZW24yVvqRHKLuQvnRkKQ&edm=ALQROFkBAAAA&ccb=7-5&oh=00_AQJF1vYqhGCDkV-v6WbrTb_PoVC4GA3QCbEGJU0y0F96BA&oe=6A9D4CB4&_nc_sid=fc8dfb",
"width": 640
}
],
"id": "54301371254",
"is_active_on_text_post_app": false,
"is_embeds_disabled": false,
"is_favorite": false,
"is_private": false,
"is_ring_creator": false,
"is_unpublished": false,
"is_verified": true,
"latest_reel_media": 0,
"pk": "54301371254",
"pk_id": "54301371254",
"profile_pic_id": "3912796961274695044_54301371254",
"profile_pic_url": "https://instagram.fyto3-1.fna.fbcdn.net/v/t51.82787-19/714930835_18032224961811255_6816644959905468206_n.jpg?stp=dst-jpg_s150x150_tt6&efg=eyJ2ZW5jb2RlX3RhZyI6InByb2ZpbGVfcGljLmRqYW5nby4xMDgwLmMyIn0&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_cat=1&_nc_oc=Q6cZ2gEBkafhMKkLe-YbSz6FVQiMLEw4juIKWvM8PNmZ2nTcMpNyjdl0kV_-fWna0tytdLA&_nc_ohc=hpoiFipmAqsQ7kNvwE3y5Ow&_nc_gid=RGZW24yVvqRHKLuQvnRkKQ&edm=ALQROFkBAAAA&ccb=7-5&oh=00_AQJu85hrHiCGK3enXnfDQExy_ZbgzCV_AGYJ4yOy7mi_QQ&oe=6A9D4CB4&_nc_sid=fc8dfb",
"show_account_transparency_details": true,
"show_ring_award": false,
"strong_id__": "54301371254",
"text_post_app_is_private": true,
"third_party_downloads_enabled": 1,
"transparency_product_enabled": false,
"user_activation_info": {},
"username": "chuuo3o"
},
"owner_id": "54301371254",
"pinned": [],
"post_date": "2026-08-22 02:48:05",
"post_id": "3969041070976751617",
"post_shortcode": "DcU3dM-gVgB",
"post_url": "https://www.instagram.com/p/DcU3dM-gVgB/",
"shortcode": "DcU3dM-gVgB",
"subcategory": "post",
"tagged_users": [
{
"full_name": "HEEJIN",
"id": "57723176039",
"username": "0ct0ber19"
}
],
"type": "post",
"username": "chuuo3o",
"video_url": "https://instagram.fyto3-1.fna.fbcdn.net/o1/v/t2/f2/m86/AQN8nhc_L3hdDSQVVRF2OXm2gKwDpsdh_jw81ZnA9Uh4nqOwukKke8QXrGCe1jrMtOXzPT5Bkuipu8RShT651VV7q6EVbjvxC-fxgUE.mp4?_nc_cat=102&_nc_oc=AdqR865Bo8jW90mx5U9WR94Q0Eh1YI3ycT_ZhhW71XQNFHxUR3aBwhQEgcQ9Z2Hnrzs&_nc_sid=5e9851&_nc_ht=instagram.fyto3-1.fna.fbcdn.net&_nc_ohc=2F0lG8gquXwQ7kNvwGIIkUT&efg=eyJ2ZW5jb2RlX3RhZyI6Inhwdl9wcm9ncmVzc2l2ZS5JTlNUQUdSQU0uQ0xJUFMuQzMuNzIwLmRhc2hfYmFzZWxpbmVfMV92MSIsInhwdl9hc3NldF9pZCI6MTYxMDAzNjE0Mzk3NzkxMywiYXNzZXRfYWdlX2RheXMiOjExLCJ2aV91c2VjYXNlX2lkIjoxMDA5OSwiZHVyYXRpb25fcyI6MTUsInVybGdlbl9zb3VyY2UiOiJ3d3cifQ%3D%3D&ccb=17-1&vs=93986893873e15f7&_nc_vs=HBksFQIYUmlnX3hwdl9yZWVsc19wZXJtYW5lbnRfc3JfcHJvZC80ODQ5NDdDRjM0NzcyNkI2RkE4RDZDRkFGQjQzQTlCQl92aWRlb19kYXNoaW5pdC5tcDQVAALIARIAFQIYUWlnX3hwdl9wbGFjZW1lbnRfcGVybWFuZW50X3YyLzkwNDc3OEIwRTY2MDYxNEY5RjA4MUMxQzNDMDVBMkFDX2F1ZGlvX2Rhc2hpbml0Lm1wNBUCAsgBEgAoABgAGwKIB3VzZV9vaWwBMRJwcm9ncmVzc2l2ZV9yZWNpcGUBMRUAACby1v-GuJTcBRUCKAJDMywXQC6qfvnbItEYEmRhc2hfYmFzZWxpbmVfMV92MREAdf4HZeadAQA&_nc_gid=RGZW24yVvqRHKLuQvnRkKQ&_nc_zt=28&_nc_ss=7a22e&oh=00_AQIrUM_ZU8b3LTxlgefSs0x6vjvCX9BV4piQfWrs74dXIg&oe=6A99835F",
"width": 1080,
"width_original": 1080
}
]
]
+9 -103
View File
@@ -276,79 +276,6 @@ viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
only matters when testing by hand.
### Collab posts: the JSON and the media disagree about whose post it is
Verified directly against two saved raw API responses (`docs/example-api-
response-DcU3dM-gVgB.json`, a 2-way collab, and `docs/example-api-response-
DcLDme7zoyo.json`, a 5-way collab owned by an external account), 2026-09-01.
For an Instagram Collab, `username`/`fullname`/`owner_id` in the listing
response are always the **original poster's**, never the scraped account's —
even when the scraped account is one of the collaborators, not the owner. The
metadata `.json` sidecar (built from `include`, above) is filed under that
same original-poster identity, since its filename template uses
`{username}`. But the **media file** for that same post is written into
whichever profile's own crawl directory triggered the download, and its
filename's `{username}` slot took the *scraped* account's name, not the true
owner's. Net effect: one physical post produces a JSON named for the real
owner and a media file (or files, for a carousel) named for whoever we were
crawling — two different identities for one post, in the same directory.
Downstream (`cosmo_normalize_instagram.py` in `Cosmo-Live-Downloads`) had to
stop matching media to its JSON by username and match on shortcode/`{num}`
only, and separately built a cross-profile `collab_with` pass so every
participant's page shows the post, not just the one whose directory JD2/
gallery-dl happened to land the JSON in. See that repo's `NOTES.md`,
2026-09-01 entries, for the full fix.
**`coauthors` is a native, richer signal for this, and is now captured.**
The raw API response carries a `coauthors` array (`{"full_name", "id",
"username"}` per collaborator) that **excludes the post's own owner**
confirmed on both saved examples, including one where the owner
(`dailyfashion_news`) is an external account with no ARTMS members in her
own name, present only via `coauthors`. Added to the post-level metadata
JSON's `include` list on 2026-09-01: a direct field instead of inferring a
collab from filename/directory identity mismatches, and it never needs the
"is this the owner" branch `coauthors` already excludes for us.
**Per-carousel-item fields got their own sidecar, added the same day.**
`width`, `height`, `width_original`, `height_original` and `tagged_users`
live on the per-FILE kwdict, not the per-post one the metadata JSON above
reads — a carousel's items can each have different dimensions and tags,
which one post-level JSON can't represent. `gdl-sync.py`'s `media_pp` is a
second `metadata` postprocessor, `event: "file"` (gallery-dl's default when
omitted), so it runs once per downloaded file and writes `<filename>.json`
alongside it — e.g. `... - 01.jpg.json` next to `... - 01.jpg`, never
colliding with the post-level `....json`, which has no per-item number.
`owner` — a full user object (profile pic URLs, privacy flags) for whoever
posted that specific item — is deliberately left out, the same reasoning as
`audio_user` above. Verified against the same two saved examples: correct
per-item `width`/`height` and `tagged_users` came back on a live re-fetch of
an already-archived carousel, with zero media re-downloaded (the existing
skip-archive still applies; only the new sidecars are new files).
### The shared archive-db dedups media across profiles too, not just within one
The skip-archive DB (see "Incremental sync" above) keys purely on
`instagram_<media_id>`, with no per-profile scoping. When the *same* media_id
is reachable from more than one profile's listing — a Collab post, or a
repost — whichever profile's crawl reaches it **first** downloads the file;
every other profile that later lists the same media_id sees it as
already-in-the-archive-db and skips the download, even though that file has
never actually landed in *that profile's own* directory tree. The metadata
`.json`/`.txt` sidecars are written regardless (they aren't gated by the
download-archive), so the symptom is a JSON with zero matching media files
in its own directory — 510 such posts were found across the real archive on
2026-09-01, entirely from this mechanism, not from anything actually missing
from Instagram.
This isn't fixable on the gallery-dl side without per-profile archive DBs
(which would defeat the point of skip-archive — re-downloading anything a
sibling profile already fetched). The fix lives downstream instead: `cosmo_
normalize_instagram.py` builds one archive-wide `shortcode → path` index once
and falls back to it when a post's own directory has no matching media. Worth
knowing before assuming a JSON-with-no-media post reflects a real scrape gap.
## Cadence, and the budget that enforces it
**Monthly for everything, daily for stories only.** Stories expire in 24h and
@@ -632,40 +559,19 @@ Verify by asking gallery-dl's own interpreter, not the shell:
- **Highlights already have two naming styles on disk**, and every undated file
has a dated twin. The scanner dedupes by index so they render once; it is
wasted disk, not a display bug.
- **Some `video_versions` entries are VP9, and format selection is
codec-blind.** The extractor picks `max(video_versions, key=lambda x:
(x["width"], x["height"], x["type"]))` — resolution only, no codec check
(`instagram.py`). Instagram appears to have started serving VP9-encoded
highest-resolution variants for some posts around when the new gdl-based
workflow started (2026-08); 87 such files were found archive-wide on
2026-09-01. VP9-in-MP4 plays fine everywhere gallery-dl was tested from
except **Safari/WebKit**, which wires VP9 decode only into its WebM
demuxer, never its MP4/ISOBMFF path — confirmed via WebKit bug trackers,
not guessed. Fetching a lower-resolution non-VP9 variant instead was
considered and rejected (quality loss); the fix is downstream, a one-time
`-c:v copy -c:a libopus` remux to `.webm` (`cosmo_remux_instagram_vp9.py`
in `Cosmo-Live-Downloads`) that keeps VP9 losslessly and only re-encodes
audio (WebM disallows AAC). Confirmed live via gallery-dl/yt-dlp that
Instagram never offers a native WebM option to request instead — this has
to be done locally, there's no source-side fix.
## Scanner work done
## Scanner work (not done yet)
Shipped in `53b1f80` ("read gallery-dl sidecars for reel type and post dates").
`useArchiveScanner` tells the three `.json` shapes apart **structurally**, not
by filename, in `src/lib/gallery-dl-sidecar.ts`:
`useArchiveScanner` currently treats any `.json` in the tree as a possible
manifest. Adding gallery-dl sidecars needs it to distinguish three things:
1. Instagram export manifests (`posts_1.json`) — top-level `media` array.
2. Instaloader `.json.xz`GraphQL node under `node` / `__typename`.
3. gallery-dl `.json`flat, `post_shortcode` + `type`, none of the above.
1. Instagram export manifests (`posts_1.json`) — existing path.
2. Instaloader `.json.xz`existing path, GraphQL node shape.
3. gallery-dl `.json`new, flat shape, identified by having
`post_shortcode` + `type` at the top level.
`post.isReel` now comes from the sidecar's `type`, which is Instagram's own
classification, and beats every fallback in `post-tabs.ts`. Post dates are
ranked rather than last-write-wins (`src/lib/post-dates.ts`): sidecar beats
filename beats mtime.
Note those files live on **`main`** — they parse the archive at display time
and are viewer code, not fetching tooling.
Once (3) is read, `source`/`isStory` and the reel flag should come from `type`
rather than from the directory and the lone-video heuristic.
## Test cases
-184
View File
@@ -1,184 +0,0 @@
slug,username,archive_dir,shortcode,type,date,post_url,json_file
choerry,cher_ryppo,cher_ryppo,C5SBC--Syik,post,2024-04-03 01:50:25,https://www.instagram.com/p/C5SBC--Syik/,2024-04-02_cher_ryppo - C5SBC--Syik.json
choerry,cher_ryppo,cher_ryppo,C811ib_ys7D,post,2024-06-30 13:16:41,https://www.instagram.com/p/C811ib_ys7D/,2024-06-30_cher_ryppo - C811ib_ys7D.json
choerry,cher_ryppo,cher_ryppo,C9JTMgnSCHF,post,2024-07-08 02:41:25,https://www.instagram.com/p/C9JTMgnSCHF/,2024-07-07_cher_ryppo - C9JTMgnSCHF.json
choerry,cher_ryppo,cher_ryppo,C-K6kxZy6qq,post,2024-08-02 14:16:09,https://www.instagram.com/p/C-K6kxZy6qq/,2024-08-02_cher_ryppo - C-K6kxZy6qq.json
choerry,cher_ryppo,cher_ryppo,DBXI0zVs_Iv,post,2024-10-20 21:45:44,https://www.instagram.com/p/DBXI0zVs_Iv/,2024-10-20_cher_ryppo - DBXI0zVs_Iv.json
choerry,cher_ryppo,cher_ryppo,DRRDiezjdyV,post,2025-11-20 05:26:24,https://www.instagram.com/p/DRRDiezjdyV/,2025-11-20_cher_ryppo - DRRDiezjdyV.json
haseul,withaseul,withaseul,Cqps3ZWBlav,post,2023-04-05 10:44:56,https://www.instagram.com/p/Cqps3ZWBlav/,2023-04-05_withaseul - Cqps3ZWBlav.json
haseul,withaseul,withaseul,CqsjfUzSelq,post,2023-04-06 13:20:43,https://www.instagram.com/p/CqsjfUzSelq/,2023-04-06_withaseul - CqsjfUzSelq.json
haseul,withaseul,withaseul,CqxiTfjhNJ2,post,2023-04-08 11:46:34,https://www.instagram.com/p/CqxiTfjhNJ2/,2023-04-08_withaseul - CqxiTfjhNJ2.json
haseul,withaseul,withaseul,Cq4qymBhftL,post,2023-04-11 06:15:24,https://www.instagram.com/p/Cq4qymBhftL/,2023-04-11_withaseul - Cq4qymBhftL.json
haseul,withaseul,withaseul,CrnhJtrhoZi,post,2023-04-29 10:55:29,https://www.instagram.com/p/CrnhJtrhoZi/,2023-04-29_withaseul - CrnhJtrhoZi.json
haseul,withaseul,withaseul,Cr7fZZyB4Sp,post,2023-05-07 05:04:58,https://www.instagram.com/p/Cr7fZZyB4Sp/,2023-05-07_withaseul - Cr7fZZyB4Sp.json
haseul,withaseul,withaseul,Cr8QeaFhFsq,post,2023-05-07 12:13:49,https://www.instagram.com/p/Cr8QeaFhFsq/,2023-05-07_withaseul - Cr8QeaFhFsq.json
haseul,withaseul,withaseul,CsBIA4HBUOX,post,2023-05-09 09:36:05,https://www.instagram.com/p/CsBIA4HBUOX/,2023-05-09_withaseul - CsBIA4HBUOX.json
haseul,withaseul,withaseul,CsgbxFDhNmj,post,2023-05-21 13:25:08,https://www.instagram.com/p/CsgbxFDhNmj/,2023-05-21_withaseul - CsgbxFDhNmj.json
haseul,withaseul,withaseul,Csn9aO4B1r-,post,2023-05-24 11:33:48,https://www.instagram.com/p/Csn9aO4B1r-/,2023-05-24_withaseul - Csn9aO4B1r-.json
haseul,withaseul,withaseul,CsvY2jrBjMc,post,2023-05-27 08:48:17,https://www.instagram.com/p/CsvY2jrBjMc/,2023-05-27_withaseul - CsvY2jrBjMc.json
haseul,withaseul,withaseul,CtEHmRDPesc,post,2023-06-04 10:01:34,https://www.instagram.com/p/CtEHmRDPesc/,2023-06-04_withaseul - CtEHmRDPesc.json
haseul,withaseul,withaseul,CtRmUjsBUoc,post,2023-06-09 15:40:09,https://www.instagram.com/p/CtRmUjsBUoc/,2023-06-09_withaseul - CtRmUjsBUoc.json
haseul,withaseul,withaseul,Cuzar_3LqAA,post,2023-07-17 15:24:04,https://www.instagram.com/p/Cuzar_3LqAA/,2023-07-17_withaseul - Cuzar_3LqAA.json
haseul,withaseul,withaseul,CvHe1VMr7C-,post,2023-07-25 10:25:06,https://www.instagram.com/p/CvHe1VMr7C-/,2023-07-25_withaseul - CvHe1VMr7C-.json
haseul,withaseul,withaseul,CvhRFq7rnHc,post,2023-08-04 10:45:15,https://www.instagram.com/p/CvhRFq7rnHc/,2023-08-04_withaseul - CvhRFq7rnHc.json
haseul,withaseul,withaseul,CwIr3qEBPgF,post,2023-08-19 18:09:39,https://www.instagram.com/p/CwIr3qEBPgF/,2023-08-19_withaseul - CwIr3qEBPgF.json
haseul,withaseul,withaseul,CwzPlMThh5m,post,2023-09-05 06:49:48,https://www.instagram.com/p/CwzPlMThh5m/,2023-09-05_withaseul - CwzPlMThh5m.json
haseul,withaseul,withaseul,Cw9s3LNBhsq,post,2023-09-09 08:18:04,https://www.instagram.com/p/Cw9s3LNBhsq/,2023-09-09_withaseul - Cw9s3LNBhsq.json
haseul,withaseul,withaseul,CxIma4nBlaP,post,2023-09-13 13:53:26,https://www.instagram.com/p/CxIma4nBlaP/,2023-09-13_withaseul - CxIma4nBlaP.json
haseul,withaseul,withaseul,CxLcZ2jBME5,post,2023-09-14 16:23:38,https://www.instagram.com/p/CxLcZ2jBME5/,2023-09-14_withaseul - CxLcZ2jBME5.json
haseul,withaseul,withaseul,CxN3g95hVDH,post,2023-09-15 14:59:00,https://www.instagram.com/p/CxN3g95hVDH/,2023-09-15_withaseul - CxN3g95hVDH.json
haseul,withaseul,withaseul,CxTGBb_BAYw,post,2023-09-17 15:41:59,https://www.instagram.com/p/CxTGBb_BAYw/,2023-09-17_withaseul - CxTGBb_BAYw.json
haseul,withaseul,withaseul,CxnTyJIhUc0,post,2023-09-25 12:07:02,https://www.instagram.com/p/CxnTyJIhUc0/,2023-09-25_withaseul - CxnTyJIhUc0.json
haseul,withaseul,withaseul,CxsPZAur6Xv,post,2023-09-27 10:04:51,https://www.instagram.com/p/CxsPZAur6Xv/,2023-09-27_withaseul - CxsPZAur6Xv.json
haseul,withaseul,withaseul,Cx7mOFqhU7R,post,2023-10-03 09:12:57,https://www.instagram.com/p/Cx7mOFqhU7R/,2023-10-03_withaseul - Cx7mOFqhU7R.json
haseul,withaseul,withaseul,CyqV0lfruX-,post,2023-10-21 12:53:58,https://www.instagram.com/p/CyqV0lfruX-/,2023-10-21_withaseul - CyqV0lfruX-.json
haseul,withaseul,withaseul,Cy7RByYh2BH,post,2023-10-28 02:39:10,https://www.instagram.com/p/Cy7RByYh2BH/,2023-10-27_withaseul - Cy7RByYh2BH.json
haseul,withaseul,withaseul,Cz7nefxBJT0,post,2023-11-22 02:26:43,https://www.instagram.com/p/Cz7nefxBJT0/,2023-11-21_withaseul - Cz7nefxBJT0.json
haseul,withaseul,withaseul,C0YxsROhG4W,post,2023-12-03 10:13:57,https://www.instagram.com/p/C0YxsROhG4W/,2023-12-03_withaseul - C0YxsROhG4W.json
haseul,withaseul,withaseul,C0zOk2JBXTY,post,2023-12-13 16:46:36,https://www.instagram.com/p/C0zOk2JBXTY/,2023-12-13_withaseul - C0zOk2JBXTY.json
haseul,withaseul,withaseul,C1q_9TzBrdt,post,2024-01-04 08:36:20,https://www.instagram.com/p/C1q_9TzBrdt/,2024-01-04_withaseul - C1q_9TzBrdt.json
haseul,withaseul,withaseul,C1t-aeHBCef,post,2024-01-05 12:20:34,https://www.instagram.com/p/C1t-aeHBCef/,2024-01-05_withaseul - C1t-aeHBCef.json
haseul,withaseul,withaseul,C1yrgmMhsmz,post,2024-01-07 08:11:35,https://www.instagram.com/p/C1yrgmMhsmz/,2024-01-07_withaseul - C1yrgmMhsmz.json
haseul,withaseul,withaseul,C1yrxqIhGBZ,post,2024-01-07 08:13:54,https://www.instagram.com/p/C1yrxqIhGBZ/,2024-01-07_withaseul - C1yrxqIhGBZ.json
haseul,withaseul,withaseul,C10p70xh-6x,post,2024-01-08 02:36:18,https://www.instagram.com/p/C10p70xh-6x/,2024-01-07_withaseul - C10p70xh-6x.json
haseul,withaseul,withaseul,C2APujnBSF-,post,2024-01-12 14:38:11,https://www.instagram.com/p/C2APujnBSF-/,2024-01-12_withaseul - C2APujnBSF-.json
haseul,withaseul,withaseul,C3enxknhzeF,post,2024-02-18 06:16:55,https://www.instagram.com/p/C3enxknhzeF/,2024-02-18_withaseul - C3enxknhzeF.json
haseul,withaseul,withaseul,C4f8UQIBZAe,post,2024-03-14 15:07:03,https://www.instagram.com/p/C4f8UQIBZAe/,2024-03-14_withaseul - C4f8UQIBZAe.json
haseul,withaseul,withaseul,C4qIrJTh3po,post,2024-03-18 14:07:26,https://www.instagram.com/p/C4qIrJTh3po/,2024-03-18_withaseul - C4qIrJTh3po.json
haseul,withaseul,withaseul,C4rjDNhhJu8,post,2024-03-19 03:17:09,https://www.instagram.com/p/C4rjDNhhJu8/,2024-03-18_withaseul - C4rjDNhhJu8.json
haseul,withaseul,withaseul,C4xSiGGhP40,post,2024-03-21 08:48:16,https://www.instagram.com/p/C4xSiGGhP40/,2024-03-21_withaseul - C4xSiGGhP40.json
haseul,withaseul,withaseul,C5GBz4FB-kE,post,2024-03-29 10:06:12,https://www.instagram.com/p/C5GBz4FB-kE/,2024-03-29_withaseul - C5GBz4FB-kE.json
haseul,withaseul,withaseul,C5GJmlABYug,post,2024-03-29 11:14:17,https://www.instagram.com/p/C5GJmlABYug/,2024-03-29_withaseul - C5GJmlABYug.json
haseul,withaseul,withaseul,C5LgFFtB8ri,post,2024-03-31 13:06:54,https://www.instagram.com/p/C5LgFFtB8ri/,2024-03-31_withaseul - C5LgFFtB8ri.json
haseul,withaseul,withaseul,C5VZGPDB-Qo,post,2024-04-04 09:18:17,https://www.instagram.com/p/C5VZGPDB-Qo/,2024-04-04_withaseul - C5VZGPDB-Qo.json
haseul,withaseul,withaseul - reels,C5bWrT3BaGy,post,2024-04-06 16:52:50,https://www.instagram.com/p/C5bWrT3BaGy/,2024-04-06_withaseul - C5bWrT3BaGy.json
haseul,withaseul,withaseul,C6LAH8vBs_a,post,2024-04-25 04:59:04,https://www.instagram.com/p/C6LAH8vBs_a/,2024-04-25_withaseul - C6LAH8vBs_a.json
haseul,withaseul,withaseul,C6ypIzVBmiL,post,2024-05-10 14:27:49,https://www.instagram.com/p/C6ypIzVBmiL/,2024-05-10_withaseul - C6ypIzVBmiL.json
haseul,withaseul,withaseul,C68cTpzhUzQ,post,2024-05-14 09:48:07,https://www.instagram.com/p/C68cTpzhUzQ/,2024-05-14_withaseul - C68cTpzhUzQ.json
haseul,withaseul,withaseul,C68oiiahwZd,post,2024-05-14 11:35:00,https://www.instagram.com/p/C68oiiahwZd/,2024-05-14_withaseul - C68oiiahwZd.json
haseul,withaseul,withaseul,C7Oh6mYB7oF,post,2024-05-21 10:23:27,https://www.instagram.com/p/C7Oh6mYB7oF/,2024-05-21_withaseul - C7Oh6mYB7oF.json
haseul,withaseul,withaseul - reels,C7RnCv6BZT3,post,2024-05-22 15:06:48,https://www.instagram.com/p/C7RnCv6BZT3/,2024-05-22_withaseul - C7RnCv6BZT3.json
haseul,withaseul,withaseul,C7R7owoBd4n,post,2024-05-22 18:05:56,https://www.instagram.com/p/C7R7owoBd4n/,2024-05-22_withaseul - C7R7owoBd4n.json
haseul,withaseul,withaseul,C7mPzLRhCRz,post,2024-05-30 15:26:55,https://www.instagram.com/p/C7mPzLRhCRz/,2024-05-30_withaseul - C7mPzLRhCRz.json
haseul,withaseul,withaseul,C7tNIlPh7NK,post,2024-06-02 08:18:19,https://www.instagram.com/p/C7tNIlPh7NK/,2024-06-02_withaseul - C7tNIlPh7NK.json
haseul,withaseul,withaseul,C70yRTmBgng,post,2024-06-05 06:57:30,https://www.instagram.com/p/C70yRTmBgng/,2024-06-05_withaseul - C70yRTmBgng.json
haseul,withaseul,withaseul,C71aPMWBfqs,post,2024-06-05 12:46:44,https://www.instagram.com/p/C71aPMWBfqs/,2024-06-05_withaseul - C71aPMWBfqs.json
haseul,withaseul,withaseul,C8NBuqRhCJv,post,2024-06-14 16:54:21,https://www.instagram.com/p/C8NBuqRhCJv/,2024-06-14_withaseul - C8NBuqRhCJv.json
haseul,withaseul,withaseul,C8PPpUKhyLf,post,2024-06-15 13:34:26,https://www.instagram.com/p/C8PPpUKhyLf/,2024-06-15_withaseul - C8PPpUKhyLf.json
haseul,withaseul,withaseul,C8UjYG8B74s,post,2024-06-17 15:03:03,https://www.instagram.com/p/C8UjYG8B74s/,2024-06-17_withaseul - C8UjYG8B74s.json
haseul,withaseul,withaseul,C87PlKyBTU6,post,2024-07-02 15:40:27,https://www.instagram.com/p/C87PlKyBTU6/,2024-07-02_withaseul - C87PlKyBTU6.json
haseul,withaseul,withaseul,C9FqomJhky4,post,2024-07-06 16:49:16,https://www.instagram.com/p/C9FqomJhky4/,2024-07-06_withaseul - C9FqomJhky4.json
haseul,withaseul,withaseul,C9ICMZ5BUfr,post,2024-07-07 14:53:36,https://www.instagram.com/p/C9ICMZ5BUfr/,2024-07-07_withaseul - C9ICMZ5BUfr.json
haseul,withaseul,withaseul,C9V8bjJBPdk,post,2024-07-13 00:32:37,https://www.instagram.com/p/C9V8bjJBPdk/,2024-07-12_withaseul - C9V8bjJBPdk.json
haseul,withaseul,withaseul,C9xK8b1SZAY,post,2024-07-23 14:18:56,https://www.instagram.com/p/C9xK8b1SZAY/,2024-07-23_withaseul - C9xK8b1SZAY.json
haseul,withaseul,withaseul,C-DLjJ6BE0y,post,2024-07-30 14:10:33,https://www.instagram.com/p/C-DLjJ6BE0y/,2024-07-30_withaseul - C-DLjJ6BE0y.json
haseul,withaseul,withaseul,C-Ncp1MyhiD,post,2024-08-03 13:52:25,https://www.instagram.com/p/C-Ncp1MyhiD/,2024-08-03_withaseul - C-Ncp1MyhiD.json
haseul,withaseul,withaseul,C_A4QkApuLO,post,2024-08-23 13:14:54,https://www.instagram.com/p/C_A4QkApuLO/,2024-08-23_withaseul - C_A4QkApuLO.json
haseul,withaseul,withaseul,C_CcGblt8X9,post,2024-08-24 03:47:20,https://www.instagram.com/p/C_CcGblt8X9/,2024-08-23_withaseul - C_CcGblt8X9.json
haseul,withaseul,withaseul,C_ZCyXPylSu,post,2024-09-01 22:28:40,https://www.instagram.com/p/C_ZCyXPylSu/,2024-09-01_withaseul - C_ZCyXPylSu.json
haseul,withaseul,withaseul,C_gRp4gPuY3,post,2024-09-04 17:53:16,https://www.instagram.com/p/C_gRp4gPuY3/,2024-09-04_withaseul - C_gRp4gPuY3.json
haseul,withaseul,withaseul,C_7XyvJMmm7,post,2024-09-15 06:26:24,https://www.instagram.com/p/C_7XyvJMmm7/,2024-09-15_withaseul - C_7XyvJMmm7.json
haseul,withaseul,withaseul,DACAVOGyOEc,post,2024-09-17 20:16:04,https://www.instagram.com/p/DACAVOGyOEc/,2024-09-17_withaseul - DACAVOGyOEc.json
haseul,withaseul,withaseul,DAqwdZIBjY5,post,2024-10-03 16:06:14,https://www.instagram.com/p/DAqwdZIBjY5/,2024-10-03_withaseul - DAqwdZIBjY5.json
haseul,withaseul,withaseul,DAv8DxvyGbr,post,2024-10-05 16:23:48,https://www.instagram.com/p/DAv8DxvyGbr/,2024-10-05_withaseul - DAv8DxvyGbr.json
haseul,withaseul,withaseul,DBhdsL1NIvC,post,2024-10-24 22:00:28,https://www.instagram.com/p/DBhdsL1NIvC/,2024-10-24_withaseul - DBhdsL1NIvC.json
haseul,withaseul,withaseul,DBrP-pYSDEn,post,2024-10-28 17:13:03,https://www.instagram.com/p/DBrP-pYSDEn/,2024-10-28_withaseul - DBrP-pYSDEn.json
haseul,withaseul,withaseul,DBu5zCHBgrT,post,2024-10-30 03:16:12,https://www.instagram.com/p/DBu5zCHBgrT/,2024-10-29_withaseul - DBu5zCHBgrT.json
haseul,withaseul,withaseul,DByVUvOBSQy,post,2024-10-31 11:14:27,https://www.instagram.com/p/DByVUvOBSQy/,2024-10-31_withaseul - DByVUvOBSQy.json
haseul,withaseul,withaseul,DB5re5DhW8a,post,2024-11-03 07:42:45,https://www.instagram.com/p/DB5re5DhW8a/,2024-11-03_withaseul - DB5re5DhW8a.json
haseul,withaseul,withaseul,DCBsfwGhFYd,post,2024-11-06 10:25:32,https://www.instagram.com/p/DCBsfwGhFYd/,2024-11-06_withaseul - DCBsfwGhFYd.json
haseul,withaseul,withaseul,DCObB0SR64m,post,2024-11-11 09:03:02,https://www.instagram.com/p/DCObB0SR64m/,2024-11-11_withaseul - DCObB0SR64m.json
haseul,withaseul,withaseul,DCRhOPphLWU,post,2024-11-12 13:54:52,https://www.instagram.com/p/DCRhOPphLWU/,2024-11-12_withaseul - DCRhOPphLWU.json
haseul,withaseul,withaseul,DCbgyVghiT-,post,2024-11-16 11:03:28,https://www.instagram.com/p/DCbgyVghiT-/,2024-11-16_withaseul - DCbgyVghiT-.json
haseul,withaseul,withaseul,DCe4qpXhgBV,post,2024-11-17 18:29:51,https://www.instagram.com/p/DCe4qpXhgBV/,2024-11-17_withaseul - DCe4qpXhgBV.json
haseul,withaseul,withaseul,DC_4KddBmo_,post,2024-11-30 14:00:24,https://www.instagram.com/p/DC_4KddBmo_/,2024-11-30_withaseul - DC_4KddBmo_.json
haseul,withaseul,withaseul,DDRNi9Bhicn,post,2024-12-07 07:34:20,https://www.instagram.com/p/DDRNi9Bhicn/,2024-12-07_withaseul - DDRNi9Bhicn.json
haseul,withaseul,withaseul,DDW2JdFBUMs,post,2024-12-09 12:05:19,https://www.instagram.com/p/DDW2JdFBUMs/,2024-12-09_withaseul - DDW2JdFBUMs.json
haseul,withaseul,withaseul,DDjWETEBWw2,post,2024-12-14 08:35:07,https://www.instagram.com/p/DDjWETEBWw2/,2024-12-14_withaseul - DDjWETEBWw2.json
haseul,withaseul,withaseul,DDpkLTyhUdt,post,2024-12-16 18:33:51,https://www.instagram.com/p/DDpkLTyhUdt/,2024-12-16_withaseul - DDpkLTyhUdt.json
haseul,withaseul,withaseul,DE1NVFihF2g,post,2025-01-15 03:36:30,https://www.instagram.com/p/DE1NVFihF2g/,2025-01-14_withaseul - DE1NVFihF2g.json
haseul,withaseul,withaseul,DE5GO2-S5PB,post,2025-01-16 15:51:26,https://www.instagram.com/p/DE5GO2-S5PB/,2025-01-16_withaseul - DE5GO2-S5PB.json
haseul,withaseul,withaseul - reels,DE5H7LcB5aX,post,2025-01-16 16:07:31,https://www.instagram.com/p/DE5H7LcB5aX/,2025-01-16_withaseul - DE5H7LcB5aX.json
haseul,withaseul,withaseul,DFFtju9hTdv,post,2025-01-21 13:25:58,https://www.instagram.com/p/DFFtju9hTdv/,2025-01-21_withaseul - DFFtju9hTdv.json
haseul,withaseul,withaseul,DFsLxphhwqE,post,2025-02-05 12:01:09,https://www.instagram.com/p/DFsLxphhwqE/,2025-02-05_withaseul - DFsLxphhwqE.json
haseul,withaseul,withaseul,DHvlOgAxE_j,post,2025-03-28 12:44:03,https://www.instagram.com/p/DHvlOgAxE_j/,2025-03-28_withaseul - DHvlOgAxE_j.json
haseul,withaseul,withaseul,DH96B0QtxfT,post,2025-04-03 02:15:11,https://www.instagram.com/p/DH96B0QtxfT/,2025-04-02_withaseul - DH96B0QtxfT.json
haseul,withaseul,withaseul,DIL6wNXRGDC,post,2025-04-08 12:50:53,https://www.instagram.com/p/DIL6wNXRGDC/,2025-04-08_withaseul - DIL6wNXRGDC.json
haseul,withaseul,withaseul,DIP1mift6QV,post,2025-04-10 01:22:50,https://www.instagram.com/p/DIP1mift6QV/,2025-04-09_withaseul - DIP1mift6QV.json
haseul,withaseul,withaseul,DIfvDIKxwuT,post,2025-04-16 05:33:25,https://www.instagram.com/p/DIfvDIKxwuT/,2025-04-16_withaseul - DIfvDIKxwuT.json
haseul,withaseul,withaseul,DIm-MshvCJy,post,2025-04-19 01:00:03,https://www.instagram.com/p/DIm-MshvCJy/,2025-04-18_withaseul - DIm-MshvCJy.json
haseul,withaseul,withaseul,DIuQyfphWnR,post,2025-04-21 20:57:37,https://www.instagram.com/p/DIuQyfphWnR/,2025-04-21_withaseul - DIuQyfphWnR.json
haseul,withaseul,withaseul,DI3hOzrBLs-,post,2025-04-25 11:14:27,https://www.instagram.com/p/DI3hOzrBLs-/,2025-04-25_withaseul - DI3hOzrBLs-.json
haseul,withaseul,withaseul,DK0lklBhrH6,post,2025-06-13 00:57:27,https://www.instagram.com/p/DK0lklBhrH6/,2025-06-12_withaseul - DK0lklBhrH6.json
haseul,withaseul,withaseul,DK1EdUZBJtr,post,2025-06-13 05:27:20,https://www.instagram.com/p/DK1EdUZBJtr/,2025-06-13_withaseul - DK1EdUZBJtr.json
haseul,withaseul,withaseul,DLJiMzrhM83,post,2025-06-21 04:12:02,https://www.instagram.com/p/DLJiMzrhM83/,2025-06-21_withaseul - DLJiMzrhM83.json
haseul,withaseul,withaseul,DLL1iePBcwm,post,2025-06-22 01:39:30,https://www.instagram.com/p/DLL1iePBcwm/,2025-06-21_withaseul - DLL1iePBcwm.json
haseul,withaseul,withaseul,DLM-sUIhK5P,post,2025-06-22 12:18:44,https://www.instagram.com/p/DLM-sUIhK5P/,2025-06-22_withaseul - DLM-sUIhK5P.json
haseul,withaseul,withaseul,DLU66yCvWZh,post,2025-06-25 14:19:41,https://www.instagram.com/p/DLU66yCvWZh/,2025-06-25_withaseul - DLU66yCvWZh.json
haseul,withaseul,withaseul,DLxAZg5hpnW,post,2025-07-06 12:06:18,https://www.instagram.com/p/DLxAZg5hpnW/,2025-07-06_withaseul - DLxAZg5hpnW.json
haseul,withaseul,withaseul,DMFfDqABJgM,post,2025-07-14 10:59:00,https://www.instagram.com/p/DMFfDqABJgM/,2025-07-14_withaseul - DMFfDqABJgM.json
haseul,withaseul,withaseul,DM0NH9yhkzT,post,2025-08-01 14:26:37,https://www.instagram.com/p/DM0NH9yhkzT/,2025-08-01_withaseul - DM0NH9yhkzT.json
haseul,withaseul,withaseul,DM9_Ol7SzmK,post,2025-08-05 09:37:35,https://www.instagram.com/p/DM9_Ol7SzmK/,2025-08-05_withaseul - DM9_Ol7SzmK.json
haseul,withaseul,withaseul,DNnih4pBd_M,post,2025-08-21 12:54:55,https://www.instagram.com/p/DNnih4pBd_M/,2025-08-21_withaseul - DNnih4pBd_M.json
haseul,withaseul,withaseul,DOIxptrgXgv,post,2025-09-03 10:42:00,https://www.instagram.com/p/DOIxptrgXgv/,2025-09-03_withaseul - DOIxptrgXgv.json
haseul,withaseul,withaseul,DOa8a_ugc2Z,post,2025-09-10 12:02:26,https://www.instagram.com/p/DOa8a_ugc2Z/,2025-09-10_withaseul - DOa8a_ugc2Z.json
haseul,withaseul,withaseul,DPMLxsmgUJJ,post,2025-09-29 14:59:24,https://www.instagram.com/p/DPMLxsmgUJJ/,2025-09-29_withaseul - DPMLxsmgUJJ.json
haseul,withaseul,withaseul,DPRez1QgUxE,post,2025-10-01 16:21:55,https://www.instagram.com/p/DPRez1QgUxE/,2025-10-01_withaseul - DPRez1QgUxE.json
haseul,withaseul,withaseul,DQg9WOOgaKD,post,2025-11-01 13:08:45,https://www.instagram.com/p/DQg9WOOgaKD/,2025-11-01_withaseul - DQg9WOOgaKD.json
haseul,withaseul,withaseul,DQ_JrMdDciC,post,2025-11-13 06:33:42,https://www.instagram.com/p/DQ_JrMdDciC/,2025-11-13_withaseul - DQ_JrMdDciC.json
haseul,withaseul,withaseul,DROhLjLjVqY,post,2025-11-19 05:47:42,https://www.instagram.com/p/DROhLjLjVqY/,2025-11-19_withaseul - DROhLjLjVqY.json
haseul,withaseul,withaseul,DRwiTEDDRHR,post,2025-12-02 10:51:38,https://www.instagram.com/p/DRwiTEDDRHR/,2025-12-02_withaseul - DRwiTEDDRHR.json
haseul,withaseul,withaseul,DTw-4d9gRUx,post,2026-01-21 08:04:12,https://www.instagram.com/p/DTw-4d9gRUx/,2026-01-21_withaseul - DTw-4d9gRUx.json
haseul,withaseul,withaseul,DT0rHDCjPpE,post,2026-01-22 18:28:24,https://www.instagram.com/p/DT0rHDCjPpE/,2026-01-22_withaseul - DT0rHDCjPpE.json
haseul,withaseul,withaseul,DT6HqajDJfF,post,2026-01-24 21:14:05,https://www.instagram.com/p/DT6HqajDJfF/,2026-01-24_withaseul - DT6HqajDJfF.json
haseul,withaseul,withaseul,DU0oU25gWso,post,2026-02-16 14:19:54,https://www.instagram.com/p/DU0oU25gWso/,2026-02-16_withaseul - DU0oU25gWso.json
haseul,withaseul,withaseul,DWZPNAKAZcl,post,2026-03-27 16:19:14,https://www.instagram.com/p/DWZPNAKAZcl/,2026-03-27_withaseul - DWZPNAKAZcl.json
haseul,withaseul,withaseul,DW3yBm2gctK,post,2026-04-08 13:00:00,https://www.instagram.com/p/DW3yBm2gctK/,2026-04-08_withaseul - DW3yBm2gctK.json
haseul,withaseul,withaseul,DZj6uVWhV8M,post,2026-06-14 09:26:10,https://www.instagram.com/p/DZj6uVWhV8M/,2026-06-14_withaseul - DZj6uVWhV8M.json
haseul,withaseul,withaseul,DaZm_xLAZeL,post,2026-07-05 05:52:43,https://www.instagram.com/p/DaZm_xLAZeL/,2026-07-05_withaseul - DaZm_xLAZeL.json
haseul,withaseul,withaseul,Dbnq4k9gb5U,post,2026-08-04 13:27:27,https://www.instagram.com/p/Dbnq4k9gb5U/,2026-08-04_withaseul - Dbnq4k9gb5U.json
heejin,0ct0ber19,0ct0ber19,CrdsY5CrSsO,post,2023-04-25 15:21:16,https://www.instagram.com/p/CrdsY5CrSsO/,2023-04-25_0ct0ber19 - CrdsY5CrSsO.json
heejin,0ct0ber19,0ct0ber19,CtohvHxLnWO,post,2023-06-18 13:22:37,https://www.instagram.com/p/CtohvHxLnWO/,2023-06-18_0ct0ber19 - CtohvHxLnWO.json
heejin,0ct0ber19,0ct0ber19,CuEOMWppp5S,post,2023-06-29 07:30:35,https://www.instagram.com/p/CuEOMWppp5S/,2023-06-29_0ct0ber19 - CuEOMWppp5S.json
heejin,0ct0ber19,0ct0ber19,CwC0Y-prGoB,post,2023-08-17 11:28:40,https://www.instagram.com/p/CwC0Y-prGoB/,2023-08-17_0ct0ber19 - CwC0Y-prGoB.json
heejin,0ct0ber19,0ct0ber19,CxsKPWgpzU4,post,2023-09-27 09:19:51,https://www.instagram.com/p/CxsKPWgpzU4/,2023-09-27_0ct0ber19 - CxsKPWgpzU4.json
heejin,0ct0ber19,0ct0ber19,CzJf78xryub,post,2023-11-02 15:18:48,https://www.instagram.com/p/CzJf78xryub/,2023-11-02_0ct0ber19 - CzJf78xryub.json
heejin,0ct0ber19,0ct0ber19,CzOVVzFLTnD,post,2023-11-04 12:22:25,https://www.instagram.com/p/CzOVVzFLTnD/,2023-11-04_0ct0ber19 - CzOVVzFLTnD.json
heejin,0ct0ber19,0ct0ber19,C14VKVmpdtC,post,2024-01-09 12:51:44,https://www.instagram.com/p/C14VKVmpdtC/,2024-01-09_0ct0ber19 - C14VKVmpdtC.json
heejin,0ct0ber19,0ct0ber19,C2FnIelpue9,post,2024-01-14 16:38:54,https://www.instagram.com/p/C2FnIelpue9/,2024-01-14_0ct0ber19 - C2FnIelpue9.json
heejin,0ct0ber19,0ct0ber19,C5SDY_UJIn6,post,2024-04-03 02:10:53,https://www.instagram.com/p/C5SDY_UJIn6/,2024-04-02_0ct0ber19 - C5SDY_UJIn6.json
heejin,0ct0ber19,0ct0ber19,C5WB6zwp_rX,post,2024-04-04 15:15:00,https://www.instagram.com/p/C5WB6zwp_rX/,2024-04-04_0ct0ber19 - C5WB6zwp_rX.json
heejin,0ct0ber19,0ct0ber19,C5dJ0xXpJpX,post,2024-04-07 09:38:46,https://www.instagram.com/p/C5dJ0xXpJpX/,2024-04-07_0ct0ber19 - C5dJ0xXpJpX.json
heejin,0ct0ber19,0ct0ber19,C5oJdT-pQBI,post,2024-04-11 16:07:12,https://www.instagram.com/p/C5oJdT-pQBI/,2024-04-11_0ct0ber19 - C5oJdT-pQBI.json
heejin,0ct0ber19,0ct0ber19,C53YPQzp7Wj,post,2024-04-17 14:04:58,https://www.instagram.com/p/C53YPQzp7Wj/,2024-04-17_0ct0ber19 - C53YPQzp7Wj.json
heejin,0ct0ber19,0ct0ber19,C5--hdUJKZH,post,2024-04-20 12:54:11,https://www.instagram.com/p/C5--hdUJKZH/,2024-04-20_0ct0ber19 - C5--hdUJKZH.json
heejin,0ct0ber19,0ct0ber19,C71iAmbpekA,post,2024-06-05 13:54:39,https://www.instagram.com/p/C71iAmbpekA/,2024-06-05_0ct0ber19 - C71iAmbpekA.json
heejin,0ct0ber19,0ct0ber19,C8M8TxnSOKI,post,2024-06-14 16:07:00,https://www.instagram.com/p/C8M8TxnSOKI/,2024-06-14_0ct0ber19 - C8M8TxnSOKI.json
heejin,0ct0ber19,0ct0ber19,C8hxpuYyWGv,post,2024-06-22 18:17:55,https://www.instagram.com/p/C8hxpuYyWGv/,2024-06-22_0ct0ber19 - C8hxpuYyWGv.json
heejin,0ct0ber19,0ct0ber19,C-AbtWSJbHB,post,2024-07-29 12:34:02,https://www.instagram.com/p/C-AbtWSJbHB/,2024-07-29_0ct0ber19 - C-AbtWSJbHB.json
heejin,0ct0ber19,0ct0ber19,C_WxFtuvqXK,post,2024-09-01 01:15:32,https://www.instagram.com/p/C_WxFtuvqXK/,2024-08-31_0ct0ber19 - C_WxFtuvqXK.json
heejin,0ct0ber19,0ct0ber19,DcgTbz-iTAH,post,2026-08-26 13:19:15,https://www.instagram.com/p/DcgTbz-iTAH/,2026-08-26_0ct0ber19 - DcgTbz-iTAH.json
jinsoul,zindoriyam,zindoriyam,CndmboMBHeB,post,2023-01-16 04:23:38,https://www.instagram.com/p/CndmboMBHeB/,2023-01-15_zindoriyam - CndmboMBHeB.json
jinsoul,zindoriyam,zindoriyam,Ctb5AnZxCwh,post,2023-06-13 15:35:51,https://www.instagram.com/p/Ctb5AnZxCwh/,2023-06-13_zindoriyam - Ctb5AnZxCwh.json
jinsoul,zindoriyam,zindoriyam,Cu6ZRz8hj_T,post,2023-07-20 08:26:26,https://www.instagram.com/p/Cu6ZRz8hj_T/,2023-07-20_zindoriyam - Cu6ZRz8hj_T.json
jinsoul,zindoriyam,zindoriyam,Cwrydt6OXmo,post,2023-09-02 09:20:42,https://www.instagram.com/p/Cwrydt6OXmo/,2023-09-02_zindoriyam - Cwrydt6OXmo.json
jinsoul,zindoriyam,zindoriyam,Cxsj3_IhwNv,post,2023-09-27 13:03:51,https://www.instagram.com/p/Cxsj3_IhwNv/,2023-09-27_zindoriyam - Cxsj3_IhwNv.json
jinsoul,zindoriyam,zindoriyam,Cza0C4gBkb5,post,2023-11-09 08:41:36,https://www.instagram.com/p/Cza0C4gBkb5/,2023-11-09_zindoriyam - Cza0C4gBkb5.json
jinsoul,zindoriyam,zindoriyam,C5LqGJUBTIK,post,2024-03-31 14:34:25,https://www.instagram.com/p/C5LqGJUBTIK/,2024-03-31_zindoriyam - C5LqGJUBTIK.json
jinsoul,zindoriyam,zindoriyam,C5QZg9BBc12,post,2024-04-02 10:45:44,https://www.instagram.com/p/C5QZg9BBc12/,2024-04-02_zindoriyam - C5QZg9BBc12.json
jinsoul,zindoriyam,zindoriyam,C8NAliShdHm,post,2024-06-14 16:44:22,https://www.instagram.com/p/C8NAliShdHm/,2024-06-14_zindoriyam - C8NAliShdHm.json
jinsoul,zindoriyam,zindoriyam,C9-Ot__hyGY,post,2024-07-28 16:02:02,https://www.instagram.com/p/C9-Ot__hyGY/,2024-07-28_zindoriyam - C9-Ot__hyGY.json
jinsoul,zindoriyam,zindoriyam,DANQU24um-N,post,2024-09-22 05:07:29,https://www.instagram.com/p/DANQU24um-N/,2024-09-22_zindoriyam - DANQU24um-N.json
jinsoul,zindoriyam,zindoriyam,DHz88e2sS_J,post,2025-03-30 05:28:16,https://www.instagram.com/p/DHz88e2sS_J/,2025-03-30_zindoriyam - DHz88e2sS_J.json
jinsoul,zindoriyam,zindoriyam,DJXAVrANWfp,post,2025-05-07 16:42:44,https://www.instagram.com/p/DJXAVrANWfp/,2025-05-07_zindoriyam - DJXAVrANWfp.json
jinsoul,zindoriyam,zindoriyam,DLhdKkZJ2_5,post,2025-06-30 11:09:49,https://www.instagram.com/p/DLhdKkZJ2_5/,2025-06-30_zindoriyam - DLhdKkZJ2_5.json
jinsoul,zindoriyam,zindoriyam,DMiZJbaJZXv,post,2025-07-25 16:25:21,https://www.instagram.com/p/DMiZJbaJZXv/,2025-07-25_zindoriyam - DMiZJbaJZXv.json
jinsoul,zindoriyam,zindoriyam,DNJCd9jpT-J,post,2025-08-09 16:37:33,https://www.instagram.com/p/DNJCd9jpT-J/,2025-08-09_zindoriyam - DNJCd9jpT-J.json
jinsoul,zindoriyam,zindoriyam,DRl8jAxFnwE,post,2025-11-28 08:09:22,https://www.instagram.com/p/DRl8jAxFnwE/,2025-11-28_zindoriyam - DRl8jAxFnwE.json
jinsoul,zindoriyam,zindoriyam,DTQR5cDk6kY,post,2026-01-08 15:15:25,https://www.instagram.com/p/DTQR5cDk6kY/,2026-01-08_zindoriyam - DTQR5cDk6kY.json
jinsoul,zindoriyam,zindoriyam,DVv1oRSEyIy,post,2026-03-11 14:26:54,https://www.instagram.com/p/DVv1oRSEyIy/,2026-03-11_zindoriyam - DVv1oRSEyIy.json
jinsoul,zindoriyam,zindoriyam,DW_6oxSkz8u,post,2026-04-11 16:49:54,https://www.instagram.com/p/DW_6oxSkz8u/,2026-04-11_zindoriyam - DW_6oxSkz8u.json
jinsoul,zindoriyam,zindoriyam,DcRLAVJEwlG,post,2026-08-20 16:16:59,https://www.instagram.com/p/DcRLAVJEwlG/,2026-08-20_zindoriyam - DcRLAVJEwlG.json
kimlip,kimxxlip,kimxxlip,CndmBanNyrR,post,2023-01-16 04:20:04,https://www.instagram.com/p/CndmBanNyrR/,2023-01-15_kimxxlip - CndmBanNyrR.json
kimlip,kimxxlip,kimxxlip,CwXG1L3vQFH,post,2023-08-25 08:34:37,https://www.instagram.com/p/CwXG1L3vQFH/,2023-08-25_kimxxlip - CwXG1L3vQFH.json
kimlip,kimxxlip,kimxxlip,CxsL_FQvHna,post,2023-09-27 09:35:06,https://www.instagram.com/p/CxsL_FQvHna/,2023-09-27_kimxxlip - CxsL_FQvHna.json
1 slug username archive_dir shortcode type date post_url json_file
2 choerry cher_ryppo cher_ryppo C5SBC--Syik post 2024-04-03 01:50:25 https://www.instagram.com/p/C5SBC--Syik/ 2024-04-02_cher_ryppo - C5SBC--Syik.json
3 choerry cher_ryppo cher_ryppo C811ib_ys7D post 2024-06-30 13:16:41 https://www.instagram.com/p/C811ib_ys7D/ 2024-06-30_cher_ryppo - C811ib_ys7D.json
4 choerry cher_ryppo cher_ryppo C9JTMgnSCHF post 2024-07-08 02:41:25 https://www.instagram.com/p/C9JTMgnSCHF/ 2024-07-07_cher_ryppo - C9JTMgnSCHF.json
5 choerry cher_ryppo cher_ryppo C-K6kxZy6qq post 2024-08-02 14:16:09 https://www.instagram.com/p/C-K6kxZy6qq/ 2024-08-02_cher_ryppo - C-K6kxZy6qq.json
6 choerry cher_ryppo cher_ryppo DBXI0zVs_Iv post 2024-10-20 21:45:44 https://www.instagram.com/p/DBXI0zVs_Iv/ 2024-10-20_cher_ryppo - DBXI0zVs_Iv.json
7 choerry cher_ryppo cher_ryppo DRRDiezjdyV post 2025-11-20 05:26:24 https://www.instagram.com/p/DRRDiezjdyV/ 2025-11-20_cher_ryppo - DRRDiezjdyV.json
8 haseul withaseul withaseul Cqps3ZWBlav post 2023-04-05 10:44:56 https://www.instagram.com/p/Cqps3ZWBlav/ 2023-04-05_withaseul - Cqps3ZWBlav.json
9 haseul withaseul withaseul CqsjfUzSelq post 2023-04-06 13:20:43 https://www.instagram.com/p/CqsjfUzSelq/ 2023-04-06_withaseul - CqsjfUzSelq.json
10 haseul withaseul withaseul CqxiTfjhNJ2 post 2023-04-08 11:46:34 https://www.instagram.com/p/CqxiTfjhNJ2/ 2023-04-08_withaseul - CqxiTfjhNJ2.json
11 haseul withaseul withaseul Cq4qymBhftL post 2023-04-11 06:15:24 https://www.instagram.com/p/Cq4qymBhftL/ 2023-04-11_withaseul - Cq4qymBhftL.json
12 haseul withaseul withaseul CrnhJtrhoZi post 2023-04-29 10:55:29 https://www.instagram.com/p/CrnhJtrhoZi/ 2023-04-29_withaseul - CrnhJtrhoZi.json
13 haseul withaseul withaseul Cr7fZZyB4Sp post 2023-05-07 05:04:58 https://www.instagram.com/p/Cr7fZZyB4Sp/ 2023-05-07_withaseul - Cr7fZZyB4Sp.json
14 haseul withaseul withaseul Cr8QeaFhFsq post 2023-05-07 12:13:49 https://www.instagram.com/p/Cr8QeaFhFsq/ 2023-05-07_withaseul - Cr8QeaFhFsq.json
15 haseul withaseul withaseul CsBIA4HBUOX post 2023-05-09 09:36:05 https://www.instagram.com/p/CsBIA4HBUOX/ 2023-05-09_withaseul - CsBIA4HBUOX.json
16 haseul withaseul withaseul CsgbxFDhNmj post 2023-05-21 13:25:08 https://www.instagram.com/p/CsgbxFDhNmj/ 2023-05-21_withaseul - CsgbxFDhNmj.json
17 haseul withaseul withaseul Csn9aO4B1r- post 2023-05-24 11:33:48 https://www.instagram.com/p/Csn9aO4B1r-/ 2023-05-24_withaseul - Csn9aO4B1r-.json
18 haseul withaseul withaseul CsvY2jrBjMc post 2023-05-27 08:48:17 https://www.instagram.com/p/CsvY2jrBjMc/ 2023-05-27_withaseul - CsvY2jrBjMc.json
19 haseul withaseul withaseul CtEHmRDPesc post 2023-06-04 10:01:34 https://www.instagram.com/p/CtEHmRDPesc/ 2023-06-04_withaseul - CtEHmRDPesc.json
20 haseul withaseul withaseul CtRmUjsBUoc post 2023-06-09 15:40:09 https://www.instagram.com/p/CtRmUjsBUoc/ 2023-06-09_withaseul - CtRmUjsBUoc.json
21 haseul withaseul withaseul Cuzar_3LqAA post 2023-07-17 15:24:04 https://www.instagram.com/p/Cuzar_3LqAA/ 2023-07-17_withaseul - Cuzar_3LqAA.json
22 haseul withaseul withaseul CvHe1VMr7C- post 2023-07-25 10:25:06 https://www.instagram.com/p/CvHe1VMr7C-/ 2023-07-25_withaseul - CvHe1VMr7C-.json
23 haseul withaseul withaseul CvhRFq7rnHc post 2023-08-04 10:45:15 https://www.instagram.com/p/CvhRFq7rnHc/ 2023-08-04_withaseul - CvhRFq7rnHc.json
24 haseul withaseul withaseul CwIr3qEBPgF post 2023-08-19 18:09:39 https://www.instagram.com/p/CwIr3qEBPgF/ 2023-08-19_withaseul - CwIr3qEBPgF.json
25 haseul withaseul withaseul CwzPlMThh5m post 2023-09-05 06:49:48 https://www.instagram.com/p/CwzPlMThh5m/ 2023-09-05_withaseul - CwzPlMThh5m.json
26 haseul withaseul withaseul Cw9s3LNBhsq post 2023-09-09 08:18:04 https://www.instagram.com/p/Cw9s3LNBhsq/ 2023-09-09_withaseul - Cw9s3LNBhsq.json
27 haseul withaseul withaseul CxIma4nBlaP post 2023-09-13 13:53:26 https://www.instagram.com/p/CxIma4nBlaP/ 2023-09-13_withaseul - CxIma4nBlaP.json
28 haseul withaseul withaseul CxLcZ2jBME5 post 2023-09-14 16:23:38 https://www.instagram.com/p/CxLcZ2jBME5/ 2023-09-14_withaseul - CxLcZ2jBME5.json
29 haseul withaseul withaseul CxN3g95hVDH post 2023-09-15 14:59:00 https://www.instagram.com/p/CxN3g95hVDH/ 2023-09-15_withaseul - CxN3g95hVDH.json
30 haseul withaseul withaseul CxTGBb_BAYw post 2023-09-17 15:41:59 https://www.instagram.com/p/CxTGBb_BAYw/ 2023-09-17_withaseul - CxTGBb_BAYw.json
31 haseul withaseul withaseul CxnTyJIhUc0 post 2023-09-25 12:07:02 https://www.instagram.com/p/CxnTyJIhUc0/ 2023-09-25_withaseul - CxnTyJIhUc0.json
32 haseul withaseul withaseul CxsPZAur6Xv post 2023-09-27 10:04:51 https://www.instagram.com/p/CxsPZAur6Xv/ 2023-09-27_withaseul - CxsPZAur6Xv.json
33 haseul withaseul withaseul Cx7mOFqhU7R post 2023-10-03 09:12:57 https://www.instagram.com/p/Cx7mOFqhU7R/ 2023-10-03_withaseul - Cx7mOFqhU7R.json
34 haseul withaseul withaseul CyqV0lfruX- post 2023-10-21 12:53:58 https://www.instagram.com/p/CyqV0lfruX-/ 2023-10-21_withaseul - CyqV0lfruX-.json
35 haseul withaseul withaseul Cy7RByYh2BH post 2023-10-28 02:39:10 https://www.instagram.com/p/Cy7RByYh2BH/ 2023-10-27_withaseul - Cy7RByYh2BH.json
36 haseul withaseul withaseul Cz7nefxBJT0 post 2023-11-22 02:26:43 https://www.instagram.com/p/Cz7nefxBJT0/ 2023-11-21_withaseul - Cz7nefxBJT0.json
37 haseul withaseul withaseul C0YxsROhG4W post 2023-12-03 10:13:57 https://www.instagram.com/p/C0YxsROhG4W/ 2023-12-03_withaseul - C0YxsROhG4W.json
38 haseul withaseul withaseul C0zOk2JBXTY post 2023-12-13 16:46:36 https://www.instagram.com/p/C0zOk2JBXTY/ 2023-12-13_withaseul - C0zOk2JBXTY.json
39 haseul withaseul withaseul C1q_9TzBrdt post 2024-01-04 08:36:20 https://www.instagram.com/p/C1q_9TzBrdt/ 2024-01-04_withaseul - C1q_9TzBrdt.json
40 haseul withaseul withaseul C1t-aeHBCef post 2024-01-05 12:20:34 https://www.instagram.com/p/C1t-aeHBCef/ 2024-01-05_withaseul - C1t-aeHBCef.json
41 haseul withaseul withaseul C1yrgmMhsmz post 2024-01-07 08:11:35 https://www.instagram.com/p/C1yrgmMhsmz/ 2024-01-07_withaseul - C1yrgmMhsmz.json
42 haseul withaseul withaseul C1yrxqIhGBZ post 2024-01-07 08:13:54 https://www.instagram.com/p/C1yrxqIhGBZ/ 2024-01-07_withaseul - C1yrxqIhGBZ.json
43 haseul withaseul withaseul C10p70xh-6x post 2024-01-08 02:36:18 https://www.instagram.com/p/C10p70xh-6x/ 2024-01-07_withaseul - C10p70xh-6x.json
44 haseul withaseul withaseul C2APujnBSF- post 2024-01-12 14:38:11 https://www.instagram.com/p/C2APujnBSF-/ 2024-01-12_withaseul - C2APujnBSF-.json
45 haseul withaseul withaseul C3enxknhzeF post 2024-02-18 06:16:55 https://www.instagram.com/p/C3enxknhzeF/ 2024-02-18_withaseul - C3enxknhzeF.json
46 haseul withaseul withaseul C4f8UQIBZAe post 2024-03-14 15:07:03 https://www.instagram.com/p/C4f8UQIBZAe/ 2024-03-14_withaseul - C4f8UQIBZAe.json
47 haseul withaseul withaseul C4qIrJTh3po post 2024-03-18 14:07:26 https://www.instagram.com/p/C4qIrJTh3po/ 2024-03-18_withaseul - C4qIrJTh3po.json
48 haseul withaseul withaseul C4rjDNhhJu8 post 2024-03-19 03:17:09 https://www.instagram.com/p/C4rjDNhhJu8/ 2024-03-18_withaseul - C4rjDNhhJu8.json
49 haseul withaseul withaseul C4xSiGGhP40 post 2024-03-21 08:48:16 https://www.instagram.com/p/C4xSiGGhP40/ 2024-03-21_withaseul - C4xSiGGhP40.json
50 haseul withaseul withaseul C5GBz4FB-kE post 2024-03-29 10:06:12 https://www.instagram.com/p/C5GBz4FB-kE/ 2024-03-29_withaseul - C5GBz4FB-kE.json
51 haseul withaseul withaseul C5GJmlABYug post 2024-03-29 11:14:17 https://www.instagram.com/p/C5GJmlABYug/ 2024-03-29_withaseul - C5GJmlABYug.json
52 haseul withaseul withaseul C5LgFFtB8ri post 2024-03-31 13:06:54 https://www.instagram.com/p/C5LgFFtB8ri/ 2024-03-31_withaseul - C5LgFFtB8ri.json
53 haseul withaseul withaseul C5VZGPDB-Qo post 2024-04-04 09:18:17 https://www.instagram.com/p/C5VZGPDB-Qo/ 2024-04-04_withaseul - C5VZGPDB-Qo.json
54 haseul withaseul withaseul - reels C5bWrT3BaGy post 2024-04-06 16:52:50 https://www.instagram.com/p/C5bWrT3BaGy/ 2024-04-06_withaseul - C5bWrT3BaGy.json
55 haseul withaseul withaseul C6LAH8vBs_a post 2024-04-25 04:59:04 https://www.instagram.com/p/C6LAH8vBs_a/ 2024-04-25_withaseul - C6LAH8vBs_a.json
56 haseul withaseul withaseul C6ypIzVBmiL post 2024-05-10 14:27:49 https://www.instagram.com/p/C6ypIzVBmiL/ 2024-05-10_withaseul - C6ypIzVBmiL.json
57 haseul withaseul withaseul C68cTpzhUzQ post 2024-05-14 09:48:07 https://www.instagram.com/p/C68cTpzhUzQ/ 2024-05-14_withaseul - C68cTpzhUzQ.json
58 haseul withaseul withaseul C68oiiahwZd post 2024-05-14 11:35:00 https://www.instagram.com/p/C68oiiahwZd/ 2024-05-14_withaseul - C68oiiahwZd.json
59 haseul withaseul withaseul C7Oh6mYB7oF post 2024-05-21 10:23:27 https://www.instagram.com/p/C7Oh6mYB7oF/ 2024-05-21_withaseul - C7Oh6mYB7oF.json
60 haseul withaseul withaseul - reels C7RnCv6BZT3 post 2024-05-22 15:06:48 https://www.instagram.com/p/C7RnCv6BZT3/ 2024-05-22_withaseul - C7RnCv6BZT3.json
61 haseul withaseul withaseul C7R7owoBd4n post 2024-05-22 18:05:56 https://www.instagram.com/p/C7R7owoBd4n/ 2024-05-22_withaseul - C7R7owoBd4n.json
62 haseul withaseul withaseul C7mPzLRhCRz post 2024-05-30 15:26:55 https://www.instagram.com/p/C7mPzLRhCRz/ 2024-05-30_withaseul - C7mPzLRhCRz.json
63 haseul withaseul withaseul C7tNIlPh7NK post 2024-06-02 08:18:19 https://www.instagram.com/p/C7tNIlPh7NK/ 2024-06-02_withaseul - C7tNIlPh7NK.json
64 haseul withaseul withaseul C70yRTmBgng post 2024-06-05 06:57:30 https://www.instagram.com/p/C70yRTmBgng/ 2024-06-05_withaseul - C70yRTmBgng.json
65 haseul withaseul withaseul C71aPMWBfqs post 2024-06-05 12:46:44 https://www.instagram.com/p/C71aPMWBfqs/ 2024-06-05_withaseul - C71aPMWBfqs.json
66 haseul withaseul withaseul C8NBuqRhCJv post 2024-06-14 16:54:21 https://www.instagram.com/p/C8NBuqRhCJv/ 2024-06-14_withaseul - C8NBuqRhCJv.json
67 haseul withaseul withaseul C8PPpUKhyLf post 2024-06-15 13:34:26 https://www.instagram.com/p/C8PPpUKhyLf/ 2024-06-15_withaseul - C8PPpUKhyLf.json
68 haseul withaseul withaseul C8UjYG8B74s post 2024-06-17 15:03:03 https://www.instagram.com/p/C8UjYG8B74s/ 2024-06-17_withaseul - C8UjYG8B74s.json
69 haseul withaseul withaseul C87PlKyBTU6 post 2024-07-02 15:40:27 https://www.instagram.com/p/C87PlKyBTU6/ 2024-07-02_withaseul - C87PlKyBTU6.json
70 haseul withaseul withaseul C9FqomJhky4 post 2024-07-06 16:49:16 https://www.instagram.com/p/C9FqomJhky4/ 2024-07-06_withaseul - C9FqomJhky4.json
71 haseul withaseul withaseul C9ICMZ5BUfr post 2024-07-07 14:53:36 https://www.instagram.com/p/C9ICMZ5BUfr/ 2024-07-07_withaseul - C9ICMZ5BUfr.json
72 haseul withaseul withaseul C9V8bjJBPdk post 2024-07-13 00:32:37 https://www.instagram.com/p/C9V8bjJBPdk/ 2024-07-12_withaseul - C9V8bjJBPdk.json
73 haseul withaseul withaseul C9xK8b1SZAY post 2024-07-23 14:18:56 https://www.instagram.com/p/C9xK8b1SZAY/ 2024-07-23_withaseul - C9xK8b1SZAY.json
74 haseul withaseul withaseul C-DLjJ6BE0y post 2024-07-30 14:10:33 https://www.instagram.com/p/C-DLjJ6BE0y/ 2024-07-30_withaseul - C-DLjJ6BE0y.json
75 haseul withaseul withaseul C-Ncp1MyhiD post 2024-08-03 13:52:25 https://www.instagram.com/p/C-Ncp1MyhiD/ 2024-08-03_withaseul - C-Ncp1MyhiD.json
76 haseul withaseul withaseul C_A4QkApuLO post 2024-08-23 13:14:54 https://www.instagram.com/p/C_A4QkApuLO/ 2024-08-23_withaseul - C_A4QkApuLO.json
77 haseul withaseul withaseul C_CcGblt8X9 post 2024-08-24 03:47:20 https://www.instagram.com/p/C_CcGblt8X9/ 2024-08-23_withaseul - C_CcGblt8X9.json
78 haseul withaseul withaseul C_ZCyXPylSu post 2024-09-01 22:28:40 https://www.instagram.com/p/C_ZCyXPylSu/ 2024-09-01_withaseul - C_ZCyXPylSu.json
79 haseul withaseul withaseul C_gRp4gPuY3 post 2024-09-04 17:53:16 https://www.instagram.com/p/C_gRp4gPuY3/ 2024-09-04_withaseul - C_gRp4gPuY3.json
80 haseul withaseul withaseul C_7XyvJMmm7 post 2024-09-15 06:26:24 https://www.instagram.com/p/C_7XyvJMmm7/ 2024-09-15_withaseul - C_7XyvJMmm7.json
81 haseul withaseul withaseul DACAVOGyOEc post 2024-09-17 20:16:04 https://www.instagram.com/p/DACAVOGyOEc/ 2024-09-17_withaseul - DACAVOGyOEc.json
82 haseul withaseul withaseul DAqwdZIBjY5 post 2024-10-03 16:06:14 https://www.instagram.com/p/DAqwdZIBjY5/ 2024-10-03_withaseul - DAqwdZIBjY5.json
83 haseul withaseul withaseul DAv8DxvyGbr post 2024-10-05 16:23:48 https://www.instagram.com/p/DAv8DxvyGbr/ 2024-10-05_withaseul - DAv8DxvyGbr.json
84 haseul withaseul withaseul DBhdsL1NIvC post 2024-10-24 22:00:28 https://www.instagram.com/p/DBhdsL1NIvC/ 2024-10-24_withaseul - DBhdsL1NIvC.json
85 haseul withaseul withaseul DBrP-pYSDEn post 2024-10-28 17:13:03 https://www.instagram.com/p/DBrP-pYSDEn/ 2024-10-28_withaseul - DBrP-pYSDEn.json
86 haseul withaseul withaseul DBu5zCHBgrT post 2024-10-30 03:16:12 https://www.instagram.com/p/DBu5zCHBgrT/ 2024-10-29_withaseul - DBu5zCHBgrT.json
87 haseul withaseul withaseul DByVUvOBSQy post 2024-10-31 11:14:27 https://www.instagram.com/p/DByVUvOBSQy/ 2024-10-31_withaseul - DByVUvOBSQy.json
88 haseul withaseul withaseul DB5re5DhW8a post 2024-11-03 07:42:45 https://www.instagram.com/p/DB5re5DhW8a/ 2024-11-03_withaseul - DB5re5DhW8a.json
89 haseul withaseul withaseul DCBsfwGhFYd post 2024-11-06 10:25:32 https://www.instagram.com/p/DCBsfwGhFYd/ 2024-11-06_withaseul - DCBsfwGhFYd.json
90 haseul withaseul withaseul DCObB0SR64m post 2024-11-11 09:03:02 https://www.instagram.com/p/DCObB0SR64m/ 2024-11-11_withaseul - DCObB0SR64m.json
91 haseul withaseul withaseul DCRhOPphLWU post 2024-11-12 13:54:52 https://www.instagram.com/p/DCRhOPphLWU/ 2024-11-12_withaseul - DCRhOPphLWU.json
92 haseul withaseul withaseul DCbgyVghiT- post 2024-11-16 11:03:28 https://www.instagram.com/p/DCbgyVghiT-/ 2024-11-16_withaseul - DCbgyVghiT-.json
93 haseul withaseul withaseul DCe4qpXhgBV post 2024-11-17 18:29:51 https://www.instagram.com/p/DCe4qpXhgBV/ 2024-11-17_withaseul - DCe4qpXhgBV.json
94 haseul withaseul withaseul DC_4KddBmo_ post 2024-11-30 14:00:24 https://www.instagram.com/p/DC_4KddBmo_/ 2024-11-30_withaseul - DC_4KddBmo_.json
95 haseul withaseul withaseul DDRNi9Bhicn post 2024-12-07 07:34:20 https://www.instagram.com/p/DDRNi9Bhicn/ 2024-12-07_withaseul - DDRNi9Bhicn.json
96 haseul withaseul withaseul DDW2JdFBUMs post 2024-12-09 12:05:19 https://www.instagram.com/p/DDW2JdFBUMs/ 2024-12-09_withaseul - DDW2JdFBUMs.json
97 haseul withaseul withaseul DDjWETEBWw2 post 2024-12-14 08:35:07 https://www.instagram.com/p/DDjWETEBWw2/ 2024-12-14_withaseul - DDjWETEBWw2.json
98 haseul withaseul withaseul DDpkLTyhUdt post 2024-12-16 18:33:51 https://www.instagram.com/p/DDpkLTyhUdt/ 2024-12-16_withaseul - DDpkLTyhUdt.json
99 haseul withaseul withaseul DE1NVFihF2g post 2025-01-15 03:36:30 https://www.instagram.com/p/DE1NVFihF2g/ 2025-01-14_withaseul - DE1NVFihF2g.json
100 haseul withaseul withaseul DE5GO2-S5PB post 2025-01-16 15:51:26 https://www.instagram.com/p/DE5GO2-S5PB/ 2025-01-16_withaseul - DE5GO2-S5PB.json
101 haseul withaseul withaseul - reels DE5H7LcB5aX post 2025-01-16 16:07:31 https://www.instagram.com/p/DE5H7LcB5aX/ 2025-01-16_withaseul - DE5H7LcB5aX.json
102 haseul withaseul withaseul DFFtju9hTdv post 2025-01-21 13:25:58 https://www.instagram.com/p/DFFtju9hTdv/ 2025-01-21_withaseul - DFFtju9hTdv.json
103 haseul withaseul withaseul DFsLxphhwqE post 2025-02-05 12:01:09 https://www.instagram.com/p/DFsLxphhwqE/ 2025-02-05_withaseul - DFsLxphhwqE.json
104 haseul withaseul withaseul DHvlOgAxE_j post 2025-03-28 12:44:03 https://www.instagram.com/p/DHvlOgAxE_j/ 2025-03-28_withaseul - DHvlOgAxE_j.json
105 haseul withaseul withaseul DH96B0QtxfT post 2025-04-03 02:15:11 https://www.instagram.com/p/DH96B0QtxfT/ 2025-04-02_withaseul - DH96B0QtxfT.json
106 haseul withaseul withaseul DIL6wNXRGDC post 2025-04-08 12:50:53 https://www.instagram.com/p/DIL6wNXRGDC/ 2025-04-08_withaseul - DIL6wNXRGDC.json
107 haseul withaseul withaseul DIP1mift6QV post 2025-04-10 01:22:50 https://www.instagram.com/p/DIP1mift6QV/ 2025-04-09_withaseul - DIP1mift6QV.json
108 haseul withaseul withaseul DIfvDIKxwuT post 2025-04-16 05:33:25 https://www.instagram.com/p/DIfvDIKxwuT/ 2025-04-16_withaseul - DIfvDIKxwuT.json
109 haseul withaseul withaseul DIm-MshvCJy post 2025-04-19 01:00:03 https://www.instagram.com/p/DIm-MshvCJy/ 2025-04-18_withaseul - DIm-MshvCJy.json
110 haseul withaseul withaseul DIuQyfphWnR post 2025-04-21 20:57:37 https://www.instagram.com/p/DIuQyfphWnR/ 2025-04-21_withaseul - DIuQyfphWnR.json
111 haseul withaseul withaseul DI3hOzrBLs- post 2025-04-25 11:14:27 https://www.instagram.com/p/DI3hOzrBLs-/ 2025-04-25_withaseul - DI3hOzrBLs-.json
112 haseul withaseul withaseul DK0lklBhrH6 post 2025-06-13 00:57:27 https://www.instagram.com/p/DK0lklBhrH6/ 2025-06-12_withaseul - DK0lklBhrH6.json
113 haseul withaseul withaseul DK1EdUZBJtr post 2025-06-13 05:27:20 https://www.instagram.com/p/DK1EdUZBJtr/ 2025-06-13_withaseul - DK1EdUZBJtr.json
114 haseul withaseul withaseul DLJiMzrhM83 post 2025-06-21 04:12:02 https://www.instagram.com/p/DLJiMzrhM83/ 2025-06-21_withaseul - DLJiMzrhM83.json
115 haseul withaseul withaseul DLL1iePBcwm post 2025-06-22 01:39:30 https://www.instagram.com/p/DLL1iePBcwm/ 2025-06-21_withaseul - DLL1iePBcwm.json
116 haseul withaseul withaseul DLM-sUIhK5P post 2025-06-22 12:18:44 https://www.instagram.com/p/DLM-sUIhK5P/ 2025-06-22_withaseul - DLM-sUIhK5P.json
117 haseul withaseul withaseul DLU66yCvWZh post 2025-06-25 14:19:41 https://www.instagram.com/p/DLU66yCvWZh/ 2025-06-25_withaseul - DLU66yCvWZh.json
118 haseul withaseul withaseul DLxAZg5hpnW post 2025-07-06 12:06:18 https://www.instagram.com/p/DLxAZg5hpnW/ 2025-07-06_withaseul - DLxAZg5hpnW.json
119 haseul withaseul withaseul DMFfDqABJgM post 2025-07-14 10:59:00 https://www.instagram.com/p/DMFfDqABJgM/ 2025-07-14_withaseul - DMFfDqABJgM.json
120 haseul withaseul withaseul DM0NH9yhkzT post 2025-08-01 14:26:37 https://www.instagram.com/p/DM0NH9yhkzT/ 2025-08-01_withaseul - DM0NH9yhkzT.json
121 haseul withaseul withaseul DM9_Ol7SzmK post 2025-08-05 09:37:35 https://www.instagram.com/p/DM9_Ol7SzmK/ 2025-08-05_withaseul - DM9_Ol7SzmK.json
122 haseul withaseul withaseul DNnih4pBd_M post 2025-08-21 12:54:55 https://www.instagram.com/p/DNnih4pBd_M/ 2025-08-21_withaseul - DNnih4pBd_M.json
123 haseul withaseul withaseul DOIxptrgXgv post 2025-09-03 10:42:00 https://www.instagram.com/p/DOIxptrgXgv/ 2025-09-03_withaseul - DOIxptrgXgv.json
124 haseul withaseul withaseul DOa8a_ugc2Z post 2025-09-10 12:02:26 https://www.instagram.com/p/DOa8a_ugc2Z/ 2025-09-10_withaseul - DOa8a_ugc2Z.json
125 haseul withaseul withaseul DPMLxsmgUJJ post 2025-09-29 14:59:24 https://www.instagram.com/p/DPMLxsmgUJJ/ 2025-09-29_withaseul - DPMLxsmgUJJ.json
126 haseul withaseul withaseul DPRez1QgUxE post 2025-10-01 16:21:55 https://www.instagram.com/p/DPRez1QgUxE/ 2025-10-01_withaseul - DPRez1QgUxE.json
127 haseul withaseul withaseul DQg9WOOgaKD post 2025-11-01 13:08:45 https://www.instagram.com/p/DQg9WOOgaKD/ 2025-11-01_withaseul - DQg9WOOgaKD.json
128 haseul withaseul withaseul DQ_JrMdDciC post 2025-11-13 06:33:42 https://www.instagram.com/p/DQ_JrMdDciC/ 2025-11-13_withaseul - DQ_JrMdDciC.json
129 haseul withaseul withaseul DROhLjLjVqY post 2025-11-19 05:47:42 https://www.instagram.com/p/DROhLjLjVqY/ 2025-11-19_withaseul - DROhLjLjVqY.json
130 haseul withaseul withaseul DRwiTEDDRHR post 2025-12-02 10:51:38 https://www.instagram.com/p/DRwiTEDDRHR/ 2025-12-02_withaseul - DRwiTEDDRHR.json
131 haseul withaseul withaseul DTw-4d9gRUx post 2026-01-21 08:04:12 https://www.instagram.com/p/DTw-4d9gRUx/ 2026-01-21_withaseul - DTw-4d9gRUx.json
132 haseul withaseul withaseul DT0rHDCjPpE post 2026-01-22 18:28:24 https://www.instagram.com/p/DT0rHDCjPpE/ 2026-01-22_withaseul - DT0rHDCjPpE.json
133 haseul withaseul withaseul DT6HqajDJfF post 2026-01-24 21:14:05 https://www.instagram.com/p/DT6HqajDJfF/ 2026-01-24_withaseul - DT6HqajDJfF.json
134 haseul withaseul withaseul DU0oU25gWso post 2026-02-16 14:19:54 https://www.instagram.com/p/DU0oU25gWso/ 2026-02-16_withaseul - DU0oU25gWso.json
135 haseul withaseul withaseul DWZPNAKAZcl post 2026-03-27 16:19:14 https://www.instagram.com/p/DWZPNAKAZcl/ 2026-03-27_withaseul - DWZPNAKAZcl.json
136 haseul withaseul withaseul DW3yBm2gctK post 2026-04-08 13:00:00 https://www.instagram.com/p/DW3yBm2gctK/ 2026-04-08_withaseul - DW3yBm2gctK.json
137 haseul withaseul withaseul DZj6uVWhV8M post 2026-06-14 09:26:10 https://www.instagram.com/p/DZj6uVWhV8M/ 2026-06-14_withaseul - DZj6uVWhV8M.json
138 haseul withaseul withaseul DaZm_xLAZeL post 2026-07-05 05:52:43 https://www.instagram.com/p/DaZm_xLAZeL/ 2026-07-05_withaseul - DaZm_xLAZeL.json
139 haseul withaseul withaseul Dbnq4k9gb5U post 2026-08-04 13:27:27 https://www.instagram.com/p/Dbnq4k9gb5U/ 2026-08-04_withaseul - Dbnq4k9gb5U.json
140 heejin 0ct0ber19 0ct0ber19 CrdsY5CrSsO post 2023-04-25 15:21:16 https://www.instagram.com/p/CrdsY5CrSsO/ 2023-04-25_0ct0ber19 - CrdsY5CrSsO.json
141 heejin 0ct0ber19 0ct0ber19 CtohvHxLnWO post 2023-06-18 13:22:37 https://www.instagram.com/p/CtohvHxLnWO/ 2023-06-18_0ct0ber19 - CtohvHxLnWO.json
142 heejin 0ct0ber19 0ct0ber19 CuEOMWppp5S post 2023-06-29 07:30:35 https://www.instagram.com/p/CuEOMWppp5S/ 2023-06-29_0ct0ber19 - CuEOMWppp5S.json
143 heejin 0ct0ber19 0ct0ber19 CwC0Y-prGoB post 2023-08-17 11:28:40 https://www.instagram.com/p/CwC0Y-prGoB/ 2023-08-17_0ct0ber19 - CwC0Y-prGoB.json
144 heejin 0ct0ber19 0ct0ber19 CxsKPWgpzU4 post 2023-09-27 09:19:51 https://www.instagram.com/p/CxsKPWgpzU4/ 2023-09-27_0ct0ber19 - CxsKPWgpzU4.json
145 heejin 0ct0ber19 0ct0ber19 CzJf78xryub post 2023-11-02 15:18:48 https://www.instagram.com/p/CzJf78xryub/ 2023-11-02_0ct0ber19 - CzJf78xryub.json
146 heejin 0ct0ber19 0ct0ber19 CzOVVzFLTnD post 2023-11-04 12:22:25 https://www.instagram.com/p/CzOVVzFLTnD/ 2023-11-04_0ct0ber19 - CzOVVzFLTnD.json
147 heejin 0ct0ber19 0ct0ber19 C14VKVmpdtC post 2024-01-09 12:51:44 https://www.instagram.com/p/C14VKVmpdtC/ 2024-01-09_0ct0ber19 - C14VKVmpdtC.json
148 heejin 0ct0ber19 0ct0ber19 C2FnIelpue9 post 2024-01-14 16:38:54 https://www.instagram.com/p/C2FnIelpue9/ 2024-01-14_0ct0ber19 - C2FnIelpue9.json
149 heejin 0ct0ber19 0ct0ber19 C5SDY_UJIn6 post 2024-04-03 02:10:53 https://www.instagram.com/p/C5SDY_UJIn6/ 2024-04-02_0ct0ber19 - C5SDY_UJIn6.json
150 heejin 0ct0ber19 0ct0ber19 C5WB6zwp_rX post 2024-04-04 15:15:00 https://www.instagram.com/p/C5WB6zwp_rX/ 2024-04-04_0ct0ber19 - C5WB6zwp_rX.json
151 heejin 0ct0ber19 0ct0ber19 C5dJ0xXpJpX post 2024-04-07 09:38:46 https://www.instagram.com/p/C5dJ0xXpJpX/ 2024-04-07_0ct0ber19 - C5dJ0xXpJpX.json
152 heejin 0ct0ber19 0ct0ber19 C5oJdT-pQBI post 2024-04-11 16:07:12 https://www.instagram.com/p/C5oJdT-pQBI/ 2024-04-11_0ct0ber19 - C5oJdT-pQBI.json
153 heejin 0ct0ber19 0ct0ber19 C53YPQzp7Wj post 2024-04-17 14:04:58 https://www.instagram.com/p/C53YPQzp7Wj/ 2024-04-17_0ct0ber19 - C53YPQzp7Wj.json
154 heejin 0ct0ber19 0ct0ber19 C5--hdUJKZH post 2024-04-20 12:54:11 https://www.instagram.com/p/C5--hdUJKZH/ 2024-04-20_0ct0ber19 - C5--hdUJKZH.json
155 heejin 0ct0ber19 0ct0ber19 C71iAmbpekA post 2024-06-05 13:54:39 https://www.instagram.com/p/C71iAmbpekA/ 2024-06-05_0ct0ber19 - C71iAmbpekA.json
156 heejin 0ct0ber19 0ct0ber19 C8M8TxnSOKI post 2024-06-14 16:07:00 https://www.instagram.com/p/C8M8TxnSOKI/ 2024-06-14_0ct0ber19 - C8M8TxnSOKI.json
157 heejin 0ct0ber19 0ct0ber19 C8hxpuYyWGv post 2024-06-22 18:17:55 https://www.instagram.com/p/C8hxpuYyWGv/ 2024-06-22_0ct0ber19 - C8hxpuYyWGv.json
158 heejin 0ct0ber19 0ct0ber19 C-AbtWSJbHB post 2024-07-29 12:34:02 https://www.instagram.com/p/C-AbtWSJbHB/ 2024-07-29_0ct0ber19 - C-AbtWSJbHB.json
159 heejin 0ct0ber19 0ct0ber19 C_WxFtuvqXK post 2024-09-01 01:15:32 https://www.instagram.com/p/C_WxFtuvqXK/ 2024-08-31_0ct0ber19 - C_WxFtuvqXK.json
160 heejin 0ct0ber19 0ct0ber19 DcgTbz-iTAH post 2026-08-26 13:19:15 https://www.instagram.com/p/DcgTbz-iTAH/ 2026-08-26_0ct0ber19 - DcgTbz-iTAH.json
161 jinsoul zindoriyam zindoriyam CndmboMBHeB post 2023-01-16 04:23:38 https://www.instagram.com/p/CndmboMBHeB/ 2023-01-15_zindoriyam - CndmboMBHeB.json
162 jinsoul zindoriyam zindoriyam Ctb5AnZxCwh post 2023-06-13 15:35:51 https://www.instagram.com/p/Ctb5AnZxCwh/ 2023-06-13_zindoriyam - Ctb5AnZxCwh.json
163 jinsoul zindoriyam zindoriyam Cu6ZRz8hj_T post 2023-07-20 08:26:26 https://www.instagram.com/p/Cu6ZRz8hj_T/ 2023-07-20_zindoriyam - Cu6ZRz8hj_T.json
164 jinsoul zindoriyam zindoriyam Cwrydt6OXmo post 2023-09-02 09:20:42 https://www.instagram.com/p/Cwrydt6OXmo/ 2023-09-02_zindoriyam - Cwrydt6OXmo.json
165 jinsoul zindoriyam zindoriyam Cxsj3_IhwNv post 2023-09-27 13:03:51 https://www.instagram.com/p/Cxsj3_IhwNv/ 2023-09-27_zindoriyam - Cxsj3_IhwNv.json
166 jinsoul zindoriyam zindoriyam Cza0C4gBkb5 post 2023-11-09 08:41:36 https://www.instagram.com/p/Cza0C4gBkb5/ 2023-11-09_zindoriyam - Cza0C4gBkb5.json
167 jinsoul zindoriyam zindoriyam C5LqGJUBTIK post 2024-03-31 14:34:25 https://www.instagram.com/p/C5LqGJUBTIK/ 2024-03-31_zindoriyam - C5LqGJUBTIK.json
168 jinsoul zindoriyam zindoriyam C5QZg9BBc12 post 2024-04-02 10:45:44 https://www.instagram.com/p/C5QZg9BBc12/ 2024-04-02_zindoriyam - C5QZg9BBc12.json
169 jinsoul zindoriyam zindoriyam C8NAliShdHm post 2024-06-14 16:44:22 https://www.instagram.com/p/C8NAliShdHm/ 2024-06-14_zindoriyam - C8NAliShdHm.json
170 jinsoul zindoriyam zindoriyam C9-Ot__hyGY post 2024-07-28 16:02:02 https://www.instagram.com/p/C9-Ot__hyGY/ 2024-07-28_zindoriyam - C9-Ot__hyGY.json
171 jinsoul zindoriyam zindoriyam DANQU24um-N post 2024-09-22 05:07:29 https://www.instagram.com/p/DANQU24um-N/ 2024-09-22_zindoriyam - DANQU24um-N.json
172 jinsoul zindoriyam zindoriyam DHz88e2sS_J post 2025-03-30 05:28:16 https://www.instagram.com/p/DHz88e2sS_J/ 2025-03-30_zindoriyam - DHz88e2sS_J.json
173 jinsoul zindoriyam zindoriyam DJXAVrANWfp post 2025-05-07 16:42:44 https://www.instagram.com/p/DJXAVrANWfp/ 2025-05-07_zindoriyam - DJXAVrANWfp.json
174 jinsoul zindoriyam zindoriyam DLhdKkZJ2_5 post 2025-06-30 11:09:49 https://www.instagram.com/p/DLhdKkZJ2_5/ 2025-06-30_zindoriyam - DLhdKkZJ2_5.json
175 jinsoul zindoriyam zindoriyam DMiZJbaJZXv post 2025-07-25 16:25:21 https://www.instagram.com/p/DMiZJbaJZXv/ 2025-07-25_zindoriyam - DMiZJbaJZXv.json
176 jinsoul zindoriyam zindoriyam DNJCd9jpT-J post 2025-08-09 16:37:33 https://www.instagram.com/p/DNJCd9jpT-J/ 2025-08-09_zindoriyam - DNJCd9jpT-J.json
177 jinsoul zindoriyam zindoriyam DRl8jAxFnwE post 2025-11-28 08:09:22 https://www.instagram.com/p/DRl8jAxFnwE/ 2025-11-28_zindoriyam - DRl8jAxFnwE.json
178 jinsoul zindoriyam zindoriyam DTQR5cDk6kY post 2026-01-08 15:15:25 https://www.instagram.com/p/DTQR5cDk6kY/ 2026-01-08_zindoriyam - DTQR5cDk6kY.json
179 jinsoul zindoriyam zindoriyam DVv1oRSEyIy post 2026-03-11 14:26:54 https://www.instagram.com/p/DVv1oRSEyIy/ 2026-03-11_zindoriyam - DVv1oRSEyIy.json
180 jinsoul zindoriyam zindoriyam DW_6oxSkz8u post 2026-04-11 16:49:54 https://www.instagram.com/p/DW_6oxSkz8u/ 2026-04-11_zindoriyam - DW_6oxSkz8u.json
181 jinsoul zindoriyam zindoriyam DcRLAVJEwlG post 2026-08-20 16:16:59 https://www.instagram.com/p/DcRLAVJEwlG/ 2026-08-20_zindoriyam - DcRLAVJEwlG.json
182 kimlip kimxxlip kimxxlip CndmBanNyrR post 2023-01-16 04:20:04 https://www.instagram.com/p/CndmBanNyrR/ 2023-01-15_kimxxlip - CndmBanNyrR.json
183 kimlip kimxxlip kimxxlip CwXG1L3vQFH post 2023-08-25 08:34:37 https://www.instagram.com/p/CwXG1L3vQFH/ 2023-08-25_kimxxlip - CwXG1L3vQFH.json
184 kimlip kimxxlip kimxxlip CxsL_FQvHna post 2023-09-27 09:35:06 https://www.instagram.com/p/CxsL_FQvHna/ 2023-09-27_kimxxlip - CxsL_FQvHna.json
+1 -14
View File
@@ -1,17 +1,4 @@
# JDownloader2 — archive fetching
> **This file lives only on the `tooling` branch.** `main` is published to
> GitHub and deliberately carries none of this — not the host details, not the
> IPs, and not the account names. `main`'s history was redacted on 2026-08-20;
> real names exist only here.
>
> There is no `npm run jd2` script — `package.json` and `CLAUDE.md` are kept
> byte-identical to `main` so that merging `main` into `tooling` never
> conflicts. Run the crawljob generator directly:
>
> ```sh
> npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run
> ```
# JDownloader2 — archive fetching quick reference
How content gets into this archive, and why the setup is shaped the way it is.
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "instaarchive-viewer",
"version": "1.8.1",
"version": "1.8.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "instaarchive-viewer",
"version": "1.8.1",
"version": "1.8.0",
"dependencies": {
"@tailwindcss/vite": "^4.1.14",
"@vitejs/plugin-react": "^5.0.4",
+4 -3
View File
@@ -1,18 +1,19 @@
{
"name": "instaarchive-viewer",
"private": true,
"version": "1.8.1",
"version": "1.8.0",
"type": "module",
"scripts": {
"dev": "vite --port=3000 --host=0.0.0.0",
"build": "vite build && npm run build:server",
"build:server": "tsc server.ts --esModuleInterop --module ESNext --target ES2022 --moduleResolution bundler --removeComments --outDir dist-server",
"build:server": "tsc server.ts --esModuleInterop --module ESNext --target ES2022 --moduleResolution bundler --outDir dist-server",
"preview": "vite preview",
"server": "tsx server.ts",
"clean": "rm -rf dist",
"lint": "tsc --noEmit",
"test": "vitest run",
"test:watch": "vitest"
"test:watch": "vitest",
"jd2": "tsx scripts/jd2-sync.ts"
},
"dependencies": {
"@tailwindcss/vite": "^4.1.14",
Binary file not shown.
Binary file not shown.
-112
View File
@@ -1,112 +0,0 @@
#!/bin/bash
# Unattended wrapper around gdl-sync.py. One argument: the run mode.
#
# stories daily ~6 requests; the only surface that cannot be backfilled
# profiles monthly every surface, --abort 50: stops enumerating a profile
# once it reaches content already held, so it costs
# ~40-60 requests and catches everything NEW
# full-sweep rarely every surface, no abort: walks each profile to the end
# for ~420 requests. The only run that notices posts
# EDITED after we archived them, and by far the most
# expensive thing here -- see TOOLING.md before running.
#
# Exits non-zero if the sync does, so cron mails you. Everything is logged.
set -eu
MODE="${1:?usage: gdl-cron.sh stories|profiles|full-sweep}"
GDL_HOME="${GDL_HOME:-$HOME/gdl}"
INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
STAGING="$GDL_HOME/staging-$MODE"
# $$ in the name because two runs in the same SECOND would otherwise share a
# log file, and `tee -a` appends -- which made a test see the previous run.
LOG="$GDL_HOME/logs/$MODE-$(date +%Y%m%d-%H%M%S)-$$.log"
PATH="$HOME/.local/bin:$PATH"; export PATH
# Pacing. These are the values the 2026-08-22 runs used by hand, after the
# scraping warning -- roughly double the caution of gdl-sync.py's own defaults
# (6-10s / 3-6s / 1M). An archive sync has no deadline; being slow is free and
# being restricted is not. Override per-run with GDL_SLEEP_REQUEST etc. if you
# ever need to, but raise them rather than lower them.
SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
SLEEP="${GDL_SLEEP:-5 10}"
RATE="${GDL_RATE:-500K}"
case "$MODE" in
# --min-interval 8, not the 20h default. The floor exists to stop an ABORTED
# RESTART re-enumerating profiles -- a minutes-to-hours concern. At 20h the
# daily timer silently did nothing whenever a manual run had happened the
# previous afternoon, which is exactly what happened on 2026-08-21: it fired,
# skipped all six sources and reported success. A stories fetch is one
# request per profile, so the worst case 8h permits is roughly twelve
# requests in a day instead of six.
stories) ARGS="--only stories --min-interval 8" ;;
profiles) ARGS="--only posts,reels,stories,highlights --abort 50" ;;
# No --abort: the whole point of full-sweep is enumerating to the end, so it
# is the only run that notices carousels edited after we archived them.
full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
# Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
# "sweep" did not say it was the exhaustive one. Catch the old names rather
# than failing with a bare error, in case something still passes them.
full) echo "mode 'full' was renamed to 'profiles'" >&2; exit 2 ;;
sweep) echo "mode 'sweep' was renamed to 'full-sweep'" >&2; exit 2 ;;
*) echo "unknown mode: $MODE (want stories|profiles|full-sweep)" >&2; exit 2 ;;
esac
mkdir -p "$GDL_HOME/logs"
# Staging is wiped every run ON PURPOSE. What we already hold is decided by the
# skip-archive (--download-archive), never by which files happen to be sitting
# in staging, so starting empty is correct -- and it keeps the publish rsync
# to just the new files instead of re-walking gigabytes each time.
rm -rf "$STAGING"
echo "=== $MODE run $(date -Is) ===" | tee -a "$LOG"
# The exit status has to survive the pipe into tee. The left-hand side of a
# pipeline runs in a SUBSHELL, so an `exit` in there sets the subshell's status
# and the script goes on to return tee's, which is always 0. An earlier version
# of this file did exactly that and reported success no matter what the sync
# did -- which is why the shebang is bash: PIPESTATUS is the fix.
# This run's output only. The check below must never see a previous run's
# lines, so it reads this rather than the (appended-to) log.
RUNOUT=$(mktemp)
trap 'rm -f "$RUNOUT"' EXIT
set +e
# shellcheck disable=SC2086
"$GDL_HOME/gdl-sync.py" \
--index "$INDEX" \
--staging "$STAGING" \
--publish "$PUBLISH" \
--archive-db "$GDL_HOME/artms.db" \
--urls-file "$GDL_HOME/artms_account_links.txt" \
--sleep-request $SLEEP_REQUEST \
--sleep $SLEEP \
--rate "$RATE" \
$ARGS --execute 2>&1 | tee -a "$LOG" "$RUNOUT"
status=${PIPESTATUS[0]}
set -e
# A stories run that skipped every source is NOT a success. It means the
# min-interval floor blocked the one surface that cannot be backfilled, and
# without this it looks identical to a clean run: exit 0, "0 step(s) failed".
# Note this is not the same as "no stories today" -- that shows up as sources
# being fetched and returning no results, which is normal and stays quiet.
if [ "$MODE" = "stories" ] && grep -q "sources : 0 to sync" "$RUNOUT"; then
echo "WARNING: every stories source was skipped by --min-interval." | tee -a "$LOG" >&2
echo " Nothing was fetched. Stories expire in 24h and cannot be" | tee -a "$LOG" >&2
echo " backfilled, so this is a real loss, not a quiet no-op." | tee -a "$LOG" >&2
[ "$status" -eq 0 ] && status=75
fi
echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
# Keep the log directory from growing without bound.
ls -1t "$GDL_HOME/logs" | tail -n +30 | while read -r old; do
rm -f "$GDL_HOME/logs/$old"
done
exit $status
+6 -201
View File
@@ -23,78 +23,18 @@ the one that would touch the archive.
Run it from the host whose public IP matches the browser the cookie came from;
using the cookie from elsewhere is what session-hijack detection looks for.
See `--help` for every flag, with worked examples for each way of selecting
what to fetch.
"""
from __future__ import annotations
EXAMPLES = """\
examples:
Routine incremental sync of the tracked profiles (what gdl-cron.sh runs) --
--abort 50 stops enumerating each profile once it reaches content already
held, so a run that has seeded once costs ~40-60 requests, not a full walk:
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \\
--staging /var/tmp/gdl --publish user@host:/path/to/archives \\
--urls-file artms_account_links.txt --archive-db /var/tmp/gdl.db \\
--abort 50 --dry-run
# ...then swap --dry-run for --execute once the plan looks right.
Cheapest possible run -- stories only, the one surface that expires in 24h
and cannot be backfilled, so it is worth doing often:
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \\
--staging /var/tmp/gdl --publish user@host:/path/to/archives \\
--urls-file artms_account_links.txt --only stories --execute
One profile, by name, without a urls-file:
./scripts/gdl-sync.py --index /path/to/archives --staging /var/tmp/gdl \\
--publish /path/to/archives --profile some_account --execute
Every profile the archive already knows about (no urls-file, no --profile):
./scripts/gdl-sync.py --index /path/to/archives --staging /var/tmp/gdl \\
--publish /path/to/archives --all --execute
An arbitrary single post or reel from an account NOT otherwise tracked --
e.g. a link someone shared. Filed under its owner like any other post; no
--index needed, since there is no profile list to plan against:
./scripts/gdl-sync.py --staging /var/tmp/gdl --publish user@host:/path \\
--post-url https://www.instagram.com/p/SHORTCODE/ --execute
Full sweep -- no --abort, walks every profile to the end. The only run that
notices a carousel edited after it was archived, and by far the most
expensive thing here (~420 requests for six profiles). Read docs/gallery-dl.md
and TOOLING.md before running this one:
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \\
--staging /var/tmp/gdl --publish user@host:/path/to/archives \\
--urls-file artms_account_links.txt --execute
Hand-paced caution after a scraping warning (roughly double the defaults;
see docs/gallery-dl.md for where these numbers come from):
./scripts/gdl-sync.py ... --sleep-request 12 20 --sleep 5 10 --rate 500K
Always --dry-run first (the default): it prints the plan and the rsync
command that would publish, without spending a single Instagram request.
"""
import argparse
import datetime as dt
import json
import os
import random
import re
import shutil
import subprocess
import sys
import time
from dataclasses import dataclass, field
from pathlib import Path
@@ -221,23 +161,6 @@ def read_urls_file(path: Path) -> list[str]:
return users
def read_post_urls_file(path: Path) -> list[str]:
"""
Read individual post/reel URLs from a file, one per line -- the output
format `reels-scrape.py` writes. Blank lines and `#` comments are skipped;
order is kept and duplicates dropped, same conventions as read_urls_file.
"""
urls: list[str] = []
seen: set[str] = set()
for raw in path.read_text().splitlines():
line = raw.split("#", 1)[0].strip()
if not line or line in seen:
continue
seen.add(line)
urls.append(line)
return urls
class ArchiveIndex:
"""
What the archive already holds, as filenames only.
@@ -325,32 +248,6 @@ def build_config(rate: str, sleep_request: list[float],
"post_shortcode", "post_id", "type", "date", "post_date",
"username", "fullname", "owner_id", "description", "count",
"likes", "post_url", "sidecar_shortcode",
# A Collab's `username`/`fullname`/`owner_id` above are always the
# original poster's, never the scraped account's -- see
# docs/gallery-dl.md. `coauthors` is the API's own list of every
# OTHER collaborator (it excludes the post's owner), a direct
# signal instead of inferring a collab from identity mismatches.
"coauthors",
],
}
# Per-carousel-item fields (`width`/`height`/`tagged_users`/`owner`) live
# on the per-FILE kwdict, not the per-post one `meta_pp` reads -- a
# carousel's items can each have different dimensions and tags, which one
# post-level JSON can't represent. `owner` is deliberately left out: it is
# a full user object (profile pic URLs, privacy flags) for whoever posted
# that specific item, the same reason `audio_user` is excluded above.
media_pp = {
"name": "metadata",
"mode": "json",
# event defaults to "file" when omitted -- runs once per downloaded
# file, appending ".json" to that file's own full name, e.g.
# "... - 01.jpg" gets "... - 01.jpg.json" alongside it. Never
# collides with the post-level "....json" above, which has no
# per-item number.
"include": [
"media_id", "shortcode", "num",
"width", "height", "width_original", "height_original",
"tagged_users",
],
}
@@ -389,7 +286,6 @@ def build_config(rate: str, sleep_request: list[float],
"postprocessors": [
{**caption_pp, "filename": stem + ".txt"},
{**meta_pp, "filename": stem + ".json"},
media_pp,
],
}
@@ -435,12 +331,6 @@ def build_config(rate: str, sleep_request: list[float],
"directory": [],
"posts": post_like(POST_STEM),
"reels": post_like(POST_STEM),
# Ad hoc single-post fetches (--post-url) go through gallery-dl's
# own post/reel extractor instead of a profile listing, so the
# owning account is never known ahead of time -- only mid-
# extraction, same reasoning as highlights below.
"post": {**post_like(POST_STEM), "directory": ["{username}"]},
"reel": {**post_like(POST_STEM), "directory": ["{username}"]},
"stories": item_like(ITEM_STEM),
"highlights": {
**item_like(ITEM_STEM),
@@ -473,11 +363,9 @@ def gdl_command(src: Source, staging: Path, config: Path, cookies: str,
cmd += ["--download-archive", str(archive_db)]
# Forced destination -- never `{username}` -- because a reels tab returns
# collab reels owned by other accounts, which would otherwise be filed
# under the wrong profile. Highlights and ad hoc single-post fetches are
# the exception: their directory is only known mid-extraction (a title, or
# the post's owner), so the config formats it instead.
dest = (staging if src.subcategory in ("highlights", "post", "reel")
else staging / src.directory)
# under the wrong profile. Highlights are the exception: their directory
# embeds a title only known mid-extraction, so the config formats it.
dest = staging if src.subcategory == "highlights" else staging / src.directory
cmd += ["--destination", str(dest)]
cmd.append(src.url)
return cmd
@@ -751,61 +639,6 @@ def publish(staging: Path, dest: str, dry_run: bool) -> int:
return subprocess.run(cmd).returncode
def run_post_urls(args) -> int:
"""
Fetch one or more individual posts/reels by URL -- an ad hoc pull outside
the tracked profile list, e.g. a link shared from some other account. Each
is a single request, not a recurring surface, so there is no archive-db
seeding and no --min-interval floor to plan around.
"""
config = build_config(args.rate, list(args.sleep_request),
list(args.sleep), abort=0)
args.staging.mkdir(parents=True, exist_ok=True)
config_path = args.staging.parent / f"{args.staging.name}.gdl-config.json"
sources = [Source("post", url, "", "post") for url in args.post_url]
print(f"post-url : {len(sources)} to fetch")
print(f"pacing : {args.sleep_request[0]}-{args.sleep_request[1]}s between "
f"requests, rate cap {args.rate}")
print(f"staging : {args.staging}")
print(f"publish : {args.publish}")
print()
if not args.execute:
for src in sources:
print(f" {src.url}")
print()
print(" " + " ".join(rsync_command(args.staging, args.publish, True)))
print("\ndry run; nothing fetched. pass --execute to run.")
return 0
config_path.write_text(json.dumps(config, indent=2))
failures = 0
for i, src in enumerate(sources):
# gallery-dl's own sleep-request only paces requests INSIDE one
# invocation; each URL here is its own subprocess, so back-to-back
# items would otherwise fire with no gap at all -- the same pacing
# applied here as between requests within a single fetch.
if i:
pause = random.uniform(*args.sleep_request)
time.sleep(pause)
print(f"==> {src.url}")
cmd = gdl_command(src, args.staging, config_path, args.cookies,
args.archive_db)
result = subprocess.run(cmd)
if result.returncode != 0:
failures += 1
print(f" FAILED (exit {result.returncode})", file=sys.stderr)
print("\n==> publish")
if publish(args.staging, args.publish, dry_run=False) != 0:
failures += 1
print(f"\ndone; {failures} step(s) failed")
return 1 if failures else 0
def main() -> int:
# A sync runs for hours and is normally watched through a redirected log,
# where Python's block buffering would withhold progress until it happened
@@ -814,13 +647,12 @@ def main() -> int:
sys.stdout.reconfigure(line_buffering=True)
sys.stderr.reconfigure(line_buffering=True)
ap = argparse.ArgumentParser(description=__doc__, epilog=EXAMPLES,
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--index",
ap.add_argument("--index", required=True,
help="existing archive listing: a local root, or the "
"viewer's base URL (only a FILE LISTING is needed, "
"never the contents). Required unless --post-url/"
"--post-urls-file")
"never the contents)")
ap.add_argument("--publish", required=True,
help="rsync destination for fetched files; a local path or "
"user@host:/path")
@@ -833,17 +665,6 @@ def main() -> int:
g.add_argument("--urls-file", type=Path,
help="file of Instagram profile URLs or usernames, one per "
"line; # comments and blank lines allowed")
g.add_argument("--post-url", action="append", default=[],
help="fetch one post or reel by URL (e.g. "
"https://www.instagram.com/p/SHORTCODE/), filed under "
"its owner's account like any other post; repeatable. "
"A one-off fetch outside the tracked profile list: no "
"archive-db seeding, no --min-interval floor")
g.add_argument("--post-urls-file", type=Path,
help="file of post/reel URLs, one per line; # comments and "
"blank lines allowed. Same handling as --post-url, "
"paced with --sleep-request between items. This is "
"the format reels-scrape.py writes")
ap.add_argument("--cookies", default="chrome:/home/matt/.config/google-chrome-devtools",
help="gallery-dl --cookies-from-browser value")
ap.add_argument("--archive-db", type=Path, default=None,
@@ -888,22 +709,6 @@ def main() -> int:
print("rsync not on PATH", file=sys.stderr)
return 2
if args.post_urls_file:
if not args.post_urls_file.is_file():
print(f"post-urls file not found: {args.post_urls_file}", file=sys.stderr)
return 2
args.post_url = read_post_urls_file(args.post_urls_file)
if not args.post_url:
print(f"no usable URLs in {args.post_urls_file}", file=sys.stderr)
return 2
if args.post_url:
return run_post_urls(args)
if not args.index:
print("--index is required unless --post-url/--post-urls-file is given",
file=sys.stderr)
return 2
index = ArchiveIndex(args.index)
names = index.profiles()
if args.urls_file:
-237
View File
@@ -1,237 +0,0 @@
#!/usr/bin/env python3
"""
Find a profile's reels by scrolling the real page, not calling the API.
gallery-dl's dedicated reels extractor POSTs to /api/v1/clips/user/, which
Instagram now 302-redirects to the home page for this account -- confirmed on
2026-08-26/27 across multiple profiles, hours apart, with a freshly-warmed
session and a correct X-IG-WWW-Claim header. The reels tab itself loads fine
in a real browser, so this drives the SAME logged-in Chrome (already exposed
on loopback CDP for MCP automation -- see TOOLING.md) via the DevTools
protocol, scrolls it like a person would, and scrapes `/reel/<code>/` links
out of the rendered page instead.
This only finds shortcodes; it never downloads anything itself. What comes
out the other end is deduped against the archive (via the same --index used
elsewhere) and printed as plain post URLs -- feed them to gdl-sync.py:
./scripts/reels-scrape.py --profile someuser \\
--index https://instaarchive.ergosteur.com > /tmp/someuser-reels.txt
./scripts/gdl-sync.py --publish user@host:/path --staging /var/tmp/gdl \\
--post-urls-file /tmp/someuser-reels.txt \\
--sleep-request 12 20 --sleep 5 10 --rate 500K --execute
Requires the `websocket-client` package (only imported inside scrape_reels,
so everything else here stays importable -- and testable -- without it).
"""
from __future__ import annotations
import argparse
import itertools
import json
import random
import re
import sys
import time
from pathlib import Path
from urllib.request import urlopen
RE_REEL_HREF = re.compile(r"/reel/([^/?#]+)")
# The reels tab has finished loading once a scroll round finds no NEW
# shortcodes this many times in a row -- lazy-loaded feeds sometimes stall
# for a round or two before producing more, so one dry round is not enough.
DEFAULT_MAX_IDLE_ROUNDS = 3
DEFAULT_MAX_SCROLLS = 200
DEFAULT_SCROLL_PAUSE = (2.0, 3.5)
SCRAPE_JS = (
"(() => {"
"window.scrollTo(0, document.body.scrollHeight);"
"return Array.from(document.querySelectorAll('a[href*=\"/reel/\"]'))"
".map(a => a.getAttribute('href'));"
"})()"
)
def extract_shortcodes(hrefs: list[str]) -> list[str]:
codes = []
for href in hrefs:
if m := RE_REEL_HREF.search(href):
codes.append(m.group(1))
return codes
class CDPError(RuntimeError):
pass
class CDP:
"""
A deliberately minimal synchronous DevTools Protocol client: one request
in flight at a time, which is all a linear scroll-and-scrape loop needs.
Anything fancier (concurrent requests, event subscriptions) is scope this
script has no reason to carry.
"""
def __init__(self, ws_url: str, timeout: float = 30.0):
import websocket # local: keep this importable without the package
self.ws = websocket.create_connection(ws_url, timeout=timeout)
self._ids = itertools.count(1)
def send(self, method: str, params: dict | None = None,
session_id: str | None = None) -> dict:
msg_id = next(self._ids)
payload = {"id": msg_id, "method": method, "params": params or {}}
if session_id:
payload["sessionId"] = session_id
self.ws.send(json.dumps(payload))
while True:
msg = json.loads(self.ws.recv())
if msg.get("id") != msg_id:
continue # an event notification, not our reply -- ignore
if "error" in msg:
raise CDPError(f"{method}: {msg['error']}")
return msg.get("result", {})
def close(self) -> None:
self.ws.close()
def scrape_reels(user: str, cdp_port: int = 9222,
scroll_pause: tuple[float, float] = DEFAULT_SCROLL_PAUSE,
max_idle_rounds: int = DEFAULT_MAX_IDLE_ROUNDS,
max_scrolls: int = DEFAULT_MAX_SCROLLS,
log=lambda msg: None) -> list[str]:
"""
Open the profile's reels tab in a NEW tab of the already-signed-in Chrome,
scroll it to the bottom repeatedly, and collect every unique `/reel/`
shortcode that appears. Closes the tab when done either way.
"""
version = json.loads(urlopen(f"http://localhost:{cdp_port}/json/version",
timeout=10).read())
browser = CDP(version["webSocketDebuggerUrl"])
target_id = None
try:
target = browser.send("Target.createTarget", {
"url": f"https://www.instagram.com/{user}/reels/"})
target_id = target["targetId"]
attach = browser.send("Target.attachToTarget", {
"targetId": target_id, "flatten": True})
session_id = attach["sessionId"]
browser.send("Page.enable", session_id=session_id)
browser.send("Runtime.enable", session_id=session_id)
time.sleep(4.0) # initial page load, before the first scroll
seen: set[str] = set()
idle = 0
for i in range(max_scrolls):
result = browser.send(
"Runtime.evaluate",
{"expression": SCRAPE_JS, "returnByValue": True},
session_id=session_id)
hrefs = result.get("result", {}).get("value") or []
codes = extract_shortcodes(hrefs)
new = [c for c in codes if c not in seen]
seen.update(new)
log(f" scroll {i + 1}: {len(seen)} unique reels so far (+{len(new)})")
if new:
idle = 0
else:
idle += 1
if idle >= max_idle_rounds:
break
time.sleep(random.uniform(*scroll_pause))
return sorted(seen)
finally:
if target_id:
try:
browser.send("Target.closeTarget", {"targetId": target_id})
except CDPError:
pass # best-effort cleanup; a leftover tab is harmless
browser.close()
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--profile", required=True,
help="Instagram username to scrape reels for")
ap.add_argument("--index", required=True,
help="existing archive listing, same as gdl-sync.py's "
"--index: a local root, or the viewer's base URL. "
"Used only to dedupe -- already-archived shortcodes "
"are dropped before anything is printed")
ap.add_argument("--cdp-port", type=int, default=9222,
help="Chrome's loopback DevTools port (see TOOLING.md)")
ap.add_argument("--scroll-pause", nargs=2, type=float,
default=list(DEFAULT_SCROLL_PAUSE), metavar=("MIN", "MAX"),
help="random pause between scrolls, seconds")
ap.add_argument("--max-idle-rounds", type=int, default=DEFAULT_MAX_IDLE_ROUNDS,
help="stop after this many consecutive scrolls with no "
"new reels")
ap.add_argument("--max-scrolls", type=int, default=DEFAULT_MAX_SCROLLS,
help="hard ceiling on scroll rounds, in case a page never "
"goes idle")
ap.add_argument("--out", type=Path,
help="write new reel URLs here, one per line (default: "
"stdout)")
args = ap.parse_args()
import importlib.util
spec = importlib.util.spec_from_file_location(
"gdl_sync", Path(__file__).with_name("gdl-sync.py"))
gdl = importlib.util.module_from_spec(spec)
sys.modules["gdl_sync"] = gdl
spec.loader.exec_module(gdl)
index = gdl.ArchiveIndex(args.index)
# Deduped against the WHOLE archive, not just this profile's own
# directories: a shortcode is globally unique, and a reels tab commonly
# surfaces reposts/collabs by OTHER tracked accounts. Missing that let a
# 2026-08-27 run re-fetch 9 reels already held under their true owner's
# directory -- gallery-dl filed them correctly there (by the post's real
# `username`, not the scraped profile), so nothing was lost, but it spent
# 9 avoidable Instagram requests to find that out. All of this is local
# requests to our own viewer, never to instagram.com, so checking every
# profile costs nothing on the budget that actually matters.
profiles = index.profiles()
have: set[str] = set()
for profile in profiles:
have.update(code for code, _ in gdl.index_existing(index.listing(profile)))
print(f"archive already holds {len(have)} shortcode(s) across "
f"{len(profiles)} profile(s)", file=sys.stderr)
print(f"scraping https://www.instagram.com/{args.profile}/reels/ ...",
file=sys.stderr)
codes = scrape_reels(args.profile, cdp_port=args.cdp_port,
scroll_pause=tuple(args.scroll_pause),
max_idle_rounds=args.max_idle_rounds,
max_scrolls=args.max_scrolls,
log=lambda msg: print(msg, file=sys.stderr))
new_codes = [c for c in codes if c not in have]
print(f"found {len(codes)} reel(s) on the page, {len(codes) - len(new_codes)} "
f"already archived, {len(new_codes)} new", file=sys.stderr)
urls = [f"https://www.instagram.com/{args.profile}/reel/{c}/"
for c in new_codes]
text = "\n".join(urls)
if args.out:
args.out.write_text(text + ("\n" if text else ""))
print(f"wrote {len(urls)} URL(s) to {args.out}", file=sys.stderr)
else:
if text:
print(text)
return 0
if __name__ == "__main__":
sys.exit(main())
-114
View File
@@ -1,114 +0,0 @@
#!/bin/bash
# The full reels pipeline for one profile: scrape by scrolling the real page,
# then fetch and publish whatever's new. See TOOLING.md ("Reels: the API is
# blocked, scrape by scrolling instead") for why this exists at all -- the
# dedicated reels API is blocked, this drives the actual signed-in browser
# session instead, and is slower by design.
#
# Usage: ./reels-sync.sh <username-or-profile-url>
# ./reels-sync.sh zindoriyam
# ./reels-sync.sh https://www.instagram.com/zindoriyam/
#
# Deliberately NOT wired into gdl-cron.sh or the timers -- see TOOLING.md.
# Exits non-zero if either step does. Everything is logged.
set -eu
RAW="${1:?usage: reels-sync.sh <username-or-profile-url>}"
GDL_HOME="${GDL_HOME:-$HOME/gdl}"
GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
# Same hand-paced pacing gdl-cron.sh uses -- see its comment for why. Override
# per-run with GDL_SLEEP_REQUEST etc. if you ever need to, but raise them
# rather than lower them.
SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
SLEEP="${GDL_SLEEP:-5 10}"
RATE="${GDL_RATE:-500K}"
SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}"
MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}"
PATH="$HOME/.local/bin:$PATH"; export PATH
if [ ! -x "$GDL_PYTHON" ]; then
echo "reels-scrape.py needs gallery-dl's own pipx venv python (websocket-client" >&2
echo "was injected there, not into the system python): $GDL_PYTHON not found" >&2
exit 2
fi
# Same username-from-URL parsing gdl-sync.py already does for --urls-file,
# reused rather than re-implemented so the two never drift apart.
PROFILE=$("$GDL_PYTHON" -c "
import re, sys, importlib.util
from pathlib import Path
spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
gdl = importlib.util.module_from_spec(spec)
sys.modules['gdl_sync'] = gdl
spec.loader.exec_module(gdl)
raw = '$RAW'
m = gdl.RE_PROFILE_URL.match(raw)
user = m.group('user') if m else raw.strip('/')
if not user or '/' in user or ' ' in user:
print(f'cannot read a username from {raw!r}', file=sys.stderr)
sys.exit(1)
print(user)
")
mkdir -p "$GDL_HOME/logs"
LOG="$GDL_HOME/logs/reels-$PROFILE-$(date +%Y%m%d-%H%M%S)-$$.log"
URLS_FILE="$GDL_HOME/$PROFILE-reels.txt"
STAGING="$GDL_HOME/staging-reels-$PROFILE"
echo "=== reels-sync $PROFILE $(date -Is) ===" | tee -a "$LOG"
# The exit status has to survive the pipe into tee -- see gdl-cron.sh's
# comment on PIPESTATUS for why this needs to be bash, not sh.
set +e
"$GDL_PYTHON" "$GDL_HOME/reels-scrape.py" \
--profile "$PROFILE" \
--index "$INDEX" \
--scroll-pause $SCROLL_PAUSE \
--max-idle-rounds "$MAX_IDLE_ROUNDS" \
--out "$URLS_FILE" 2>&1 | tee -a "$LOG"
scrape_status=${PIPESTATUS[0]}
set -e
if [ "$scrape_status" -ne 0 ]; then
echo "=== exit $scrape_status (scrape failed) at $(date -Is) ===" | tee -a "$LOG"
exit "$scrape_status"
fi
if [ ! -s "$URLS_FILE" ]; then
echo "no new reels for $PROFILE; nothing to fetch" | tee -a "$LOG"
echo "=== exit 0 at $(date -Is) ===" | tee -a "$LOG"
exit 0
fi
# Staging is wiped every run on purpose -- same reasoning as gdl-cron.sh: what
# we already hold is decided by the archive dedupe in reels-scrape.py, not by
# what happens to be sitting in staging.
rm -rf "$STAGING"
set +e
# shellcheck disable=SC2086
"$GDL_HOME/gdl-sync.py" \
--publish "$PUBLISH" \
--staging "$STAGING" \
--post-urls-file "$URLS_FILE" \
--sleep-request $SLEEP_REQUEST \
--sleep $SLEEP \
--rate "$RATE" \
--execute 2>&1 | tee -a "$LOG"
status=${PIPESTATUS[0]}
set -e
echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
# Keep the log directory from growing without bound -- scoped to this
# script's own logs so it never touches gdl-cron.sh's rotation.
ls -1t "$GDL_HOME/logs" | grep '^reels-' | tail -n +30 | while read -r old; do
rm -f "$GDL_HOME/logs/$old"
done
exit "$status"
-16
View File
@@ -1,16 +0,0 @@
[Unit]
# One templated service for all three modes; the instance name (%i) is the
# mode: stories, full or sweep.
Description=Instagram archive sync (%i)
Documentation=file:%h/gdl/TOOLING.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=%h/gdl/gdl-cron.sh %i
# A sync has no deadline and the pacing is deliberately slow; a full sweep can
# run for hours. Never let systemd kill one midway -- a half-published run is
# the one state the publish step is designed to avoid.
TimeoutStartSec=infinity
Nice=10
-18
View File
@@ -1,18 +0,0 @@
[Unit]
Description=Full Instagram archive sweep, no abort (~420 requests)
# The only run that enumerates every profile to the end, and so the only one
# that notices carousels edited after we archived them (test case 15).
#
# It is also by far the most expensive thing here: ~420 requests to
# instagram.com, the same order as the run that preceded the 2026-08-21
# scraping warning, spent to catch a handful of retroactively edited posts.
# Consider running it by hand when you mean to, rather than on a timer.
[Timer]
# Month names are not valid in OnCalendar's date field -- numeric only.
OnCalendar=*-01,04,07,10-07 04:00:00
RandomizedDelaySec=45m
Persistent=true
[Install]
WantedBy=timers.target
-10
View File
@@ -1,10 +0,0 @@
[Unit]
Description=Monthly Instagram profile sync (all surfaces, --abort 50)
[Timer]
OnCalendar=*-*-03 04:00:00
RandomizedDelaySec=45m
Persistent=true
[Install]
WantedBy=timers.target
-15
View File
@@ -1,15 +0,0 @@
[Unit]
Description=Daily Instagram stories sync
# Stories expire in 24h and cannot be backfilled. This is the only timer whose
# missed run means content is gone for good, which is what Persistent= is for.
[Timer]
OnCalendar=*-*-* 09:00:00
# Not a fixed time: a job firing at exactly 09:00 every day is obviously a
# machine, and the whole safety model is about not looking like one.
RandomizedDelaySec=45m
# Catch up after the host was asleep or off. cron would silently skip.
Persistent=true
[Install]
WantedBy=timers.target
-72
View File
@@ -209,60 +209,6 @@ class Publishing(unittest.TestCase):
self.assertIn("--dry-run", cmd)
class PostUrl(unittest.TestCase):
"""--post-url: a one-off fetch outside the tracked profile list, whose
owning account is only known mid-extraction -- same reasoning as
highlights, so it must be exempted from the same forced-destination rule."""
def test_config_keys_a_username_directory(self):
config = gdl.build_config("1M", [6.0, 10.0], [3.0, 6.0])
for kind in ("post", "reel"):
self.assertEqual(
config["extractor"]["instagram"][kind]["directory"],
["{username}"])
def test_destination_is_not_forced_like_posts_and_reels(self):
staging = Path("/stage")
for subcategory in ("post", "reel"):
src = gdl.Source(subcategory, "https://www.instagram.com/p/ABC/",
"", subcategory)
cmd = gdl.gdl_command(src, staging, Path("/cfg.json"), "chrome:x",
None)
self.assertIn(str(staging), cmd)
self.assertNotIn(str(staging / subcategory), cmd)
class MetadataFields(unittest.TestCase):
"""Extra fields captured from the raw API response, verified against two
saved real examples on 2026-09-01 (see docs/gallery-dl.md)."""
def test_post_level_json_captures_coauthors(self):
config = gdl.build_config("1M", [6.0, 10.0], [3.0, 6.0])
pps = config["extractor"]["instagram"]["posts"]["postprocessors"]
post_json = next(pp for pp in pps if pp.get("filename", "").endswith(".json"))
self.assertIn("coauthors", post_json["include"])
def test_per_item_dimensions_and_tags_get_their_own_sidecar(self):
config = gdl.build_config("1M", [6.0, 10.0], [3.0, 6.0])
pps = config["extractor"]["instagram"]["posts"]["postprocessors"]
# No "filename" of its own -- unlike the other two, which are keyed
# by POST_STEM -- because it must vary per carousel item, not per post.
media_pp = next(pp for pp in pps if "filename" not in pp)
for field in ("width", "height", "width_original", "height_original",
"tagged_users", "shortcode", "num"):
self.assertIn(field, media_pp["include"])
# `owner` is a full user object (profile pic URLs, privacy flags) for
# whoever posted that item -- deliberately excluded, same reasoning
# as `audio_user` on the post-level json.
self.assertNotIn("owner", media_pp["include"])
def test_reels_get_the_same_capture_as_posts(self):
config = gdl.build_config("1M", [6.0, 10.0], [3.0, 6.0])
posts_pps = config["extractor"]["instagram"]["posts"]["postprocessors"]
reels_pps = config["extractor"]["instagram"]["reels"]["postprocessors"]
self.assertEqual(posts_pps, reels_pps)
class UrlsFile(unittest.TestCase):
def test_reads_every_form_a_person_might_paste(self):
with tempfile.TemporaryDirectory() as d:
@@ -281,23 +227,5 @@ class UrlsFile(unittest.TestCase):
self.assertEqual(gdl.read_urls_file(p), ["a", "b", "c", "d", "e"])
class PostUrlsFile(unittest.TestCase):
"""The format reels-scrape.py writes: whole URLs, not usernames."""
def test_skips_comments_blanks_and_duplicates(self):
with tempfile.TemporaryDirectory() as d:
p = Path(d) / "reels.txt"
p.write_text(
"# scraped 2026-08-27\n"
"https://www.instagram.com/u/reel/AAA/\n"
"\n"
"https://www.instagram.com/u/reel/BBB/\n"
"https://www.instagram.com/u/reel/AAA/\n") # duplicate
self.assertEqual(gdl.read_post_urls_file(p), [
"https://www.instagram.com/u/reel/AAA/",
"https://www.instagram.com/u/reel/BBB/",
])
if __name__ == "__main__":
unittest.main(verbosity=2)
-40
View File
@@ -1,40 +0,0 @@
#!/usr/bin/env python3
"""
Tests for the pure parts of reels-scrape.py -- everything except the actual
CDP session, which needs a live signed-in Chrome and is exercised by hand.
python3 -m unittest discover -s scripts -p 'test_*.py'
"""
import importlib.util
import sys
import unittest
from pathlib import Path
_spec = importlib.util.spec_from_file_location(
"reels_scrape", Path(__file__).with_name("reels-scrape.py"))
scrape = importlib.util.module_from_spec(_spec)
sys.modules["reels_scrape"] = scrape
_spec.loader.exec_module(scrape)
class ExtractShortcodes(unittest.TestCase):
def test_reads_relative_and_absolute_hrefs(self):
hrefs = [
"/someuser/reel/ABC123/",
"https://www.instagram.com/someuser/reel/DEF456/",
"/reel/GHI789/?img_index=1",
]
self.assertEqual(scrape.extract_shortcodes(hrefs),
["ABC123", "DEF456", "GHI789"])
def test_ignores_non_reel_links(self):
hrefs = ["/someuser/", "/someuser/p/ABC123/", "/explore/tags/foo/"]
self.assertEqual(scrape.extract_shortcodes(hrefs), [])
def test_empty_input(self):
self.assertEqual(scrape.extract_shortcodes([]), [])
if __name__ == "__main__":
unittest.main(verbosity=2)
+28 -28
View File
@@ -3,39 +3,39 @@ import { classifyDirectory, groupArchiveDirectories } from './archive-grouping';
describe('classifyDirectory', () => {
it('treats a bare profile directory as the base', () => {
expect(classifyDirectory('4utumn07')).toEqual({
owner: '4utumn07',
source: { kind: 'posts', dir: '4utumn07' },
expect(classifyDirectory('0ct0ber19')).toEqual({
owner: '0ct0ber19',
source: { kind: 'posts', dir: '0ct0ber19' },
});
});
it('recognises a reels sidecar', () => {
expect(classifyDirectory('4utumn07 - reels')).toEqual({
owner: '4utumn07',
source: { kind: 'reels', dir: '4utumn07 - reels' },
expect(classifyDirectory('0ct0ber19 - reels')).toEqual({
owner: '0ct0ber19',
source: { kind: 'reels', dir: '0ct0ber19 - reels' },
});
});
it('recognises a stories sidecar', () => {
expect(classifyDirectory('story - dawn_petal')).toEqual({
owner: 'dawn_petal',
source: { kind: 'stories', dir: 'story - dawn_petal' },
expect(classifyDirectory('story - cher_ryppo')).toEqual({
owner: 'cher_ryppo',
source: { kind: 'stories', dir: 'story - cher_ryppo' },
});
});
it('splits highlight owner from title', () => {
const { owner, source } = classifyDirectory('story highlights - 4utumn07 - Sunstory');
expect(owner).toBe('4utumn07');
const { owner, source } = classifyDirectory('story highlights - 0ct0ber19 - Heestory');
expect(owner).toBe('0ct0ber19');
expect(source.kind).toBe('highlight');
expect(source.title).toBe('Sunstory');
expect(source.title).toBe('Heestory');
});
it.each([
['story highlights - theoldlyricmuseinsta - 💙1999-2005 era', 'theoldlyricmuseinsta', '💙1999-2005 era'],
['story highlights - member_theworld - [Bracket]', 'member_theworld', '[Bracket]'],
['story highlights - official_band - Tour Schedule', 'official_band', 'Tour Schedule'],
['story highlights - 4utumn07 - Sketching', '4utumn07', 'Sketching'],
['story highlights - official_band - A.B.C', 'official_band', 'A.B.C'],
['story highlights - theoldtaylorswiftinsta - 💙2014-1989 era', 'theoldtaylorswiftinsta', '💙2014-1989 era'],
['story highlights - heejin_theworld - [Dall]', 'heejin_theworld', '[Dall]'],
['story highlights - official_artms - Cosmo Schedule', 'official_artms', 'Cosmo Schedule'],
['story highlights - 0ct0ber19 - Drawheeing', '0ct0ber19', 'Drawheeing'],
['story highlights - official_artms - G.C.I', 'official_artms', 'G.C.I'],
])('handles real-world title %s', (dir, owner, title) => {
const result = classifyDirectory(dir);
expect(result.owner).toBe(owner);
@@ -57,25 +57,25 @@ describe('classifyDirectory', () => {
describe('groupArchiveDirectories', () => {
const dirs = [
'4utumn07',
'4utumn07 - reels',
'story - 4utumn07',
'story highlights - 4utumn07 - Sunstory',
'story highlights - 4utumn07 - Sketching',
'kestrelsings',
'0ct0ber19',
'0ct0ber19 - reels',
'story - 0ct0ber19',
'story highlights - 0ct0ber19 - Heestory',
'story highlights - 0ct0ber19 - Drawheeing',
'carlyraejepsen',
];
it('folds sidecars into their base profile', () => {
const groups = groupArchiveDirectories(dirs);
expect([...groups.keys()].sort()).toEqual(['4utumn07', 'kestrelsings']);
expect(groups.get('4utumn07')).toHaveLength(5);
expect(groups.get('kestrelsings')).toHaveLength(1);
expect([...groups.keys()].sort()).toEqual(['0ct0ber19', 'carlyraejepsen']);
expect(groups.get('0ct0ber19')).toHaveLength(5);
expect(groups.get('carlyraejepsen')).toHaveLength(1);
});
it('orders sources posts, reels, stories, then highlights by title', () => {
const sources = groupArchiveDirectories(dirs).get('4utumn07')!;
const sources = groupArchiveDirectories(dirs).get('0ct0ber19')!;
expect(sources.map(s => s.kind)).toEqual(['posts', 'reels', 'stories', 'highlight', 'highlight']);
expect(sources.slice(3).map(s => s.title)).toEqual(['Sketching', 'Sunstory']);
expect(sources.slice(3).map(s => s.title)).toEqual(['Drawheeing', 'Heestory']);
});
it('still groups a sidecar whose base profile is missing', () => {
+4 -4
View File
@@ -18,10 +18,10 @@ export interface ArchiveSource {
/**
* Sidecar directories sit next to the profile directory they belong to:
*
* 4utumn07 -> posts (base)
* 4utumn07 - reels -> reels
* story - 4utumn07 -> stories
* story highlights - 4utumn07 - Sunstory -> highlight "Sunstory"
* 0ct0ber19 -> posts (base)
* 0ct0ber19 - reels -> reels
* story - 0ct0ber19 -> stories
* story highlights - 0ct0ber19 - Heestory -> highlight "Heestory"
*
* Instagram usernames cannot contain spaces, so matching the username as a
* run of non-space characters reliably separates it from a highlight title
+5 -5
View File
@@ -14,11 +14,11 @@ describe('isSystemDirectory', () => {
});
it.each([
'4utumn07',
'4utumn07 - reels',
'story - dawn_petal',
'story highlights - official_band - A.B.C',
'story highlights - theoldlyricmuseinsta - 💙1999-2005 era',
'0ct0ber19',
'0ct0ber19 - reels',
'story - cher_ryppo',
'story highlights - official_artms - G.C.I',
'story highlights - theoldtaylorswiftinsta - 💙2014-1989 era',
'Heejin_Bubble heejinmedia',
'gallery-dl',
'posts',
+26 -26
View File
@@ -3,10 +3,10 @@ import { canonicalItemId, parseArchiveFilename, scopedPostId } from './archive-p
describe('parseArchiveFilename — Instagram export format', () => {
it('parses a single-image post', () => {
expect(parseArchiveFilename('2023-04-19_4utumn07 - CrORBIcJJbM.mp4')).toEqual({
expect(parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM.mp4')).toEqual({
postId: 'CrORBIcJJbM',
date: '2023-04-19',
username: '4utumn07',
username: '0ct0ber19',
index: 1,
ext: 'mp4',
isStory: false,
@@ -15,7 +15,7 @@ describe('parseArchiveFilename — Instagram export format', () => {
});
it('parses a carousel slide index', () => {
const parsed = parseArchiveFilename('2023-04-12_4utumn07 - Cq8LrxSJAJE - 3.jpg');
const parsed = parseArchiveFilename('2023-04-12_0ct0ber19 - Cq8LrxSJAJE - 3.jpg');
expect(parsed).toMatchObject({ postId: 'Cq8LrxSJAJE', index: 3, ext: 'jpg' });
});
@@ -27,7 +27,7 @@ describe('parseArchiveFilename — Instagram export format', () => {
});
it('parses caption sidecar files', () => {
expect(parseArchiveFilename('2023-04-12_4utumn07 - Cq8LrxSJAJE.txt')).toMatchObject({
expect(parseArchiveFilename('2023-04-12_0ct0ber19 - Cq8LrxSJAJE.txt')).toMatchObject({
postId: 'Cq8LrxSJAJE',
ext: 'txt',
});
@@ -39,8 +39,8 @@ describe('parseArchiveFilename — Instagram export format', () => {
it('parses the story sidecar layout (date_user - N - shortcode)', () => {
// Files in `story - <user>` carry a per-day ordinal before the shortcode.
const parsed = parseArchiveFilename('2025-10-26_4utumn07 - 2 - DQRuDx9iW5Q.jpg', 'stories');
expect(parsed).toMatchObject({ date: '2025-10-26', username: '4utumn07', ext: 'jpg' });
const parsed = parseArchiveFilename('2025-10-26_0ct0ber19 - 2 - DQRuDx9iW5Q.jpg', 'stories');
expect(parsed).toMatchObject({ date: '2025-10-26', username: '0ct0ber19', ext: 'jpg' });
expect(parsed!.postId).toContain('DQRuDx9iW5Q');
});
@@ -71,9 +71,9 @@ describe('parseArchiveFilename — Instaloader format', () => {
describe('parseArchiveFilename — story highlights', () => {
it('parses the dateless highlight layout', () => {
expect(parseArchiveFilename('4utumn07 - C5dQPEYpd9W.mp4', 'highlight')).toMatchObject({
expect(parseArchiveFilename('0ct0ber19 - C5dQPEYpd9W.mp4', 'highlight')).toMatchObject({
postId: 'C5dQPEYpd9W',
username: '4utumn07',
username: '0ct0ber19',
ext: 'mp4',
isStory: false,
});
@@ -95,7 +95,7 @@ describe('parseArchiveFilename — story highlights', () => {
});
describe('parseArchiveFilename — non-matching files', () => {
it.each(['4utumn07.jpg', 'profile_pic.jpg', 'README.md', 'no-separator.png'])(
it.each(['0ct0ber19.jpg', 'profile_pic.jpg', 'README.md', 'no-separator.png'])(
'returns null for %s',
name => expect(parseArchiveFilename(name)).toBeNull(),
);
@@ -107,8 +107,8 @@ describe('scopedPostId', () => {
});
it('namespaces sidecar ids by directory', () => {
expect(scopedPostId('C5dQ', 'highlight', 'story highlights - u - Sunstory'))
.toBe('story highlights - u - Sunstory/C5dQ');
expect(scopedPostId('C5dQ', 'highlight', 'story highlights - u - Heestory'))
.toBe('story highlights - u - Heestory/C5dQ');
});
it('keeps the same shortcode distinct across sources', () => {
@@ -125,8 +125,8 @@ describe('scopedPostId', () => {
*/
describe('gallery-dl / JDownloader naming interop', () => {
it('treats a single-media post the same with or without an index', () => {
const jd2 = parseArchiveFilename('2023-04-19_4utumn07 - CrORBIcJJbM.mp4')!;
const gdl = parseArchiveFilename('2023-04-19_4utumn07 - CrORBIcJJbM - 1.mp4')!;
const jd2 = parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM.mp4')!;
const gdl = parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM - 1.mp4')!;
expect(jd2.postId).toBe(gdl.postId);
expect(jd2.index).toBe(gdl.index);
expect(jd2.index).toBe(1);
@@ -136,21 +136,21 @@ describe('gallery-dl / JDownloader naming interop', () => {
// JD2 pads to the width of the media count (10+ items -> "01"), and
// gallery-dl's count can be one higher, so the same post may be padded
// by one tool and not the other.
expect(parseArchiveFilename('2024-04-17_4utumn07 - C53YPQzp7Wj - 09.jpg')!.index).toBe(9);
expect(parseArchiveFilename('2024-04-17_4utumn07 - C53YPQzp7Wj - 9.jpg')!.index).toBe(9);
expect(parseArchiveFilename('2023-11-03_4utumn07 - CzM8Uf6B6H_ - 01.jpg')!.index).toBe(1);
expect(parseArchiveFilename('2024-04-17_0ct0ber19 - C53YPQzp7Wj - 09.jpg')!.index).toBe(9);
expect(parseArchiveFilename('2024-04-17_0ct0ber19 - C53YPQzp7Wj - 9.jpg')!.index).toBe(9);
expect(parseArchiveFilename('2023-11-03_0ct0ber19 - CzM8Uf6B6H_ - 01.jpg')!.index).toBe(1);
});
it('reads a gallery-dl story name, which carries a per-item shortcode', () => {
const p = parseArchiveFilename('2026-08-16_official_band - DcF9OyhBJ1H.jpg', 'stories')!;
const p = parseArchiveFilename('2026-08-16_official_artms - DcF9OyhBJ1H.jpg', 'stories')!;
expect(p.postId).toBe('DcF9OyhBJ1H');
expect(p.date).toBe('2026-08-16');
});
it('gives a dated highlight a real date instead of the mtime fallback', () => {
const mtime = Date.parse('2026-08-17T00:00:00Z');
const undated = parseArchiveFilename('4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
const dated = parseArchiveFilename('2024-08-04_4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
const undated = parseArchiveFilename('0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
const dated = parseArchiveFilename('2024-08-04_0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
// Same item either way, so re-fetching cannot split it into two posts.
expect(dated.postId).toBe(undated.postId);
expect(undated.date).toBe('2026-08-17');
@@ -167,13 +167,13 @@ describe('dateFromMtime', () => {
const mtime = Date.parse('2026-08-17T00:00:00Z');
it('flags an undated highlight name as mtime-dated', () => {
const p = parseArchiveFilename('4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
const p = parseArchiveFilename('0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
expect(p.date).toBe('2026-08-17');
expect(p.dateFromMtime).toBe(true);
});
it('does not flag a highlight that carries its own date', () => {
const p = parseArchiveFilename('2024-08-04_4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
const p = parseArchiveFilename('2024-08-04_0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
expect(p.date).toBe('2024-08-04');
expect(p.dateFromMtime).toBe(false);
});
@@ -184,7 +184,7 @@ describe('dateFromMtime', () => {
});
it('leaves the date empty rather than guessing when no mtime is given', () => {
const p = parseArchiveFilename('4utumn07 - C-IImhvpFuk.jpg', 'highlight')!;
const p = parseArchiveFilename('0ct0ber19 - C-IImhvpFuk.jpg', 'highlight')!;
expect(p.date).toBe('');
expect(p.dateFromMtime).toBe(false);
});
@@ -196,9 +196,9 @@ describe('dateFromMtime', () => {
*/
describe('canonicalItemId', () => {
it('collapses the two highlight naming conventions onto one id', () => {
const dir = 'story highlights - 4utumn07 - Sunstory';
const undated = parseArchiveFilename('4utumn07 - C5dQPEYpd9W.mp4', 'highlight', 1)!;
const dated = parseArchiveFilename('2024-04-07_4utumn07 - 01 - C5dQPEYpd9W.mp4', 'highlight')!;
const dir = 'story highlights - 0ct0ber19 - Heestory';
const undated = parseArchiveFilename('0ct0ber19 - C5dQPEYpd9W.mp4', 'highlight', 1)!;
const dated = parseArchiveFilename('2024-04-07_0ct0ber19 - 01 - C5dQPEYpd9W.mp4', 'highlight')!;
expect(scopedPostId(dated.postId, 'highlight', dir))
.toBe(scopedPostId(undated.postId, 'highlight', dir));
});
@@ -216,7 +216,7 @@ describe('canonicalItemId', () => {
});
it('leaves a shortcode that merely starts with digits alone', () => {
expect(canonicalItemId('4utumn07')).toBe('4utumn07');
expect(canonicalItemId('0ct0ber19')).toBe('0ct0ber19');
expect(canonicalItemId('C5dQPEYpd9W')).toBe('C5dQPEYpd9W');
expect(canonicalItemId('12345')).toBe('12345');
});
+2 -2
View File
@@ -7,13 +7,13 @@ import {
const REEL: GalleryDlSidecar = {
post_shortcode: 'Db-lNCoib9m', post_id: '3962768346034323302', type: 'reel',
date: '2026-08-13 11:00:44', post_date: '2026-08-13 11:00:44',
username: 'official_band', fullname: 'Official ARTMS',
username: 'official_artms', fullname: 'Official ARTMS',
description: 'Dancing in the spotlight', count: 1, likes: 22914,
};
const FEED_VIDEO: GalleryDlSidecar = { ...REEL, post_shortcode: 'DbdG9L9jU4m', type: 'post', count: 2 };
const HIGHLIGHT: GalleryDlSidecar = {
post_shortcode: 'BATVdRZi_3', post_id: '18099435932626935', type: 'highlight',
date: '2026-08-08 16:22:09', username: 'official_band', count: 154,
date: '2026-08-08 16:22:09', username: 'official_artms', count: 154,
};
describe('isGalleryDlSidecar', () => {
+1 -1
View File
@@ -16,7 +16,7 @@ import { Tab } from './routing';
/**
* The shortcode shared by every copy of a post, regardless of which source
* directory it came from. Sidecar ids are directory-scoped
* (`4utumn07 - reels/Cq8LrxSJAJE`); the trailing segment is the shortcode.
* (`0ct0ber19 - reels/Cq8LrxSJAJE`); the trailing segment is the shortcode.
*/
const shortcode = (post: Post): string => post.id.split('/').pop() ?? post.id;
+16 -16
View File
@@ -12,21 +12,21 @@ describe('parseRoute', () => {
});
it('reads a profile', () => {
expect(parseRoute('/4utumn07/')).toEqual({ archive: '4utumn07', tab: 'posts', post: null });
expect(parseRoute('/0ct0ber19/')).toEqual({ archive: '0ct0ber19', tab: 'posts', post: null });
});
it('reads a profile without a trailing slash', () => {
expect(parseRoute('/4utumn07')).toEqual({ archive: '4utumn07', tab: 'posts', post: null });
expect(parseRoute('/0ct0ber19')).toEqual({ archive: '0ct0ber19', tab: 'posts', post: null });
});
it('reads a tab', () => {
expect(parseRoute('/4utumn07/reels/').tab).toBe('reels');
expect(parseRoute('/4utumn07/saved/').tab).toBe('saved');
expect(parseRoute('/0ct0ber19/reels/').tab).toBe('reels');
expect(parseRoute('/0ct0ber19/saved/').tab).toBe('saved');
});
it('reads a post in Instagram form', () => {
expect(parseRoute('/4utumn07/p/Db5tIoRCcvm/')).toEqual({
archive: '4utumn07', tab: 'posts', post: 'Db5tIoRCcvm',
expect(parseRoute('/0ct0ber19/p/Db5tIoRCcvm/')).toEqual({
archive: '0ct0ber19', tab: 'posts', post: 'Db5tIoRCcvm',
});
});
@@ -41,8 +41,8 @@ describe('parseRoute', () => {
});
it('still understands the legacy query form', () => {
expect(parseRoute('/', '?a=4utumn07&t=reels&p=ABC')).toEqual({
archive: '4utumn07', tab: 'reels', post: 'ABC',
expect(parseRoute('/', '?a=0ct0ber19&t=reels&p=ABC')).toEqual({
archive: '0ct0ber19', tab: 'reels', post: 'ABC',
});
});
@@ -54,9 +54,9 @@ describe('parseRoute', () => {
describe('buildPath', () => {
it.each([
[{ archive: null, tab: 'posts', post: null }, '/'],
[{ archive: '4utumn07', tab: 'posts', post: null }, '/4utumn07/'],
[{ archive: '4utumn07', tab: 'reels', post: null }, '/4utumn07/reels/'],
[{ archive: '4utumn07', tab: 'posts', post: 'Db5tIoRCcvm' }, '/4utumn07/p/Db5tIoRCcvm/'],
[{ archive: '0ct0ber19', tab: 'posts', post: null }, '/0ct0ber19/'],
[{ archive: '0ct0ber19', tab: 'reels', post: null }, '/0ct0ber19/reels/'],
[{ archive: '0ct0ber19', tab: 'posts', post: 'Db5tIoRCcvm' }, '/0ct0ber19/p/Db5tIoRCcvm/'],
] as const)('builds %j', (route, expected) => {
expect(buildPath(route as any)).toBe(expected);
});
@@ -71,8 +71,8 @@ describe('buildPath', () => {
it('round-trips through parseRoute', () => {
for (const route of [
{ archive: '4utumn07', tab: 'posts' as const, post: null },
{ archive: '4utumn07', tab: 'reels' as const, post: null },
{ archive: '0ct0ber19', tab: 'posts' as const, post: null },
{ archive: '0ct0ber19', tab: 'reels' as const, post: null },
{ archive: 'Heejin_Bubble heejinmedia', tab: 'posts' as const, post: null },
]) {
expect(parseRoute(buildPath(route))).toEqual(route);
@@ -86,12 +86,12 @@ describe('postSlug / findPostBySlug', () => {
});
it('strips the sidecar directory from the slug', () => {
expect(postSlug(post('story highlights - u - Sunstory/C5dQPEYpd9W'))).toBe('C5dQPEYpd9W');
expect(postSlug(post('story highlights - u - Heestory/C5dQPEYpd9W'))).toBe('C5dQPEYpd9W');
});
it('resolves a slug back to its post', () => {
const posts = [post('AAA'), post('4utumn07 - reels/BBB', 'reels')];
expect(findPostBySlug(posts, 'BBB')?.id).toBe('4utumn07 - reels/BBB');
const posts = [post('AAA'), post('0ct0ber19 - reels/BBB', 'reels')];
expect(findPostBySlug(posts, 'BBB')?.id).toBe('0ct0ber19 - reels/BBB');
expect(findPostBySlug(posts, 'AAA')?.id).toBe('AAA');
});