Compare commits
1
Commits
84c573b3ed
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
  ergosteurandClaude Sonnet 5
|
600949e475 |
@@ -1,6 +0,0 @@
|
||||
https://www.instagram.com/0ct0ber19/
|
||||
https://www.instagram.com/kimxxlip/
|
||||
https://www.instagram.com/withaseul/
|
||||
https://www.instagram.com/cher_ryppo/
|
||||
https://www.instagram.com/zindoriyam/
|
||||
https://www.instagram.com/official_artms/
|
||||
@@ -1,605 +0,0 @@
|
||||
# gallery-dl — a CLI replacement for JDownloader2
|
||||
|
||||
Status: **in production.** All six ARTMS profiles are synced with
|
||||
`scripts/gdl-sync.py`; JD2 is no longer used for them.
|
||||
|
||||
Everything below was measured against the live site and the real archive on
|
||||
2026-08-16 and 2026-08-20, not inferred from documentation.
|
||||
|
||||
## Why gallery-dl and not a hand-rolled script
|
||||
|
||||
The hard parts of fetching Instagram are pagination, cookie handling, CDN URL
|
||||
expiry and resumption. gallery-dl already has all of them, plus extractors that
|
||||
map 1:1 onto our sidecar directory layout (`posts`, `reels`, `stories`,
|
||||
`highlights`). Rolling our own would mean reimplementing the ban-sensitive part
|
||||
by hand.
|
||||
|
||||
## The account was suspended on 2026-08-17 — read this first
|
||||
|
||||
The account used for all of the below was suspended the same day this tooling
|
||||
was built, for "activity that doesn't follow our Community Standards on spam".
|
||||
The fetching was not the expensive part. **Verification was.**
|
||||
|
||||
**It was restored, and synced normally again on 2026-08-20** — a full run
|
||||
across all six profiles with 0 failures and 0 CDN 429s. That is not evidence
|
||||
the limits were imagined; it is one data point on a restored account that has
|
||||
been treated carefully since. Everything below still applies, and the budget is
|
||||
still per session rather than per command.
|
||||
|
||||
What was actually spent against `instagram.com` in a few hours, from one
|
||||
session and one IP:
|
||||
|
||||
| activity | rough requests | downloaded |
|
||||
|---|---:|---|
|
||||
| enumerating a profile grid by scrolling it in an automated browser | ~18 pages | nothing |
|
||||
| the same profile again, after a bug in the scraping selector | ~18 pages | nothing |
|
||||
| a Reels tab enumerated the same way | ~9 pages | nothing |
|
||||
| full `-j` metadata dumps of one profile, twice | ~16 pages | nothing |
|
||||
| `--simulate` runs over the same profile, three times | ~24 pages | nothing |
|
||||
| single-post `/p/<code>/` fetches while testing filename formats | ~8 | a handful |
|
||||
| an aborted sync that re-ran every listing pass before dying | ~40 pages | ~270 MB |
|
||||
| the real sync, 24 sources across 6 profiles | ~150 pages | 2.2 GB |
|
||||
|
||||
The two rows that actually mattered to the archive are the last one and part of
|
||||
the second-to-last. **Everything above them produced no files at all**, and
|
||||
together they were a comparable number of requests.
|
||||
|
||||
The warnings arrived in this order and were each rationalised:
|
||||
|
||||
1. `429 Too Many Requests` from `scontent-*.cdninstagram.com`, losing two
|
||||
videos. Treated as a pacing problem — pacing was lowered and the run
|
||||
continued.
|
||||
2. `400 Bad Request` from `/api/v1/highlights/<id>/highlights_tray/`, on an
|
||||
endpoint that had worked hours earlier. Correctly read as a possible block;
|
||||
requests stopped.
|
||||
3. Suspension.
|
||||
|
||||
**Treat the first CDN 429 as a stop signal for the session, not a tuning
|
||||
parameter.** It is the tolerant surface complaining; if that surface is
|
||||
complaining, the rate-limited one has been unhappy for a while.
|
||||
|
||||
### Rules that follow from this
|
||||
|
||||
- **Count verification requests against the same budget as fetching.** A
|
||||
`--simulate`, a `-j` dump and a browser scroll all hit `instagram.com` and
|
||||
download nothing. Being read-only does not make them free; it makes them
|
||||
invisible, which is worse.
|
||||
- **Never enumerate the live site with an automated browser.** Scrolling a
|
||||
214-post grid is ~18 paginated GraphQL loads at machine speed with no dwell
|
||||
time between them. It is the most obviously non-human thing in this whole
|
||||
document, and it was done here twice on one profile.
|
||||
- **Verify against the archive, not against Instagram.** Every naming, dating
|
||||
and classification question answered in this file could have been answered
|
||||
from files already on disk plus a single listing pass.
|
||||
- **`probe_live` is not cached, so every restart re-enumerates everything.**
|
||||
The aborted run cost a full duplicate set of listing passes for five
|
||||
profiles. Cache probe output to disk before running anything twice.
|
||||
- **Budget per session, not per command.** Nothing in the tooling knows what
|
||||
the last command spent.
|
||||
|
||||
### For a replacement account
|
||||
|
||||
- Let it exist and be used normally for a while before pointing any tool at it.
|
||||
- Keep the cookie on one machine and one public IP, as before.
|
||||
- Start with a single small profile and stop for the day afterwards.
|
||||
- Prefer Instagram's own "Download a copy" export where possible: it is
|
||||
first-party, costs no scraping requests, and carries the metadata this whole
|
||||
document works around not having.
|
||||
|
||||
## The safety model — read this before changing any option
|
||||
|
||||
The ban vector is **requests to `instagram.com`**, not bandwidth. See
|
||||
`docs/jdownloader.md` for the history; Instaloader got this account banned by
|
||||
asking `instagram.com` a question *per post*.
|
||||
|
||||
gallery-dl has two API backends and the difference is exactly that vector:
|
||||
|
||||
```python
|
||||
if self.config("api") == "graphql":
|
||||
self.api = InstagramGraphqlAPI(self) # per-post api.media() for every
|
||||
else: # video and every carousel
|
||||
self.api = InstagramRestAPI(self) # <- default, listing-only
|
||||
```
|
||||
|
||||
The REST backend paginates at `count: 30` (feed) / `page_size: 50` (clips), and
|
||||
those responses already carry `carousel_media`, `image_versions2`,
|
||||
`video_versions` and `product_type`. **No per-post request.** A 300-post
|
||||
profile costs roughly 10 requests to `instagram.com`.
|
||||
|
||||
Rules, in order of importance:
|
||||
|
||||
1. **`"api": "rest"` always.** Never `graphql`. This is the whole ballgame.
|
||||
2. **Never enable `metadata`-style options that trigger extra calls.** If a
|
||||
field is not already in the listing response, it is not worth a request.
|
||||
3. **Pace it.** `"sleep-request": [4.0, 7.0]` — a randomised gap, not a fixed
|
||||
one. Also `"sleep": [1.0, 3.0]` between downloads.
|
||||
4. **Cap the download rate** (`downloader.http.rate`) so the CDN side looks like
|
||||
a person, not a mirror.
|
||||
5. **Run from the same public IP as the browser the cookie came from.** At time
|
||||
of writing that is `mattellite` (`66.23.52.196`); the dev workstation is a
|
||||
*different* public IP and using the cookie from there is precisely what
|
||||
session-hijack detection looks for.
|
||||
6. **No programmatic login, ever.** gallery-dl's username/password path is
|
||||
disabled upstream anyway; use `--cookies-from-browser`.
|
||||
|
||||
Do not add proxy rotation, fingerprint spoofing or account rotation. Throttling
|
||||
and request-avoidance are welcome; evasion is not.
|
||||
|
||||
### Cookies
|
||||
|
||||
The logged-in Chrome on `mattellite` runs with a non-default profile:
|
||||
|
||||
```
|
||||
--user-data-dir=/home/matt/.config/google-chrome-devtools
|
||||
```
|
||||
|
||||
so the cookie flag is:
|
||||
|
||||
```
|
||||
--cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools"
|
||||
```
|
||||
|
||||
Plain `--cookies-from-browser chrome` fails with "Unable to find chrome cookies
|
||||
database" because it looks in `~/.config/google-chrome/`.
|
||||
|
||||
Anonymous access is **not** a viable fallback: it serves lower-resolution media,
|
||||
caps profile pagination at 12 posts, and returns `AuthRequired` for stories and
|
||||
highlights.
|
||||
|
||||
## Output format
|
||||
|
||||
The viewer's parser is the contract, not JD2's exact bytes. `EXPORT_RE` in
|
||||
`src/lib/archive-patterns.ts` accepts all of these, and normalises the index
|
||||
with `parseInt`, so **JD2 and gallery-dl naming interoperate**:
|
||||
|
||||
```
|
||||
"… - CrORBIcJJbM.mp4" -> postId=CrORBIcJJbM index=1
|
||||
"… - CrORBIcJJbM - 1.mp4" -> postId=CrORBIcJJbM index=1
|
||||
"… - C53YPQzp7Wj - 09.jpg" -> postId=C53YPQzp7Wj index=9
|
||||
```
|
||||
|
||||
That means zero-padding and the presence/absence of ` - N` on single-media posts
|
||||
are cosmetic. Don't spend effort forcing them.
|
||||
|
||||
### Directory layout
|
||||
|
||||
| kind | directory | note |
|
||||
|---|---|---|
|
||||
| posts | `<user>` | |
|
||||
| reels | `<user> - reels` | |
|
||||
| stories | `story - <user>` | |
|
||||
| highlights | `story highlights - <user> - <title>` | |
|
||||
|
||||
**Force the directory with `-D`; never use `{username}` for it.** A profile's
|
||||
reels tab returns *collab reels owned by other accounts* — `/0ct0ber19/reels/`
|
||||
served 6 reels owned by `official_artms` and 1 by `chuuo3o`. With
|
||||
`{username}` those would scatter into `official_artms - reels/`. JD2 got this
|
||||
right and the archive proves it: `chuuo3o` and `official_artms` filenames sit
|
||||
inside `0ct0ber19 - reels/`.
|
||||
|
||||
So: **owner in the filename, crawl scope in the directory.**
|
||||
|
||||
### Filenames
|
||||
|
||||
```jsonc
|
||||
"filename": {
|
||||
"sidecar_shortcode and count >= 10":
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num:02}.{extension}",
|
||||
"sidecar_shortcode":
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num}.{extension}",
|
||||
"":
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.{extension}"
|
||||
}
|
||||
```
|
||||
|
||||
`sidecar_shortcode` is set only when the post is a carousel, so it is the
|
||||
carousel discriminator. Conditions are evaluated in order, first match wins
|
||||
(`path.py:265`).
|
||||
|
||||
Stories and highlights use the per-item `{shortcode}`, not `{post_shortcode}`
|
||||
(which is the *reel's* id, shared by every item in it):
|
||||
|
||||
```
|
||||
"{date:Olocal/%Y-%m-%d}_{username} - {shortcode}.{extension}"
|
||||
```
|
||||
|
||||
`{date}` on a story/highlight file is the **per-item** `taken_at`
|
||||
(`instagram.py:337` prefers `item["taken_at"]`), verified on a 154-item
|
||||
highlight whose items carried distinct times while `post_date` stayed pinned to
|
||||
the reel. Highlights therefore gain real dates — today they fall back to
|
||||
directory mtime.
|
||||
|
||||
### The timezone is not UTC
|
||||
|
||||
JD2 stamped filenames in **desktop local time (US Eastern)**. Measured across
|
||||
212 comparable posts:
|
||||
|
||||
| model | mismatches |
|
||||
|---|---:|
|
||||
| UTC | 19 |
|
||||
| UTC−5 (EST) | 10 |
|
||||
| UTC−4 (EDT) | **0** |
|
||||
| America/New_York (DST-aware) | **0** |
|
||||
|
||||
`{date:Olocal/%Y-%m-%d}` uses the machine's local zone with per-timestamp DST
|
||||
awareness, which reproduces it — `mattellite` is `America/Toronto`, the same
|
||||
offsets. Note the **trailing `/` must be omitted**: `Olocal/%Y-%m-%d/` puts the
|
||||
separator into the strftime format and it sanitises to an underscore, giving
|
||||
`2026-08-15__0ct0ber19`.
|
||||
|
||||
If the sync ever moves to a host in another timezone, set an explicit
|
||||
`{date:O-4/…}` or the dates will silently shift for ~9% of posts.
|
||||
|
||||
### Caption sidecars
|
||||
|
||||
JD2 writes one `.txt` per post, named without the index, containing the caption
|
||||
with **no trailing newline**, and writes nothing when the caption is empty
|
||||
(measured: 197 of 217 posts, 86 of 86 reels, 0 of 10 stories, 0 of 16
|
||||
highlights). gallery-dl reproduces this exactly with the default
|
||||
`"empty": false`:
|
||||
|
||||
```jsonc
|
||||
{ "name": "metadata", "event": "post", "mode": "custom",
|
||||
"content-format": "{description}", "extension": "txt",
|
||||
"filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.txt" }
|
||||
```
|
||||
|
||||
`"event": "post"` is what makes it one file per post rather than per media file.
|
||||
|
||||
### Metadata sidecar (new — JD2 had no equivalent)
|
||||
|
||||
```jsonc
|
||||
{ "name": "metadata", "event": "post", "mode": "json",
|
||||
"filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.json",
|
||||
"include": ["post_shortcode","post_id","type","date","post_date","username",
|
||||
"fullname","owner_id","description","count","likes","post_url",
|
||||
"sidecar_shortcode"] }
|
||||
```
|
||||
|
||||
Use **`include`**, not `fields` — `fields` is for `mode: custom` and silently
|
||||
does nothing here, leaving `audio_user` blobs (including another user's profile
|
||||
picture URL) in the output.
|
||||
|
||||
The payoff is `type`, which is Instagram's own classification:
|
||||
|
||||
```json
|
||||
{ "post_shortcode": "DbdG9L9jU4m", "type": "post", "count": 2 } // feed video
|
||||
{ "post_shortcode": "Db-lNCoib9m", "type": "reel", "count": 1 } // real reel
|
||||
```
|
||||
|
||||
This is the `product_type: "clips"` signal, delivered free in the listing
|
||||
response. It is the authoritative answer to "is this a reel", and would let the
|
||||
viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
|
||||
"Scanner work" below.
|
||||
|
||||
**`type` is only populated by listing extractors.** Extracting a single
|
||||
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
|
||||
only matters when testing by hand.
|
||||
|
||||
## Cadence, and the budget that enforces it
|
||||
|
||||
**Monthly for everything, daily for stories only.** Stories expire in 24h and
|
||||
cannot be backfilled, so they are the one surface where missing a day means
|
||||
losing the content permanently. Everything else can wait — the skip-archive
|
||||
means an infrequent full sync costs barely more than a frequent one, because it
|
||||
only fetches what is new.
|
||||
|
||||
```
|
||||
# monthly, everything
|
||||
gdl-sync.py --index <viewer-url> --staging ~/gdl/staging \
|
||||
--publish <user>@<nas>:<archives> --archive-db ~/gdl/artms.db \
|
||||
--urls-file artms_account_links.txt --execute
|
||||
|
||||
# daily, stories only -- one request per profile
|
||||
gdl-sync.py ... --only stories --execute
|
||||
```
|
||||
|
||||
A stories-only run is one source per profile and **never seeds**, because a
|
||||
story cannot be in the archive before it is fetched; probing would double the
|
||||
cost of the cheapest surface for no benefit. Six profiles is a handful of
|
||||
requests.
|
||||
|
||||
When scheduling it, **randomise the minute and avoid the hour boundary**. A job
|
||||
that fires at exactly 09:00 every day is a machine; one that fires somewhere in
|
||||
a window looks like someone opening the app.
|
||||
|
||||
The tool now refuses to repeat itself:
|
||||
|
||||
| flag | default | what it prevents |
|
||||
|---|---|---|
|
||||
| `--min-interval` | 20h | re-fetching a source touched recently — the aborted-restart case that re-enumerated five profiles |
|
||||
| `--probe-ttl` | 24h | paying for a listing pass twice within a run cycle |
|
||||
| `--max-sources` | off | a runaway list touching more than intended |
|
||||
| `--force` | off | (escape hatch: ignores both guards) |
|
||||
|
||||
State lives beside the archive DB as `<db>.state.json`, recording per source
|
||||
when it was seeded and last fetched. **Seeding is a one-time bootstrap**: after
|
||||
the first successful sync the archive DB records everything gallery-dl has
|
||||
seen, so the source is never probed again. That is the single biggest saving
|
||||
here — a second full sync costs roughly half what the first did.
|
||||
|
||||
## Incremental sync — why the fetch host needs no copy of the archive
|
||||
|
||||
gallery-dl can skip already-held media two ways, and the difference decides
|
||||
whether the fetcher needs the archive mounted:
|
||||
|
||||
- **By file existence** (default). Needs the destination to already contain the
|
||||
files, so it only works if the archive is mounted where gallery-dl writes.
|
||||
- **By skip-archive** (`--download-archive`). A sqlite DB of ids. Needs nothing
|
||||
on disk.
|
||||
|
||||
We use the second, so the fetch host can write to **local disk and rsync
|
||||
afterwards**. That avoids writing tens of thousands of small files over CIFS,
|
||||
and keeps a mid-sync failure from leaving partial files on the live Resilio
|
||||
share.
|
||||
|
||||
The key is `archive_prefix + archive_fmt`, which for this extractor is the
|
||||
literal `instagram` plus the per-media numeric pk (`instagram.py:25`,
|
||||
`job.py:713-719`). Verified: a 3-image carousel produced
|
||||
|
||||
```
|
||||
instagram3079387627521318672
|
||||
instagram3079387627521429433
|
||||
instagram3079387627529716672
|
||||
```
|
||||
|
||||
and a second run skipped every media file, rewriting only the idempotent
|
||||
`.txt`/`.json` sidecars.
|
||||
|
||||
**Seeding.** `media_id` is not in our filenames, so the DB cannot be built from
|
||||
names alone — but one listing pass (the pass we make anyway) maps every live
|
||||
item to its `media_id`, and the archive's *file listing* says which we already
|
||||
hold. No extra Instagram requests, and no archive content — a listing is
|
||||
enough, which `GET /api/archives/:name/files` already serves.
|
||||
|
||||
Measured on `0ct0ber19`: 2275 live media items, 2248 seeded from the existing
|
||||
listing, **27 left to download** — precisely the media of the two posts added
|
||||
since the last crawl.
|
||||
|
||||
The one trap, which silently seeds almost nothing if you get it backwards:
|
||||
|
||||
| surface | filed under | why |
|
||||
|---|---|---|
|
||||
| posts, reels | `post_shortcode` | carousel children each have their own `shortcode`, which never appears in a filename |
|
||||
| stories, highlights | `shortcode` (per item) | `post_shortcode` is the containing reel's id, shared by every item |
|
||||
|
||||
`live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
|
||||
|
||||
### The skip-archive saves the CDN, not `instagram.com`
|
||||
|
||||
Worth being exact about, because the two costs land on different surfaces and
|
||||
only one of them bans accounts:
|
||||
|
||||
| what | which surface | scales with |
|
||||
|---|---|---|
|
||||
| downloading media | `scontent-*.cdninstagram.com` | how much is **new** |
|
||||
| enumerating the profile to find it | `instagram.com` | how **big** the profile is |
|
||||
|
||||
The skip-archive suppresses the first. It does nothing about the second, so a
|
||||
2275-post profile costs ~76 pages of pagination every run, forever, whether it
|
||||
has three new posts or none. Seeding (above) saved a *second* full pass, not
|
||||
the first.
|
||||
|
||||
Measured on the 2026-08-20 run, from sidecar write times in staging — free,
|
||||
since the run was paying for the listing anyway:
|
||||
|
||||
```
|
||||
1787248852 2026-08-19 … DcOeoVxkthi new, +0s
|
||||
1787248944 2026-08-18 … DcLpfoJCZtp new, +92s
|
||||
1787249058 2026-08-17 … DcIlGbxCUk0 new, +114s
|
||||
1787249162 2026-07-24 … DbKr1TxlPSX ┐ all one second: nothing
|
||||
1787249162 2026-08-15 … DcD-FdBCYGm ┘ downloaded, sidecars only
|
||||
```
|
||||
|
||||
Three posts took ~100s each; the remaining 2272 were enumeration with nothing
|
||||
to show for it.
|
||||
|
||||
**Pinned posts do not break early abort.** Test case 16 previously claimed
|
||||
`0ct0ber19` returns its 3 pinned posts out of date order — that is true of the
|
||||
*web grid*, but the REST `/posts/` listing came back strictly
|
||||
reverse-chronological, newest first, no hoisting. That matters because
|
||||
front-loaded old posts are the one thing that would make `skip: abort:N`
|
||||
dangerous: it would trip on them and abort before reaching anything new.
|
||||
|
||||
So `skip: abort:N` is viable, and cuts ~420 requests per run to ~40-60:
|
||||
|
||||
| surface | live items | pages | with `abort:50` |
|
||||
|---|---:|---:|---:|
|
||||
| posts, 6 profiles | 11,248 | ~377 | ~12 |
|
||||
| reels, 6 profiles | 1,080 | ~24 | ~8 |
|
||||
| stories + highlights | — | ~20 | ~20 |
|
||||
|
||||
N counts consecutive skipped **files**, not posts, so it must clear the largest
|
||||
already-held carousel — `DcD-FdBCYGm` alone is 22 media. 50 is comfortable; 5
|
||||
would not be.
|
||||
|
||||
**The tradeoff is edited carousels.** Test case 15 is a post that gained items
|
||||
after we archived it, and only a full enumeration finds those. Suggested
|
||||
policy: `abort:50` for routine runs, a full sweep occasionally.
|
||||
|
||||
Measured the same day, resuming a stopped run with `--abort 50`:
|
||||
|
||||
| source | live items | enumerated |
|
||||
|---|---:|---:|
|
||||
| `cher_ryppo` posts | 2,151 | **7** |
|
||||
| `cher_ryppo` reels | 92 | 53 |
|
||||
|
||||
One page instead of 72, and every new post was still caught. The 7 is roughly
|
||||
3 new posts plus 4 already-held carousels making up the 50 skipped files.
|
||||
Reels need 53 because they are single-media, so 50 consecutive skips really is
|
||||
50 reels — another reminder that N counts files, and that the same N behaves
|
||||
very differently on a carousel-heavy surface than on a reels tab.
|
||||
|
||||
## Publishing
|
||||
|
||||
The fetch host stages to local disk and rsyncs afterwards. `rsync
|
||||
--ignore-existing` is not an optimisation but the safety property: the archive
|
||||
deliberately outlives Instagram, so publishing must only ever **add**. No
|
||||
`--delete`, and nothing already present is overwritten — including sidecars,
|
||||
which are rewritten every run and would otherwise churn the synced share.
|
||||
|
||||
Publishing happens once at the end of a run, so a profile that fails midway
|
||||
never reaches the archive half-written.
|
||||
|
||||
## Status
|
||||
|
||||
In use for all six ARTMS profiles.
|
||||
|
||||
`withaseul` first — 322 files added (74 media, 241 `.json`, 7 `.txt`), nothing
|
||||
overwritten or deleted. Of the 74 new media, **zero** duplicated media already
|
||||
held under a different name, which is the check that says JD2 and gallery-dl
|
||||
naming really do converge.
|
||||
|
||||
**2026-08-20**, the first full incremental sync, four days after the previous
|
||||
one. 184 new media, 299 files published, 0 failures and **0 CDN 429s**:
|
||||
|
||||
| profile | posts | reels | stories | files added |
|
||||
|---|---:|---:|---:|---:|
|
||||
| 0ct0ber19 | 58 | 2 | 4 | +77 |
|
||||
| official_artms | 12 | — | 2 | +85 |
|
||||
| cher_ryppo | 41 | 1 | 8 | +63 |
|
||||
| zindoriyam | 23 | — | 4 | +35 |
|
||||
| kimxxlip | 16 | — | 2 | +23 |
|
||||
| withaseul | 10 | — | — | +16 |
|
||||
|
||||
The 20 story items are the part that could not have been recovered later.
|
||||
|
||||
Two things made it cheap, and both are worth keeping:
|
||||
|
||||
- The archive DB was already seeded from the previous run, so `--min-interval`
|
||||
and the recorded `seeded` state meant **no probe passes at all**. A state
|
||||
file has to exist for this; if one is missing after a manual run, write it
|
||||
rather than letting the tool re-seed 24 sources.
|
||||
- `--abort 50` (see above) cut the remaining listing cost by roughly 85%.
|
||||
|
||||
The run was deliberately **stopped and resumed** halfway to pick up `--abort`.
|
||||
That is safe precisely because of the state file: the 12 finished sources were
|
||||
already marked `fetched`, so the 20h floor skipped them and only the remaining
|
||||
12 re-ran. Stopping a run is cheap now; it was not before.
|
||||
|
||||
Published files land owned by the SSH user rather than `rslsync`. The viewer
|
||||
reads them fine (world-readable), but Resilio does not own what it syncs; worth
|
||||
a `chown` if that ever matters. This also makes **`rsync` exit 23**
|
||||
("some files/attrs were not transferred") the *normal* outcome of a publish —
|
||||
it is the failed `chown`, not lost data. Confirm by re-running the same rsync
|
||||
with `--dry-run`: an empty file list means everything arrived.
|
||||
|
||||
The profiles to fetch live in `artms_account_links.txt` at the archive root,
|
||||
passed with `--urls-file`.
|
||||
|
||||
## Verified run
|
||||
|
||||
`withaseul`, all four surfaces, staged locally and published to a scratch
|
||||
directory before the live publish above:
|
||||
|
||||
```
|
||||
==> withaseul / posts seeded 915 of 984 live items
|
||||
==> withaseul / reels seeded 28 of 34 live items
|
||||
==> withaseul / stories no results (none active)
|
||||
==> withaseul / highlights no results
|
||||
```
|
||||
|
||||
Output landed correctly, including the collab-reel case — `withaseul - reels`
|
||||
contains 53 files owned by `withaseul`, 10 by `cher_ryppo`, 3 by `0ct0ber19`
|
||||
and 2 by `official_artms`, all with the owner in the filename and the crawl
|
||||
scope as the directory.
|
||||
|
||||
### The CDN rate-limits, and the first run tripped it
|
||||
|
||||
At `rate: 3M` with `sleep: [1.0, 3.0]`, `scontent-*.cdninstagram.com` returned
|
||||
**`429 Too Many Requests`** and two videos were lost (gallery-dl retried, then
|
||||
gave up with exit 4). This is the *tolerant* surface complaining, which is a
|
||||
clear signal the pacing was too aggressive.
|
||||
|
||||
Defaults are now:
|
||||
|
||||
| option | value |
|
||||
|---|---|
|
||||
| `--rate` | `1M` |
|
||||
| `--sleep-request` | 6–10 s |
|
||||
| `--sleep` | 3–6 s |
|
||||
| `sleep-429` | 120 s |
|
||||
| `retries` (extractor and downloader) | 8 |
|
||||
|
||||
Re-running with those recovered both videos and produced **0 failures and 0
|
||||
429s**. Do not raise them for speed; an archive sync has no deadline.
|
||||
|
||||
### yt-dlp is worth installing
|
||||
|
||||
Without it, gallery-dl logs `Cannot import yt-dlp or youtube-dl` and falls back
|
||||
to a progressive URL for DASH videos. The fallback mostly works but is what the
|
||||
429s hit hardest.
|
||||
|
||||
**`pipx install yt-dlp` does not work** — it was the advice here until
|
||||
2026-08-20, and it is wrong. It gives yt-dlp its own venv, so the binary lands
|
||||
on `PATH` while gallery-dl, in a *different* venv, still cannot `import yt_dlp`.
|
||||
The symptom is that everything looks installed and the log keeps saying
|
||||
`Cannot import yt-dlp`. gallery-dl needs it importable, not runnable:
|
||||
|
||||
```sh
|
||||
pipx inject gallery-dl yt-dlp
|
||||
```
|
||||
|
||||
Verify by asking gallery-dl's own interpreter, not the shell:
|
||||
|
||||
```sh
|
||||
/home/matt/.local/share/pipx/venvs/gallery-dl/bin/python -c 'import yt_dlp'
|
||||
```
|
||||
|
||||
## Known quirks
|
||||
|
||||
- **`count` is not the emitted file count.** For 135 of 214 posts it was exactly
|
||||
one higher than the number of files written. This makes the `count >= 10`
|
||||
padding condition mis-pad a handful of 9-item posts (10 of 214 measured). Since
|
||||
the parser normalises the index, this is cosmetic — but it means a re-fetch
|
||||
over an existing JD2 tree writes `- 01.jpg` beside an existing `- 1.jpg`.
|
||||
- **Carousels get edited.** Two posts had a different media count live than on
|
||||
disk. Padding width follows the count *at download time*, so a grown carousel
|
||||
produces mixed widths — the archive already contains one such post from JD2.
|
||||
- **Highlights already have two naming styles on disk**, and every undated file
|
||||
has a dated twin. The scanner dedupes by index so they render once; it is
|
||||
wasted disk, not a display bug.
|
||||
|
||||
## Scanner work (not done yet)
|
||||
|
||||
`useArchiveScanner` currently treats any `.json` in the tree as a possible
|
||||
manifest. Adding gallery-dl sidecars needs it to distinguish three things:
|
||||
|
||||
1. Instagram export manifests (`posts_1.json`) — existing path.
|
||||
2. Instaloader `.json.xz` — existing path, GraphQL node shape.
|
||||
3. gallery-dl `.json` — new, flat shape, identified by having
|
||||
`post_shortcode` + `type` at the top level.
|
||||
|
||||
Once (3) is read, `source`/`isStory` and the reel flag should come from `type`
|
||||
rather than from the directory and the lone-video heuristic.
|
||||
|
||||
## Test cases
|
||||
|
||||
Real subjects, all present in the archive today. See
|
||||
`scripts/gdl-sync.py --selftest` for the harness.
|
||||
|
||||
| # | case | shortcode | expected |
|
||||
|---|---|---|---|
|
||||
| 1 | single image | `CwcXnQhOqFG` | one `.jpg`, no index |
|
||||
| 2 | single feed video | `DbdG9L9jU4m` | one `.mp4`, `type: post` |
|
||||
| 3 | carousel, images only | `Cq8LrxSJAJE` | `- 1 … - 3` |
|
||||
| 4 | carousel, image + video | `CtohvHxLnWO` | `- 1.jpg … - 4.mp4`, **no `.txt`** |
|
||||
| 5 | carousel of exactly 9 | `Cv2Hb_brx_N` | 1-digit index |
|
||||
| 6 | carousel of 10+ | `CzM8Uf6B6H_` | 2-digit index `- 01 … - 10` |
|
||||
| 7 | reel shown on the posts grid | `C8FHM6EJl15` | in `<user>`, `type: reel` |
|
||||
| 8 | reel on the reels tab | `Db-lNCoib9m` | in `<user> - reels`, `type: reel` |
|
||||
| 9 | collab reel (other owner) | `DYcZOb0h6Sv` | dir `0ct0ber19 - reels`, filename `chuuo3o` |
|
||||
| 10 | story | live only | `story - <user>`, per-item shortcode + date |
|
||||
| 11 | story highlight | `C-IImhvpFuk` | `story highlights - <user> - <title>` |
|
||||
| 12 | highlight, unicode title | `Drawheeing⠀` | trailing U+2800 preserved in dirname |
|
||||
| 13 | empty caption | `CrdsY5CrSsO` | media written, `.txt` absent |
|
||||
| 14 | deleted post | `C0TgI7sphfZ` | on disk, absent live — must not be removed |
|
||||
| 15 | edited carousel | `C7zG7-jJMlq` | 18 on disk, 8 live — must not be removed |
|
||||
| 16 | pinned posts | `0ct0ber19` | REST listing is strictly reverse-chronological; see below |
|
||||
| 17 | profile avatar | `0ct0ber19.jpg` | base dir, undated |
|
||||
|
||||
Cases 14–16 are reconciliation, not naming: **a sync must never delete**, since
|
||||
the archive deliberately outlives Instagram.
|
||||
|
||||
Not covered, decide before relying on them: the `/reposts/` tab (`0ct0ber19`
|
||||
has one) and `/tagged/`. Neither is fetched today.
|
||||
@@ -1,194 +0,0 @@
|
||||
# JDownloader2 — archive fetching
|
||||
|
||||
> **This file lives only on the `tooling` branch.** `main` is published to
|
||||
> GitHub and deliberately carries none of this — not the host details, not the
|
||||
> IPs, and not the account names. `main`'s history was redacted on 2026-08-20;
|
||||
> real names exist only here.
|
||||
>
|
||||
> There is no `npm run jd2` script — `package.json` and `CLAUDE.md` are kept
|
||||
> byte-identical to `main` so that merging `main` into `tooling` never
|
||||
> conflicts. Run the crawljob generator directly:
|
||||
>
|
||||
> ```sh
|
||||
> npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run
|
||||
> ```
|
||||
|
||||
How content gets into this archive, and why the setup is shaped the way it is.
|
||||
|
||||
## Why JDownloader and not Instaloader
|
||||
|
||||
There are two surfaces, and they're treated very differently:
|
||||
|
||||
| Surface | What hits it | Risk |
|
||||
|---|---|---|
|
||||
| `instagram.com` | profile pages, GraphQL/API metadata | Tied to your session, heavily rate-limited. **This is where bans come from.** |
|
||||
| `scontent*.cdninstagram.com` | the actual media | Signed URLs, CDN-served, tolerant. Mostly a bandwidth question. |
|
||||
|
||||
JDownloader does nearly all its work on the CDN. Instaloader's value — the rich
|
||||
`.json.xz` metadata — comes from asking `instagram.com` a question *per post*.
|
||||
|
||||
Concretely, from this archive: `rivvsofficial` has 188 post-metadata files, so
|
||||
backfilling it cost 188 API requests for one 605-file profile. That's the ban
|
||||
vector. Downloading the 238 photos was never the problem.
|
||||
|
||||
Instaloader got this account banned once. JDownloader with throttling did not.
|
||||
|
||||
> **The account was suspended anyway, on 2026-08-17, for "spam".** Not by
|
||||
> JDownloader, and not by downloading. It was suspended during a day of
|
||||
> *building and verifying* the gallery-dl replacement — automated browser
|
||||
> scrolling to enumerate profile grids, repeated `--simulate` and `-j` metadata
|
||||
> passes, and one aborted sync that re-ran every listing pass before dying.
|
||||
>
|
||||
> The framing above is right about which surface is dangerous and wrong about
|
||||
> what reaches it. **Every read of `instagram.com` counts, including the ones
|
||||
> that download nothing** — and read-only work is easy not to count precisely
|
||||
> because it leaves no files behind. See the post-mortem at the top of
|
||||
> `docs/gallery-dl.md`.
|
||||
>
|
||||
> The rule that would have prevented it: *verify against the archive, never
|
||||
> against the live site*, and treat the first CDN `429` as the end of the
|
||||
> session rather than a pacing knob.
|
||||
|
||||
### What the metadata gap actually costs
|
||||
|
||||
Comparing a JDownloader profile against an Instaloader one:
|
||||
|
||||
| | JDownloader | Instaloader |
|
||||
|---|---|---|
|
||||
| Media | ✅ | ✅ |
|
||||
| Captions (`.txt`) | ✅ | ✅ |
|
||||
| Dates (from filenames) | ✅ | ✅ |
|
||||
| Bio / full name | ❌ | ✅ |
|
||||
| Follower counts | ❌ | ✅ |
|
||||
| External URL | ❌ | ✅ |
|
||||
|
||||
Captions already work — the viewer reads the `.txt` sidecars. Everything missing
|
||||
lives in a *single* profile-level record, not the per-post ones. That's why
|
||||
JDownloader-sourced profiles show "0 followers" and a placeholder bio.
|
||||
|
||||
Not worth extra requests. If you ever want it, the zero-request option is a
|
||||
hand-written `profile.json` sidecar (not implemented yet — ask).
|
||||
|
||||
## Settings that matter
|
||||
|
||||
**Chunks per download → 1.** The single most important one. JDownloader splits
|
||||
each file into multiple ranged requests by default; that `Range` pattern looks
|
||||
nothing like a browser or the app. One chunk = one sequential GET per file.
|
||||
`jd2-sync` sets `chunks=1` per job, so no global change is needed — but set it
|
||||
globally too if you ever add links by hand.
|
||||
|
||||
**Max simultaneous downloads → 2–3**, connections-per-host low. Concurrency is
|
||||
what turns "a user" into a statistic.
|
||||
|
||||
**Leave reconnect / IP-change features off.** A mid-session IP change on a live
|
||||
cookie is a *stronger* anomaly signal than the request rate you'd be avoiding.
|
||||
|
||||
## The cookie
|
||||
|
||||
Exported manually from a real browser session. This is the right approach — no
|
||||
programmatic login anywhere, which is the thing that actually gets flagged.
|
||||
|
||||
- Use it from the **same public IP** as the browser it came from. A cookie used
|
||||
from a different network is what session-hijack detection looks for.
|
||||
- When it expires, **re-export from the browser**. Never add a login step to a tool.
|
||||
- It's a full account credential. Keep it off the NAS share and out of the repo.
|
||||
|
||||
## Workflow
|
||||
|
||||
Two URLs per profile, because the profile grid misses some reels:
|
||||
|
||||
```
|
||||
https://www.instagram.com/<user>/
|
||||
https://www.instagram.com/<user>/reels/
|
||||
```
|
||||
|
||||
They overlap slightly — a reel caught by both lands in each directory and shows
|
||||
up twice in the viewer. That's correct and matches Instagram, which also shows
|
||||
reels in the profile grid *and* the Reels tab.
|
||||
|
||||
## Generating jobs
|
||||
|
||||
Instead of pasting URLs and setting output folders by hand:
|
||||
|
||||
```bash
|
||||
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives --dry-run
|
||||
```
|
||||
|
||||
Review, then write it into JDownloader's folder-watch directory:
|
||||
|
||||
```bash
|
||||
npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives \
|
||||
--out ~/.jd2/folderwatch
|
||||
```
|
||||
|
||||
JDownloader runs on the desktop while the archive lives on the NAS, so tell it
|
||||
the path *it* sees:
|
||||
|
||||
```bash
|
||||
npm run jd2 -- --archives /mnt/nas/Instagram-archive/archives \
|
||||
--download-base 'Z:\Instagram-archive\archives' \
|
||||
--out ~/.jd2/folderwatch
|
||||
```
|
||||
|
||||
| Flag | Purpose |
|
||||
|---|---|
|
||||
| `--archives <dir>` | Archive root to scan (or `$ARCHIVES_DIR`) |
|
||||
| `--out <dir>` | JDownloader folder-watch directory |
|
||||
| `--download-base <dir>` | Root path as JDownloader sees it (Windows paths fine) |
|
||||
| `--user <name>` | Just this profile (repeatable) |
|
||||
| `--skip <name>` | Never emit jobs for this directory (repeatable) |
|
||||
| `--chunks <n>` | Connections per file (default 1) |
|
||||
| `--auto-start` | Start immediately instead of parking in LinkGrabber |
|
||||
| `--all-reels` | Emit a reels job even where no reels directory exists |
|
||||
| `--dry-run` | Print instead of writing |
|
||||
|
||||
Defaults are deliberately conservative: `chunks=1`, and links park in the
|
||||
LinkGrabber for review rather than auto-starting.
|
||||
|
||||
Only posts and reels are emitted. Highlight URLs need a numeric id and story
|
||||
URLs expire, so those stay manual.
|
||||
|
||||
Directories that aren't Instagram profiles are skipped by username shape
|
||||
(letters, digits, dots, underscores, ≤30 chars) — pointing a crawl at those
|
||||
spends `instagram.com` requests to be told the profile doesn't exist. For names
|
||||
that *look* like usernames but aren't, use `--skip` or a `.jd2ignore` file in
|
||||
the archive root, one name per line.
|
||||
|
||||
Format reference: `src/org/jdownloader/extensions/folderwatchV2/explain.txt`.
|
||||
JDownloader develops on SVN — read it via the daily mirror at
|
||||
<https://github.com/mycodedoesnotcompile2/jdownloader_mirror> (`svn_trunk/`),
|
||||
not one of the abandoned GitHub copies.
|
||||
|
||||
## Expected layout
|
||||
|
||||
Everything downloads into `<archives>/`, one directory per source:
|
||||
|
||||
```
|
||||
archives/
|
||||
0ct0ber19/ posts
|
||||
0ct0ber19 - reels/ reels
|
||||
story - 0ct0ber19/ stories
|
||||
story highlights - 0ct0ber19 - Heestory/ a highlight
|
||||
```
|
||||
|
||||
Non-archive directories (tool output, exports from elsewhere) live *outside*
|
||||
`archives/` so they never reach the viewer.
|
||||
|
||||
The server picks up changes automatically — its index is keyed on directory
|
||||
mtime, so a new file invalidates only that directory.
|
||||
|
||||
## If something goes wrong
|
||||
|
||||
**429 / rate limited** — stop for hours, not seconds. Retrying into a limit is
|
||||
what converts a soft throttle into something worse.
|
||||
|
||||
**Cookie stops working** — re-export from the browser. Don't add a login step.
|
||||
|
||||
**Files land in the wrong folder** — a Packagizer rule is overriding the job.
|
||||
Generated jobs set `overwritePackagizerEnabled=TRUE` to prevent this; check that
|
||||
rules aren't set to run after it.
|
||||
|
||||
**Viewer doesn't show new posts** — check the file is in the right directory and
|
||||
matches the naming pattern (`YYYY-MM-DD_<user> - <shortcode>[ - NN].<ext>`).
|
||||
The index refreshes on directory mtime, so a genuinely new file is picked up on
|
||||
the next request.
|
||||
@@ -1,843 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Fetch Instagram profiles into the archive layout using gallery-dl.
|
||||
|
||||
The CLI replacement for the JDownloader2 workflow. See docs/gallery-dl.md for
|
||||
the measurements behind every choice here — especially the safety model, which
|
||||
is the reason this script exists in this shape rather than a simpler one.
|
||||
|
||||
The fetch host needs no copy of the archive. It stages locally and rsyncs
|
||||
afterwards; what it already holds is learned from a *file listing* alone
|
||||
(`--index`), which the viewer's own API serves.
|
||||
|
||||
Usage:
|
||||
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \\
|
||||
--staging /var/tmp/gdl --publish user@host:/path/to/archives \\
|
||||
--urls-file artms_account_links.txt --dry-run
|
||||
|
||||
# ...then swap --dry-run for --execute. --index also accepts a local path,
|
||||
# and --profile / --all work instead of --urls-file.
|
||||
|
||||
Always --dry-run first: it prints the plan, and the publish step it reports is
|
||||
the one that would touch the archive.
|
||||
|
||||
Run it from the host whose public IP matches the browser the cookie came from;
|
||||
using the cookie from elsewhere is what session-hijack detection looks for.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Archive layout
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
# Mirrors src/lib/archive-grouping.ts. Instagram usernames cannot contain
|
||||
# spaces, which is what makes the username separable from a highlight title.
|
||||
RE_HIGHLIGHT = re.compile(r"^story highlights - ([^ ]+) - (.+)$")
|
||||
RE_STORIES = re.compile(r"^story - ([^ ]+)$")
|
||||
RE_REELS = re.compile(r"^([^ ]+) - reels$")
|
||||
|
||||
DATE_FMT = "{date:Olocal/%Y-%m-%d}"
|
||||
"""Local-time date. JD2 stamped US Eastern, NOT UTC (0/212 mismatches vs 19 for
|
||||
UTC). `Olocal` is DST-aware per timestamp. The trailing separator must be
|
||||
omitted or it lands in the strftime format and sanitises to an underscore."""
|
||||
|
||||
POST_STEM = DATE_FMT + "_{username} - {post_shortcode}"
|
||||
ITEM_STEM = DATE_FMT + "_{username} - {shortcode}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Source:
|
||||
"""One gallery-dl invocation: a URL fetched into a specific directory."""
|
||||
|
||||
kind: str # posts | reels | stories | highlights
|
||||
url: str
|
||||
directory: str # relative to the archives root
|
||||
subcategory: str # gallery-dl config key
|
||||
title: str | None = None # highlight title, when known
|
||||
|
||||
|
||||
@dataclass
|
||||
class Profile:
|
||||
user: str
|
||||
existing: dict[str, str] = field(default_factory=dict) # kind -> dirname
|
||||
|
||||
def sources(self, kinds: set[str]) -> list[Source]:
|
||||
u = self.user
|
||||
base = f"https://www.instagram.com/{u}"
|
||||
all_sources = [
|
||||
Source("posts", f"{base}/posts/", u, "posts"),
|
||||
Source("reels", f"{base}/reels/", f"{u} - reels", "reels"),
|
||||
# Stories expire after 24h, so these can only ever be captured
|
||||
# live. There is no backfill and no re-fetch -- which is why they
|
||||
# are the one surface worth visiting daily.
|
||||
Source("stories", f"https://www.instagram.com/stories/{u}/",
|
||||
f"story - {u}", "stories"),
|
||||
# Highlight directories embed the title, which gallery-dl only
|
||||
# learns mid-extraction -- so this one source fans out into many
|
||||
# directories and is handled with a directory format string.
|
||||
Source("highlights", f"{base}/highlights", "", "highlights"),
|
||||
]
|
||||
return [s for s in all_sources if s.kind in kinds]
|
||||
|
||||
|
||||
def scan_archives(root: Path) -> dict[str, Profile]:
|
||||
"""Group existing directories into profiles, as the server does."""
|
||||
profiles: dict[str, Profile] = {}
|
||||
|
||||
def get(user: str) -> Profile:
|
||||
return profiles.setdefault(user, Profile(user))
|
||||
|
||||
for entry in sorted(os.listdir(root)):
|
||||
if not (root / entry).is_dir() or entry.startswith("."):
|
||||
continue
|
||||
if m := RE_HIGHLIGHT.match(entry):
|
||||
get(m.group(1)).existing.setdefault("highlights", entry)
|
||||
elif m := RE_STORIES.match(entry):
|
||||
get(m.group(1)).existing["stories"] = entry
|
||||
elif m := RE_REELS.match(entry):
|
||||
get(m.group(1)).existing["reels"] = entry
|
||||
else:
|
||||
get(entry).existing["posts"] = entry
|
||||
return profiles
|
||||
|
||||
|
||||
RE_PROFILE_URL = re.compile(
|
||||
r"^(?:https?://)?(?:www\.)?instagram\.com/(?P<user>[^/?#\s]+)/?", re.I)
|
||||
|
||||
# Path segments that are Instagram features, not profiles. A line like
|
||||
# ".../p/ABC123/" names a post, and treating "p" as a username would silently
|
||||
# sync nothing under a nonsense directory.
|
||||
RESERVED_SEGMENTS = {
|
||||
"p", "reel", "reels", "stories", "explore", "accounts", "direct",
|
||||
"tv", "s", "invites", "challenge", "about", "developer",
|
||||
}
|
||||
|
||||
|
||||
def read_urls_file(path: Path) -> list[str]:
|
||||
"""
|
||||
Read profile URLs (or bare usernames) from a file, one per line.
|
||||
|
||||
Written for hand-maintained lists: blank lines are skipped, `#` starts a
|
||||
comment, and either a full URL or a bare username works. Order is kept and
|
||||
duplicates dropped, so a list can be appended to without care.
|
||||
"""
|
||||
users: list[str] = []
|
||||
seen: set[str] = set()
|
||||
|
||||
for lineno, raw in enumerate(path.read_text().splitlines(), 1):
|
||||
line = raw.split("#", 1)[0].strip()
|
||||
if not line:
|
||||
continue
|
||||
|
||||
m = RE_PROFILE_URL.match(line)
|
||||
user = m.group("user") if m else line.strip("/")
|
||||
|
||||
if not user or "/" in user or " " in user:
|
||||
print(f"{path}:{lineno}: cannot read a username from {raw.strip()!r}",
|
||||
file=sys.stderr)
|
||||
continue
|
||||
if user.lower() in RESERVED_SEGMENTS:
|
||||
print(f"{path}:{lineno}: {user!r} is an Instagram path, not a "
|
||||
f"profile — skipping", file=sys.stderr)
|
||||
continue
|
||||
if user in seen:
|
||||
continue
|
||||
|
||||
seen.add(user)
|
||||
users.append(user)
|
||||
|
||||
return users
|
||||
|
||||
|
||||
class ArchiveIndex:
|
||||
"""
|
||||
What the archive already holds, as filenames only.
|
||||
|
||||
Deliberately never reads file *contents*, so the fetch host does not need a
|
||||
copy of the archive — it can stage locally and rsync afterwards. Backed
|
||||
either by a local directory or by the viewer's own API, which already
|
||||
serves exactly this listing and is the cheaper option when the archive
|
||||
lives on network storage (a full walk there took ~52s).
|
||||
"""
|
||||
|
||||
def __init__(self, source: str):
|
||||
self.remote = source.startswith(("http://", "https://"))
|
||||
self.source = source.rstrip("/") if self.remote else None
|
||||
self.root = None if self.remote else Path(source)
|
||||
if self.root and not self.root.is_dir():
|
||||
raise SystemExit(f"archive index not found: {source}")
|
||||
self._cache: dict[str, list[str]] = {}
|
||||
|
||||
def _get(self, path: str):
|
||||
from urllib.request import urlopen
|
||||
with urlopen(f"{self.source}{path}", timeout=60) as resp:
|
||||
return json.load(resp)
|
||||
|
||||
def profiles(self) -> set[str]:
|
||||
if self.remote:
|
||||
return {a["name"] for a in self._get("/api/archives")}
|
||||
return set(scan_archives(self.root))
|
||||
|
||||
def listing(self, user: str) -> list[str]:
|
||||
"""Every filename belonging to a profile, across all its sidecars."""
|
||||
if user in self._cache:
|
||||
return self._cache[user]
|
||||
|
||||
names: list[str] = []
|
||||
if self.remote:
|
||||
try:
|
||||
data = self._get(f"/api/archives/{user}/files")
|
||||
except Exception:
|
||||
data = []
|
||||
files = data if isinstance(data, list) else data.get("files", [])
|
||||
names = [f["path"] for f in files]
|
||||
else:
|
||||
prof = scan_archives(self.root).get(user)
|
||||
for dirname in (prof.existing.values() if prof else ()):
|
||||
d = self.root / dirname
|
||||
if d.is_dir():
|
||||
names += [f"{dirname}/{n}" for n in os.listdir(d)]
|
||||
|
||||
self._cache[user] = names
|
||||
return names
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# gallery-dl configuration
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
def build_config(rate: str, sleep_request: list[float],
|
||||
sleep: list[float], abort: int = 0) -> dict:
|
||||
"""
|
||||
The config is generated rather than checked in so the safety-critical
|
||||
options cannot drift out of sync with the docs.
|
||||
|
||||
`api: rest` is the single most important line in this file. The graphql
|
||||
backend issues one request PER POST for every video and carousel, which is
|
||||
the pattern that got this account banned once already.
|
||||
"""
|
||||
caption_pp = {
|
||||
"name": "metadata",
|
||||
"event": "post",
|
||||
"mode": "custom",
|
||||
"content-format": "{description}",
|
||||
"extension": "txt",
|
||||
# JD2 wrote no .txt when the caption was empty; "empty": false (the
|
||||
# default) reproduces that.
|
||||
}
|
||||
meta_pp = {
|
||||
"name": "metadata",
|
||||
"event": "post",
|
||||
"mode": "json",
|
||||
# `include`, NOT `fields` -- `fields` applies to mode:custom and
|
||||
# silently does nothing here, dumping audio_user blobs that contain
|
||||
# unrelated users' profile picture URLs.
|
||||
"include": [
|
||||
"post_shortcode", "post_id", "type", "date", "post_date",
|
||||
"username", "fullname", "owner_id", "description", "count",
|
||||
"likes", "post_url", "sidecar_shortcode",
|
||||
],
|
||||
}
|
||||
|
||||
def post_like(stem: str) -> dict:
|
||||
"""Naming for surfaces whose unit is a post (posts, reels)."""
|
||||
skip: dict = {}
|
||||
if abort:
|
||||
# Stop enumerating once `abort` consecutive files are already in
|
||||
# the skip-archive. The listing pass -- not the downloading -- is
|
||||
# what costs `instagram.com` requests, and it otherwise walks the
|
||||
# whole profile every run to find three new posts.
|
||||
#
|
||||
# Safe here only because the REST listing is strictly
|
||||
# reverse-chronological: the web grid hoists pinned posts to the
|
||||
# front, but this endpoint does not (measured 2026-08-20), so old
|
||||
# posts never appear before new ones.
|
||||
#
|
||||
# Counted in FILES, not posts, so it must clear the largest
|
||||
# already-held carousel -- 22 media for one real post in this
|
||||
# archive. It also means edited carousels (test case 15) stop
|
||||
# being noticed, so a full sweep is still worth running
|
||||
# occasionally.
|
||||
skip["skip"] = f"abort:{abort}"
|
||||
return {
|
||||
**skip,
|
||||
# `sidecar_shortcode` is set only for carousels, so it is the
|
||||
# carousel discriminator. First matching condition wins.
|
||||
"filename": {
|
||||
"sidecar_shortcode and count >= 10":
|
||||
stem + " - {num:02}.{extension}",
|
||||
"sidecar_shortcode":
|
||||
stem + " - {num}.{extension}",
|
||||
"":
|
||||
stem + ".{extension}",
|
||||
},
|
||||
"postprocessors": [
|
||||
{**caption_pp, "filename": stem + ".txt"},
|
||||
{**meta_pp, "filename": stem + ".json"},
|
||||
],
|
||||
}
|
||||
|
||||
def item_like(stem: str) -> dict:
|
||||
"""
|
||||
Naming for surfaces whose unit is an item inside a reel (stories,
|
||||
highlights). `{shortcode}` is per item; `{post_shortcode}` is the
|
||||
reel's id and is shared by every item in it.
|
||||
|
||||
The media filename uses the per-item shortcode, but the sidecar cannot:
|
||||
it runs at `event: post`, where the kwdict describes the *reel* and has
|
||||
no `shortcode` at all -- which silently formatted as the literal
|
||||
"None", producing one "<date>_<user> - None.json" per reel. It is keyed
|
||||
by `post_shortcode` instead, and is genuinely reel-level data (the
|
||||
reel's own date and item count); per-item dates live in the media
|
||||
filenames, which is the more precise source anyway.
|
||||
"""
|
||||
return {
|
||||
"filename": stem + ".{extension}",
|
||||
"postprocessors": [
|
||||
{**meta_pp,
|
||||
"filename": DATE_FMT + "_{username} - {post_shortcode}.json"},
|
||||
],
|
||||
}
|
||||
|
||||
return {
|
||||
"extractor": {
|
||||
"base-directory": ".",
|
||||
"instagram": {
|
||||
"api": "rest", # never "graphql" -- see docstring
|
||||
"sleep-request": sleep_request,
|
||||
"sleep": sleep,
|
||||
# The CDN does rate-limit: a first run at 3M/1-3s drew
|
||||
# '429 Too Many Requests' from scontent-*.cdninstagram.com and
|
||||
# lost two videos. Back off hard rather than retry fast.
|
||||
"sleep-429": 120.0,
|
||||
"retries": 8,
|
||||
"videos": True,
|
||||
"include": "", # never "all"; sources are explicit
|
||||
# Directory is forced per-invocation with -D, because a reels
|
||||
# tab returns collab reels owned by OTHER accounts and
|
||||
# {username} would scatter them into the wrong profile.
|
||||
"directory": [],
|
||||
"posts": post_like(POST_STEM),
|
||||
"reels": post_like(POST_STEM),
|
||||
"stories": item_like(ITEM_STEM),
|
||||
"highlights": {
|
||||
**item_like(ITEM_STEM),
|
||||
# The only surface that must derive its own directory,
|
||||
# since the title is not known until extraction.
|
||||
"directory": ["story highlights - {username} - {highlight_title}"],
|
||||
},
|
||||
},
|
||||
},
|
||||
# `retries` here is the CDN-side counterpart to sleep-429 above.
|
||||
"downloader": {"http": {"rate": rate, "retries": 8}},
|
||||
"output": {"mode": "null"},
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Planning and execution
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
def gdl_command(src: Source, staging: Path, config: Path, cookies: str,
|
||||
archive_db: Path | None) -> list[str]:
|
||||
cmd = [
|
||||
"gallery-dl",
|
||||
"--config", str(config),
|
||||
"--cookies-from-browser", cookies,
|
||||
]
|
||||
if archive_db:
|
||||
# Without a seeded skip-archive, staging is empty and every file is
|
||||
# re-downloaded; see seed_archive_db.
|
||||
cmd += ["--download-archive", str(archive_db)]
|
||||
# Forced destination -- never `{username}` -- because a reels tab returns
|
||||
# collab reels owned by other accounts, which would otherwise be filed
|
||||
# under the wrong profile. Highlights are the exception: their directory
|
||||
# embeds a title only known mid-extraction, so the config formats it.
|
||||
dest = staging if src.subcategory == "highlights" else staging / src.directory
|
||||
cmd += ["--destination", str(dest)]
|
||||
cmd.append(src.url)
|
||||
return cmd
|
||||
|
||||
|
||||
# gallery-dl keys its skip-archive on `archive_prefix + archive_fmt`, which for
|
||||
# this extractor is the literal "instagram" followed by the per-media numeric
|
||||
# pk (`instagram.py:25`, `job.py:713-719`). Verified against a real run: a
|
||||
# 3-image carousel produced 3 rows, one per item.
|
||||
ARCHIVE_KEY = "instagram{}".format
|
||||
ARCHIVE_SCHEMA = "CREATE TABLE IF NOT EXISTS archive (entry TEXT PRIMARY KEY)"
|
||||
|
||||
RE_ARCHIVED = re.compile(
|
||||
r"^(\d{4}-\d{2}-\d{2})_(.+?) - ([A-Za-z0-9_-]+?)(?: - (\d+))?\.(\w+)$")
|
||||
NON_MEDIA = {"txt", "json"}
|
||||
|
||||
|
||||
def index_existing(listing: list[str]) -> set[tuple[str, int]]:
|
||||
"""
|
||||
Reduce a flat list of filenames to the (shortcode, index) pairs already
|
||||
held. Only names matter — never the bytes — which is what lets the sync run
|
||||
on a host that has no copy of the archive.
|
||||
"""
|
||||
have: set[tuple[str, int]] = set()
|
||||
for name in listing:
|
||||
m = RE_ARCHIVED.match(name.rsplit("/", 1)[-1])
|
||||
if not m or m.group(5).lower() in NON_MEDIA:
|
||||
continue
|
||||
# An absent index means a single-media post, which is index 1 — the
|
||||
# same normalisation the viewer's EXPORT_RE applies.
|
||||
have.add((m.group(3), int(m.group(4) or 1)))
|
||||
return have
|
||||
|
||||
|
||||
def live_key(item: dict, kind: str) -> tuple[str, int]:
|
||||
"""
|
||||
The (shortcode, index) a live item *would* be filed under, mirroring the
|
||||
filename template exactly.
|
||||
|
||||
The two surfaces disagree about which shortcode identifies a file, and
|
||||
getting this wrong silently seeds almost nothing:
|
||||
|
||||
posts/reels filed under {post_shortcode} — for a carousel, each
|
||||
child item ALSO has its own `shortcode`, which is not
|
||||
what appears in the filename.
|
||||
stories/highlights filed under the per-item {shortcode}, because
|
||||
`post_shortcode` there is the containing reel's id and
|
||||
is shared by every item in it.
|
||||
"""
|
||||
if kind in ("stories", "highlights"):
|
||||
return (item.get("shortcode"), 1)
|
||||
return (item.get("post_shortcode"), item.get("num"))
|
||||
|
||||
|
||||
def seed_archive_db(db: Path, existing: set[tuple[str, int]],
|
||||
live: list[dict], kind: str) -> int:
|
||||
"""
|
||||
Mark everything already held as downloaded, so a fetch into an empty
|
||||
directory pulls only what is missing.
|
||||
|
||||
`live` is the metadata of one listing pass — the pass we have to make
|
||||
anyway — each entry carrying at least `media_id` plus the shortcode fields
|
||||
`live_key` needs. Seeding costs no additional Instagram requests, and needs
|
||||
only a *listing* of the archive, never its contents.
|
||||
"""
|
||||
import sqlite3
|
||||
|
||||
db.parent.mkdir(parents=True, exist_ok=True)
|
||||
con = sqlite3.connect(db)
|
||||
con.execute(ARCHIVE_SCHEMA)
|
||||
rows = [
|
||||
(ARCHIVE_KEY(item["media_id"]),)
|
||||
for item in live
|
||||
if live_key(item, kind) in existing
|
||||
]
|
||||
con.executemany("INSERT OR IGNORE INTO archive (entry) VALUES (?)", rows)
|
||||
con.commit()
|
||||
con.close()
|
||||
return len(rows)
|
||||
|
||||
|
||||
def probe_live(src: Source, config: Path, cookies: str) -> list[dict]:
|
||||
"""
|
||||
One metadata-only listing pass. `sleep` is forced to 0 because it otherwise
|
||||
applies per *file* even with no download — 2275 files at 1-3s each is over
|
||||
an hour for a single profile.
|
||||
"""
|
||||
out = subprocess.run(
|
||||
["gallery-dl", "-j", "--config", str(config),
|
||||
"--cookies-from-browser", cookies, "-o", "sleep=0", src.url],
|
||||
capture_output=True, text=True, check=True,
|
||||
)
|
||||
items: list[dict] = []
|
||||
|
||||
def walk(node):
|
||||
if isinstance(node, dict):
|
||||
if "media_id" in node and "shortcode" in node:
|
||||
items.append(node)
|
||||
for value in node.values():
|
||||
walk(value)
|
||||
elif isinstance(node, list):
|
||||
for value in node:
|
||||
walk(value)
|
||||
|
||||
walk(json.loads(out.stdout))
|
||||
return items
|
||||
|
||||
|
||||
ALL_KINDS = ("posts", "reels", "stories", "highlights")
|
||||
|
||||
# Stories cannot be backfilled and expire in 24h, so a run that only wants
|
||||
# stories is both cheap and the one worth scheduling daily.
|
||||
STORIES_ONLY = {"stories"}
|
||||
|
||||
|
||||
class SyncState:
|
||||
"""
|
||||
What has already been spent against `instagram.com`.
|
||||
|
||||
Exists because nothing else in this tool has any memory: every invocation
|
||||
used to start from zero and happily re-enumerate profiles it had listed
|
||||
minutes earlier. That is what suspended the account — the listing passes,
|
||||
not the downloads.
|
||||
|
||||
Two facts are tracked per source:
|
||||
|
||||
seeded the skip-archive has been primed from the archive listing.
|
||||
This is a ONE-TIME bootstrap: afterwards the archive DB records
|
||||
every item gallery-dl has seen, so the source never needs
|
||||
probing again. This is the single biggest request saving here.
|
||||
fetched when it was last downloaded, so a re-run soon after is refused
|
||||
rather than silently repeating the whole pass.
|
||||
"""
|
||||
|
||||
VERSION = 1
|
||||
|
||||
def __init__(self, path: Path):
|
||||
self.path = path
|
||||
self.data = {"version": self.VERSION, "sources": {}}
|
||||
if path.is_file():
|
||||
try:
|
||||
loaded = json.loads(path.read_text())
|
||||
if loaded.get("version") == self.VERSION:
|
||||
self.data = loaded
|
||||
except Exception:
|
||||
pass # a corrupt state file must never block a sync
|
||||
|
||||
def _entry(self, url: str) -> dict:
|
||||
return self.data.setdefault("sources", {}).setdefault(url, {})
|
||||
|
||||
def needs_seed(self, url: str) -> bool:
|
||||
return not self._entry(url).get("seeded")
|
||||
|
||||
def mark_seeded(self, url: str, stamp: str) -> None:
|
||||
self._entry(url)["seeded"] = stamp
|
||||
|
||||
def last_fetch(self, url: str) -> str | None:
|
||||
return self._entry(url).get("fetched")
|
||||
|
||||
def mark_fetched(self, url: str, stamp: str) -> None:
|
||||
self._entry(url)["fetched"] = stamp
|
||||
|
||||
def save(self) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self.path.write_text(json.dumps(self.data, indent=1, sort_keys=True))
|
||||
|
||||
|
||||
def hours_since(stamp: str | None, now: float) -> float:
|
||||
"""Hours between an ISO stamp and `now`; infinite when never."""
|
||||
if not stamp:
|
||||
return float("inf")
|
||||
try:
|
||||
then = dt.datetime.fromisoformat(stamp)
|
||||
except ValueError:
|
||||
return float("inf")
|
||||
if then.tzinfo is None:
|
||||
then = then.replace(tzinfo=dt.timezone.utc)
|
||||
return (now - then.timestamp()) / 3600.0
|
||||
|
||||
|
||||
def plan_source(src: Source, state: SyncState, now: float,
|
||||
min_interval: float) -> tuple[bool, bool, str]:
|
||||
"""
|
||||
Decide what a source needs: (fetch, seed, reason).
|
||||
|
||||
Seeding is skipped once done, and skipped entirely for stories — a story
|
||||
cannot exist in the archive before it is fetched, so there is nothing to
|
||||
seed from, and probing would double the request cost of the cheapest
|
||||
surface we have.
|
||||
"""
|
||||
since = hours_since(state.last_fetch(src.url), now)
|
||||
if since < min_interval:
|
||||
return (False, False, f"fetched {since:.1f}h ago, under the "
|
||||
f"{min_interval:g}h floor")
|
||||
if src.kind == "stories":
|
||||
return (True, False, "stories: no seed needed")
|
||||
if state.needs_seed(src.url):
|
||||
return (True, True, "first run: seeding from the archive listing")
|
||||
return (True, False, "already seeded; the skip-archive knows what we hold")
|
||||
|
||||
|
||||
class ProbeCache:
|
||||
"""
|
||||
Listing-pass results, kept so an interrupted run does not pay for them
|
||||
twice. Yesterday an aborted sync re-enumerated five profiles on restart.
|
||||
"""
|
||||
|
||||
def __init__(self, path: Path, ttl_hours: float):
|
||||
self.path = path
|
||||
self.ttl = ttl_hours
|
||||
self.data: dict = {}
|
||||
if path.is_file():
|
||||
try:
|
||||
self.data = json.loads(path.read_text())
|
||||
except Exception:
|
||||
self.data = {}
|
||||
|
||||
def get(self, url: str, now: float) -> list[dict] | None:
|
||||
entry = self.data.get(url)
|
||||
if not entry or hours_since(entry.get("at"), now) > self.ttl:
|
||||
return None
|
||||
return entry.get("items")
|
||||
|
||||
def put(self, url: str, items: list[dict], stamp: str) -> None:
|
||||
# Only the fields seeding needs, so the cache stays small.
|
||||
self.data[url] = {"at": stamp, "items": [
|
||||
{k: i.get(k) for k in ("shortcode", "post_shortcode", "num", "media_id")}
|
||||
for i in items
|
||||
]}
|
||||
|
||||
def save(self) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self.path.write_text(json.dumps(self.data))
|
||||
|
||||
|
||||
def rsync_command(staging: Path, dest: str, dry_run: bool) -> list[str]:
|
||||
"""
|
||||
Publish a staging tree into the archive.
|
||||
|
||||
`--ignore-existing` is not an optimisation, it is the safety property: the
|
||||
archive deliberately outlives Instagram (posts exist here that Instagram no
|
||||
longer serves), so publishing must only ever *add*. No `--delete`, and
|
||||
nothing already present is overwritten — including sidecars, which get
|
||||
rewritten on every run and would otherwise churn the synced share.
|
||||
|
||||
`dest` may be a local path or any rsync destination (`user@host:/path`),
|
||||
because the archive usually is not writable from the fetch host.
|
||||
"""
|
||||
cmd = ["rsync", "-a", "--ignore-existing", "--partial", "--info=stats2",
|
||||
# Belt and braces: the config lives outside staging, but nothing
|
||||
# resembling tooling output should ever reach the archive. Archive
|
||||
# sidecars are always "<date>_<user> - <code>.json", so none of
|
||||
# these can match real content.
|
||||
"--exclude", "gdl-sync*.json",
|
||||
"--exclude", "*.gdl-config.json",
|
||||
"--exclude", ".gdl-*",
|
||||
"--exclude", "*.sqlite", "--exclude", "*.db"]
|
||||
if dry_run:
|
||||
cmd.append("--dry-run")
|
||||
# Trailing slash: copy the *contents* of staging into dest.
|
||||
cmd += [f"{staging}/", dest if dest.endswith("/") else dest + "/"]
|
||||
return cmd
|
||||
|
||||
|
||||
def publish(staging: Path, dest: str, dry_run: bool) -> int:
|
||||
if not any(staging.iterdir()):
|
||||
print(" nothing staged; skipping publish")
|
||||
return 0
|
||||
cmd = rsync_command(staging, dest, dry_run)
|
||||
print(" " + " ".join(cmd))
|
||||
return subprocess.run(cmd).returncode
|
||||
|
||||
|
||||
def main() -> int:
|
||||
# A sync runs for hours and is normally watched through a redirected log,
|
||||
# where Python's block buffering would withhold progress until it happened
|
||||
# to flush -- and the gallery-dl subprocesses write to the same descriptor
|
||||
# unbuffered, so the log would also interleave out of order.
|
||||
sys.stdout.reconfigure(line_buffering=True)
|
||||
sys.stderr.reconfigure(line_buffering=True)
|
||||
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--index", required=True,
|
||||
help="existing archive listing: a local root, or the "
|
||||
"viewer's base URL (only a FILE LISTING is needed, "
|
||||
"never the contents)")
|
||||
ap.add_argument("--publish", required=True,
|
||||
help="rsync destination for fetched files; a local path or "
|
||||
"user@host:/path")
|
||||
ap.add_argument("--staging", type=Path, required=True,
|
||||
help="local scratch directory gallery-dl writes into")
|
||||
g = ap.add_mutually_exclusive_group(required=True)
|
||||
g.add_argument("--profile", action="append", default=[],
|
||||
help="profile to sync; repeatable")
|
||||
g.add_argument("--all", action="store_true", help="every profile on disk")
|
||||
g.add_argument("--urls-file", type=Path,
|
||||
help="file of Instagram profile URLs or usernames, one per "
|
||||
"line; # comments and blank lines allowed")
|
||||
ap.add_argument("--cookies", default="chrome:/home/matt/.config/google-chrome-devtools",
|
||||
help="gallery-dl --cookies-from-browser value")
|
||||
ap.add_argument("--archive-db", type=Path, default=None,
|
||||
help="gallery-dl skip-archive sqlite path")
|
||||
ap.add_argument("--rate", default="1M", help="per-download rate cap")
|
||||
ap.add_argument("--sleep-request", nargs=2, type=float, default=[6.0, 10.0],
|
||||
metavar=("MIN", "MAX"))
|
||||
ap.add_argument("--sleep", nargs=2, type=float, default=[3.0, 6.0],
|
||||
metavar=("MIN", "MAX"))
|
||||
ap.add_argument("--only", default=",".join(ALL_KINDS),
|
||||
help="comma-separated surfaces to sync: "
|
||||
"posts,reels,stories,highlights. Use --only stories "
|
||||
"for the cheap daily run.")
|
||||
ap.add_argument("--min-interval", type=float, default=20.0, metavar="HOURS",
|
||||
help="refuse to re-fetch a source touched more recently "
|
||||
"than this (default 20h); the guard that makes a "
|
||||
"restart cheap instead of a repeat")
|
||||
ap.add_argument("--max-sources", type=int, default=0, metavar="N",
|
||||
help="hard ceiling on sources touched in one run "
|
||||
"(0 = no limit)")
|
||||
ap.add_argument("--abort", type=int, default=0, metavar="N",
|
||||
help="stop enumerating posts/reels after N consecutive "
|
||||
"already-archived FILES (0 = walk everything, the "
|
||||
"default). 50 is a safe routine value; it cuts the "
|
||||
"per-run listing cost by roughly 85%%, at the price "
|
||||
"of no longer noticing edited carousels")
|
||||
ap.add_argument("--probe-ttl", type=float, default=24.0, metavar="HOURS",
|
||||
help="reuse cached listing results younger than this")
|
||||
ap.add_argument("--force", action="store_true",
|
||||
help="ignore --min-interval and the probe cache")
|
||||
mode = ap.add_mutually_exclusive_group()
|
||||
mode.add_argument("--dry-run", action="store_true", default=True,
|
||||
help="print the plan and the config; default")
|
||||
mode.add_argument("--execute", action="store_true",
|
||||
help="actually run gallery-dl")
|
||||
args = ap.parse_args()
|
||||
|
||||
if not shutil.which("gallery-dl"):
|
||||
print("gallery-dl not on PATH", file=sys.stderr)
|
||||
return 2
|
||||
if not shutil.which("rsync"):
|
||||
print("rsync not on PATH", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
index = ArchiveIndex(args.index)
|
||||
names = index.profiles()
|
||||
if args.urls_file:
|
||||
if not args.urls_file.is_file():
|
||||
print(f"urls file not found: {args.urls_file}", file=sys.stderr)
|
||||
return 2
|
||||
wanted = read_urls_file(args.urls_file)
|
||||
if not wanted:
|
||||
print(f"no usable profiles in {args.urls_file}", file=sys.stderr)
|
||||
return 2
|
||||
print(f"read {len(wanted)} profile(s) from {args.urls_file}")
|
||||
selected = [Profile(p) for p in wanted]
|
||||
elif args.profile:
|
||||
for p in args.profile:
|
||||
if p not in names:
|
||||
print(f"note: {p} is not in the index yet; it will be created")
|
||||
selected = [Profile(p) for p in args.profile]
|
||||
else:
|
||||
selected = [Profile(p) for p in sorted(names)]
|
||||
|
||||
config = build_config(args.rate, list(args.sleep_request),
|
||||
list(args.sleep), args.abort)
|
||||
args.staging.mkdir(parents=True, exist_ok=True)
|
||||
# Deliberately a SIBLING of the staging directory, not inside it: staging is
|
||||
# rsynced wholesale into the archive, and a dry run caught this file being
|
||||
# published to the archive root.
|
||||
config_path = args.staging.parent / f"{args.staging.name}.gdl-config.json"
|
||||
|
||||
kinds = {k.strip() for k in args.only.split(",") if k.strip()}
|
||||
unknown = kinds - set(ALL_KINDS)
|
||||
if unknown:
|
||||
print(f"unknown surface(s): {', '.join(sorted(unknown))}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
state_path = (args.archive_db.with_suffix(".state.json") if args.archive_db
|
||||
else args.staging.parent / f"{args.staging.name}.state.json")
|
||||
state = SyncState(state_path)
|
||||
now = dt.datetime.now(dt.timezone.utc)
|
||||
now_ts, stamp = now.timestamp(), now.isoformat()
|
||||
min_interval = 0.0 if args.force else args.min_interval
|
||||
|
||||
plan: list[tuple[Profile, Source, bool]] = []
|
||||
skipped = 0
|
||||
for prof in selected:
|
||||
for src in prof.sources(kinds):
|
||||
fetch, seed, reason = plan_source(src, state, now_ts, min_interval)
|
||||
if not fetch:
|
||||
skipped += 1
|
||||
print(f" skip {prof.user}/{src.kind}: {reason}")
|
||||
continue
|
||||
if args.max_sources and len(plan) >= args.max_sources:
|
||||
skipped += 1
|
||||
continue
|
||||
plan.append((prof, src, seed))
|
||||
|
||||
print(f"profiles : {len(selected)}")
|
||||
print(f"surfaces : {','.join(k for k in ALL_KINDS if k in kinds)}")
|
||||
print(f"sources : {len(plan)} to sync, {skipped} skipped")
|
||||
print(f"pacing : {args.sleep_request[0]}-{args.sleep_request[1]}s between "
|
||||
f"requests, rate cap {args.rate}")
|
||||
print(f"staging : {args.staging}")
|
||||
print(f"publish : {args.publish}")
|
||||
print()
|
||||
|
||||
if not args.execute:
|
||||
for prof, src, seed in plan:
|
||||
dest = src.directory or "(per-highlight)"
|
||||
note = " [will seed]" if seed else ""
|
||||
print(f" {prof.user:<20} {src.kind:<11} -> {dest}{note}")
|
||||
print()
|
||||
print(" " + " ".join(rsync_command(args.staging, args.publish, True)))
|
||||
print("\ndry run; nothing fetched. pass --execute to run.")
|
||||
return 0
|
||||
|
||||
config_path.write_text(json.dumps(config, indent=2))
|
||||
probes = ProbeCache(state_path.with_suffix(".probes.json"),
|
||||
0.0 if args.force else args.probe_ttl)
|
||||
failures = 0
|
||||
|
||||
for prof, src, seed in plan:
|
||||
print(f"==> {prof.user} / {src.kind}")
|
||||
stage_dir = args.staging / (src.directory or ".")
|
||||
stage_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Prime the skip-archive from what the archive already holds, so
|
||||
# fetching into an empty staging directory pulls only what is missing.
|
||||
# Done once per source, ever: afterwards the archive DB records
|
||||
# everything gallery-dl has seen and no listing pass is needed.
|
||||
if seed and args.archive_db:
|
||||
try:
|
||||
live = probes.get(src.url, now_ts)
|
||||
if live is None:
|
||||
live = probe_live(src, config_path, args.cookies)
|
||||
probes.put(src.url, live, stamp)
|
||||
probes.save()
|
||||
else:
|
||||
print(f" reusing {len(live)} cached listing items")
|
||||
held = index_existing(index.listing(prof.user))
|
||||
seeded = seed_archive_db(args.archive_db, held, live,
|
||||
src.subcategory)
|
||||
print(f" seeded {seeded} of {len(live)} live items")
|
||||
state.mark_seeded(src.url, stamp)
|
||||
state.save()
|
||||
except subprocess.CalledProcessError as exc:
|
||||
failures += 1
|
||||
print(f" probe FAILED: {exc}", file=sys.stderr)
|
||||
continue
|
||||
|
||||
cmd = gdl_command(src, args.staging, config_path, args.cookies,
|
||||
args.archive_db)
|
||||
result = subprocess.run(cmd)
|
||||
if result.returncode != 0:
|
||||
failures += 1
|
||||
# Keep going: one private or renamed profile must not abort the run.
|
||||
print(f" FAILED (exit {result.returncode})", file=sys.stderr)
|
||||
else:
|
||||
# Recorded even for an empty fetch: the request was still spent.
|
||||
state.mark_fetched(src.url, stamp)
|
||||
state.save()
|
||||
|
||||
# Publish once, at the end, so a partially-fetched profile never reaches
|
||||
# the archive mid-run. Only ever adds -- see rsync_command.
|
||||
print("\n==> publish")
|
||||
if publish(args.staging, args.publish, dry_run=False) != 0:
|
||||
failures += 1
|
||||
|
||||
print(f"\ndone; {failures} step(s) failed")
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,277 +0,0 @@
|
||||
/**
|
||||
* Generate JDownloader2 .crawljob files for the archives on disk.
|
||||
*
|
||||
* The manual flow is: paste a profile URL into JDownloader, paste the /reels
|
||||
* URL separately (the profile page misses some reels), and set the output
|
||||
* folder by hand — times however many profiles you keep. This emits one
|
||||
* crawljob per source with the folder already pointed at the right directory,
|
||||
* so JDownloader's folder-watch picks the whole batch up at once.
|
||||
*
|
||||
* Profiles and their sidecar directories are derived with the same grouping
|
||||
* logic the server uses, so the output folders always match what the viewer
|
||||
* expects to find.
|
||||
*
|
||||
* Only posts and reels are emitted. Story and highlight URLs can't be rebuilt
|
||||
* from a directory name — highlights need their numeric id and stories expire —
|
||||
* so those stay manual.
|
||||
*
|
||||
* Crawljob format verified against JDownloader's own docs for the extension:
|
||||
* src/org/jdownloader/extensions/folderwatchV2/explain.txt. JDownloader
|
||||
* develops on SVN; read it via the daily mirror at
|
||||
* https://github.com/mycodedoesnotcompile2/jdownloader_mirror (svn_trunk/),
|
||||
* not one of the abandoned GitHub copies — several are a decade stale.
|
||||
*
|
||||
* Entries are separated by `->NEW ENTRY<-` and any property may be omitted.
|
||||
* There is also a `setBeforePackagizerEnabled` companion to
|
||||
* `overwritePackagizerEnabled`, if the Packagizer ever needs to see these
|
||||
* values before they're applied.
|
||||
*
|
||||
* Usage:
|
||||
* npx tsx scripts/jd2-sync.ts --archives <dir> [options]
|
||||
*
|
||||
* --archives <dir> Archive root to scan (default: $ARCHIVES_DIR)
|
||||
* --out <dir> JDownloader folder-watch directory to write into
|
||||
* --download-base <dir> Root path as *JDownloader* sees it, when it runs on
|
||||
* a different machine than this script (e.g. a mapped
|
||||
* drive). Defaults to --archives.
|
||||
* --user <name> Only this profile (repeatable)
|
||||
* --skip <name> Never emit jobs for this directory (repeatable).
|
||||
* Also read from a `.jd2ignore` file in the archive
|
||||
* root, one name per line.
|
||||
* --chunks <n> Connections per file (default 1: multi-chunk ranged
|
||||
* requests are the one CDN pattern that doesn't look
|
||||
* like a browser)
|
||||
* --auto-start Start downloads immediately instead of parking them
|
||||
* in the LinkGrabber for review
|
||||
* --all-reels Emit a reels job even where no reels directory
|
||||
* exists yet
|
||||
* --dry-run Print the crawljob instead of writing it
|
||||
*/
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { groupArchiveDirectories, ArchiveSource } from '../src/lib/archive-grouping.js';
|
||||
|
||||
interface Options {
|
||||
archives: string;
|
||||
out: string | null;
|
||||
downloadBase: string;
|
||||
users: string[];
|
||||
skip: Set<string>;
|
||||
chunks: number;
|
||||
autoStart: boolean;
|
||||
allReels: boolean;
|
||||
dryRun: boolean;
|
||||
}
|
||||
|
||||
const parseArgs = (argv: string[]): Options => {
|
||||
const opts: Options = {
|
||||
archives: process.env.ARCHIVES_DIR ?? '',
|
||||
out: null,
|
||||
downloadBase: '',
|
||||
users: [],
|
||||
skip: new Set(),
|
||||
chunks: 1,
|
||||
autoStart: false,
|
||||
allReels: false,
|
||||
dryRun: false,
|
||||
};
|
||||
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const arg = argv[i];
|
||||
const next = () => argv[++i];
|
||||
switch (arg) {
|
||||
case '--archives': opts.archives = path.resolve(next()); break;
|
||||
case '--out': opts.out = path.resolve(next()); break;
|
||||
case '--download-base': opts.downloadBase = next(); break;
|
||||
case '--user': opts.users.push(next()); break;
|
||||
case '--skip': opts.skip.add(next()); break;
|
||||
case '--chunks': opts.chunks = parseInt(next(), 10); break;
|
||||
case '--auto-start': opts.autoStart = true; break;
|
||||
case '--all-reels': opts.allReels = true; break;
|
||||
case '--dry-run': opts.dryRun = true; break;
|
||||
case '--help': case '-h': printUsage(); process.exit(0);
|
||||
default:
|
||||
console.error(`Unknown argument: ${arg}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
if (!opts.archives) {
|
||||
console.error('No archive root. Pass --archives <dir> or set ARCHIVES_DIR.');
|
||||
process.exit(1);
|
||||
}
|
||||
if (!opts.downloadBase) opts.downloadBase = opts.archives;
|
||||
if (!opts.out && !opts.dryRun) {
|
||||
console.error('No destination. Pass --out <folder-watch dir>, or --dry-run to preview.');
|
||||
process.exit(1);
|
||||
}
|
||||
return opts;
|
||||
};
|
||||
|
||||
const printUsage = () => {
|
||||
const header = readHeaderComment();
|
||||
console.log(header);
|
||||
};
|
||||
|
||||
/** Print the usage block from this file's own header comment. */
|
||||
const readHeaderComment = () => {
|
||||
try {
|
||||
const self = fs.readFileSync(new URL(import.meta.url), 'utf8');
|
||||
const usage = self.slice(self.indexOf(' * Usage:'), self.indexOf(' */'));
|
||||
return usage.split('\n').map(l => l.replace(/^ \* ?/, '')).join('\n');
|
||||
} catch {
|
||||
return 'See the comment at the top of scripts/jd2-sync.ts';
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* JDownloader escapes nothing in crawljob values, so a stray newline would
|
||||
* silently split a property. Paths with spaces are fine as-is.
|
||||
*/
|
||||
const sanitise = (value: string) => value.replace(/[\r\n]+/g, ' ').trim();
|
||||
|
||||
/**
|
||||
* Instagram usernames are 1–30 characters of letters, digits, dots and
|
||||
* underscores. Archive roots also collect directories that aren't profiles at
|
||||
* all — tool output, exports from other services — and pointing a crawl at
|
||||
* those spends requests on instagram.com to be told the profile doesn't exist.
|
||||
* That's the exact traffic worth not spending.
|
||||
*/
|
||||
const USERNAME_RE = /^[A-Za-z0-9._]{1,30}$/;
|
||||
|
||||
/** Directory names to skip, from `.jd2ignore` in the archive root. */
|
||||
const readIgnoreFile = (archives: string): string[] => {
|
||||
try {
|
||||
return fs.readFileSync(path.join(archives, '.jd2ignore'), 'utf8')
|
||||
.split('\n').map(l => l.trim()).filter(l => l && !l.startsWith('#'));
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
};
|
||||
|
||||
interface Job {
|
||||
user: string;
|
||||
kind: 'posts' | 'reels';
|
||||
url: string;
|
||||
packageName: string;
|
||||
downloadFolder: string;
|
||||
fileCount: number | null;
|
||||
}
|
||||
|
||||
const buildJobs = (opts: Options): Job[] => {
|
||||
const dirNames = fs.readdirSync(opts.archives, { withFileTypes: true })
|
||||
.filter(e => e.isDirectory() && !/^[.@_]/.test(e.name))
|
||||
.map(e => e.name);
|
||||
|
||||
const groups = groupArchiveDirectories(dirNames);
|
||||
const jobs: Job[] = [];
|
||||
const skipped: string[] = [];
|
||||
|
||||
for (const name of readIgnoreFile(opts.archives)) opts.skip.add(name);
|
||||
|
||||
const countFiles = (dir: string): number | null => {
|
||||
try {
|
||||
return fs.readdirSync(path.join(opts.archives, dir)).length;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
// JDownloader must be given the path *it* can see, which differs from the
|
||||
// scan path whenever the archive lives on a share.
|
||||
const downloadFolderFor = (dir: string) =>
|
||||
opts.downloadBase.includes('\\')
|
||||
? `${opts.downloadBase.replace(/\\$/, '')}\\${dir}`
|
||||
: path.posix.join(opts.downloadBase, dir);
|
||||
|
||||
for (const [user, sources] of [...groups].sort(([a], [b]) => a.localeCompare(b))) {
|
||||
if (opts.users.length && !opts.users.includes(user)) continue;
|
||||
|
||||
if (opts.skip.has(user)) { skipped.push(`${user} (ignored)`); continue; }
|
||||
if (!USERNAME_RE.test(user)) { skipped.push(`${user} (not a username)`); continue; }
|
||||
|
||||
const has = (kind: ArchiveSource['kind']) => sources.find(s => s.kind === kind);
|
||||
const base = has('posts');
|
||||
if (!base) continue; // sidecar-only group: nothing sensible to point a URL at
|
||||
|
||||
jobs.push({
|
||||
user, kind: 'posts',
|
||||
url: `https://www.instagram.com/${encodeURIComponent(user)}/`,
|
||||
packageName: base.dir,
|
||||
downloadFolder: downloadFolderFor(base.dir),
|
||||
fileCount: countFiles(base.dir),
|
||||
});
|
||||
|
||||
const reels = has('reels');
|
||||
if (reels || opts.allReels) {
|
||||
const dir = reels?.dir ?? `${user} - reels`;
|
||||
jobs.push({
|
||||
user, kind: 'reels',
|
||||
url: `https://www.instagram.com/${encodeURIComponent(user)}/reels/`,
|
||||
packageName: dir,
|
||||
downloadFolder: downloadFolderFor(dir),
|
||||
fileCount: reels ? countFiles(dir) : null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (skipped.length) {
|
||||
console.error(`Skipped ${skipped.length} director${skipped.length === 1 ? 'y' : 'ies'}:`);
|
||||
for (const s of skipped) console.error(` - ${s}`);
|
||||
console.error('');
|
||||
}
|
||||
|
||||
return jobs;
|
||||
};
|
||||
|
||||
const renderCrawljob = (jobs: Job[], opts: Options): string =>
|
||||
jobs.map(job => [
|
||||
`text=${sanitise(job.url)}`,
|
||||
`packageName=${sanitise(job.packageName)}`,
|
||||
`downloadFolder=${sanitise(job.downloadFolder)}`,
|
||||
`chunks=${opts.chunks}`,
|
||||
// Without this a Packagizer rule can override downloadFolder and scatter
|
||||
// files away from the directory the viewer reads.
|
||||
'overwritePackagizerEnabled=TRUE',
|
||||
`autoStart=${opts.autoStart ? 'TRUE' : 'FALSE'}`,
|
||||
`autoConfirm=${opts.autoStart ? 'TRUE' : 'FALSE'}`,
|
||||
'enabled=TRUE',
|
||||
`comment=instaarchive jd2-sync (${job.kind})`,
|
||||
].join('\n')).join('\n->NEW ENTRY<-\n');
|
||||
|
||||
const main = () => {
|
||||
const opts = parseArgs(process.argv.slice(2));
|
||||
const jobs = buildJobs(opts);
|
||||
|
||||
if (!jobs.length) {
|
||||
console.error('No profiles matched.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.error(`Archive root : ${opts.archives}`);
|
||||
console.error(`JD sees root : ${opts.downloadBase}`);
|
||||
console.error(`Jobs : ${jobs.length} (${new Set(jobs.map(j => j.user)).size} profiles)\n`);
|
||||
for (const job of jobs) {
|
||||
const count = job.fileCount === null ? 'new' : `${job.fileCount} files`;
|
||||
console.error(` ${job.kind.padEnd(5)} ${job.user.padEnd(24)} -> ${job.packageName} (${count})`);
|
||||
}
|
||||
console.error('');
|
||||
|
||||
const body = renderCrawljob(jobs, opts);
|
||||
|
||||
if (opts.dryRun || !opts.out) {
|
||||
console.log(body);
|
||||
return;
|
||||
}
|
||||
|
||||
fs.mkdirSync(opts.out, { recursive: true });
|
||||
const file = path.join(opts.out, `instaarchive-${new Date().toISOString().replace(/[:.]/g, '-')}.crawljob`);
|
||||
fs.writeFileSync(file, body, 'utf8');
|
||||
console.error(`Wrote ${file}`);
|
||||
console.error(opts.autoStart
|
||||
? 'Downloads will start automatically.'
|
||||
: 'Links land in the LinkGrabber for review; start them when ready.');
|
||||
};
|
||||
|
||||
main();
|
||||
@@ -1,231 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Tests for the request-budget logic in gdl-sync.py.
|
||||
|
||||
python3 -m unittest discover -s scripts -p 'test_*.py'
|
||||
|
||||
Deliberately stdlib-only, so it runs anywhere the sync itself runs. What is
|
||||
covered here is the part that decides whether to spend a request — the part
|
||||
whose absence got the archive's Instagram account suspended.
|
||||
"""
|
||||
|
||||
import datetime as dt
|
||||
import importlib.util
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"gdl_sync", Path(__file__).with_name("gdl-sync.py"))
|
||||
gdl = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["gdl_sync"] = gdl
|
||||
_spec.loader.exec_module(gdl)
|
||||
|
||||
NOW = dt.datetime(2026, 8, 18, 12, 0, tzinfo=dt.timezone.utc)
|
||||
NOW_TS = NOW.timestamp()
|
||||
|
||||
|
||||
def ago(hours: float) -> str:
|
||||
return (NOW - dt.timedelta(hours=hours)).isoformat()
|
||||
|
||||
|
||||
class SourceSelection(unittest.TestCase):
|
||||
def test_only_stories_is_a_single_cheap_source(self):
|
||||
srcs = gdl.Profile("u").sources(gdl.STORIES_ONLY)
|
||||
self.assertEqual([s.kind for s in srcs], ["stories"])
|
||||
self.assertEqual(srcs[0].directory, "story - u")
|
||||
|
||||
def test_full_sync_covers_every_surface(self):
|
||||
srcs = gdl.Profile("u").sources(set(gdl.ALL_KINDS))
|
||||
self.assertEqual([s.kind for s in srcs], list(gdl.ALL_KINDS))
|
||||
|
||||
def test_reels_and_stories_go_to_their_own_directories(self):
|
||||
by_kind = {s.kind: s for s in gdl.Profile("u").sources(set(gdl.ALL_KINDS))}
|
||||
self.assertEqual(by_kind["posts"].directory, "u")
|
||||
self.assertEqual(by_kind["reels"].directory, "u - reels")
|
||||
# Highlights derive their directory from the title mid-extraction.
|
||||
self.assertEqual(by_kind["highlights"].directory, "")
|
||||
|
||||
|
||||
class PlanSource(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.state = gdl.SyncState(Path(self.tmp.name) / "state.json")
|
||||
self.posts = gdl.Profile("u").sources({"posts"})[0]
|
||||
self.stories = gdl.Profile("u").sources({"stories"})[0]
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def test_first_run_seeds(self):
|
||||
fetch, seed, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
|
||||
self.assertTrue(fetch)
|
||||
self.assertTrue(seed)
|
||||
|
||||
def test_seeding_happens_only_once(self):
|
||||
self.state.mark_seeded(self.posts.url, ago(720))
|
||||
fetch, seed, reason = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
|
||||
self.assertTrue(fetch)
|
||||
self.assertFalse(seed, "a seeded source must never be re-probed")
|
||||
self.assertIn("already seeded", reason)
|
||||
|
||||
def test_stories_never_seed(self):
|
||||
# A story cannot be in the archive before it is fetched, so probing
|
||||
# would double the cost of the cheapest surface for no benefit.
|
||||
_, seed, reason = gdl.plan_source(self.stories, self.state, NOW_TS, 20)
|
||||
self.assertFalse(seed)
|
||||
self.assertIn("no seed", reason)
|
||||
|
||||
def test_recent_fetch_is_refused(self):
|
||||
self.state.mark_fetched(self.posts.url, ago(3))
|
||||
fetch, _, reason = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
|
||||
self.assertFalse(fetch)
|
||||
self.assertIn("under the", reason)
|
||||
|
||||
def test_an_old_fetch_is_allowed_again(self):
|
||||
self.state.mark_fetched(self.posts.url, ago(30))
|
||||
fetch, _, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
|
||||
self.assertTrue(fetch)
|
||||
|
||||
def test_daily_stories_pass_a_20h_floor(self):
|
||||
# The cadence this is built for: once a day, every day.
|
||||
self.state.mark_fetched(self.stories.url, ago(24))
|
||||
fetch, _, _ = gdl.plan_source(self.stories, self.state, NOW_TS, 20)
|
||||
self.assertTrue(fetch)
|
||||
|
||||
def test_force_disables_the_floor(self):
|
||||
self.state.mark_fetched(self.posts.url, ago(1))
|
||||
fetch, _, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 0.0)
|
||||
self.assertTrue(fetch)
|
||||
|
||||
def test_the_aborted_run_scenario(self):
|
||||
"""
|
||||
Yesterday's failure: a run died mid-way and the restart re-enumerated
|
||||
every profile. Seeded-but-not-fetched must not re-probe.
|
||||
"""
|
||||
self.state.mark_seeded(self.posts.url, ago(0.5))
|
||||
fetch, seed, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
|
||||
self.assertTrue(fetch, "the fetch still needs to happen")
|
||||
self.assertFalse(seed, "but the listing pass must not be paid for twice")
|
||||
|
||||
|
||||
class StatePersistence(unittest.TestCase):
|
||||
def test_state_survives_a_reload(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
path = Path(d) / "state.json"
|
||||
a = gdl.SyncState(path)
|
||||
a.mark_seeded("https://x/", ago(1))
|
||||
a.mark_fetched("https://x/", ago(1))
|
||||
a.save()
|
||||
b = gdl.SyncState(path)
|
||||
self.assertFalse(b.needs_seed("https://x/"))
|
||||
self.assertEqual(b.last_fetch("https://x/"), ago(1))
|
||||
|
||||
def test_a_corrupt_state_file_never_blocks_a_sync(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
path = Path(d) / "state.json"
|
||||
path.write_text("{ not json")
|
||||
self.assertTrue(gdl.SyncState(path).needs_seed("https://x/"))
|
||||
|
||||
|
||||
class ProbeCaching(unittest.TestCase):
|
||||
def test_fresh_entries_are_reused_and_stale_ones_are_not(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
cache = gdl.ProbeCache(Path(d) / "p.json", ttl_hours=24)
|
||||
cache.put("https://x/", [{"shortcode": "A", "post_shortcode": "A",
|
||||
"num": 1, "media_id": "1"}], ago(1))
|
||||
self.assertEqual(len(cache.get("https://x/", NOW_TS)), 1)
|
||||
|
||||
cache.put("https://y/", [{"shortcode": "B", "post_shortcode": "B",
|
||||
"num": 1, "media_id": "2"}], ago(48))
|
||||
self.assertIsNone(cache.get("https://y/", NOW_TS))
|
||||
|
||||
def test_cache_keeps_only_the_fields_seeding_needs(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
path = Path(d) / "p.json"
|
||||
cache = gdl.ProbeCache(path, ttl_hours=24)
|
||||
cache.put("https://x/", [{"shortcode": "A", "post_shortcode": "A",
|
||||
"num": 1, "media_id": "1",
|
||||
"description": "x" * 5000}], ago(0))
|
||||
cache.save()
|
||||
self.assertNotIn("description", path.read_text())
|
||||
|
||||
def test_a_miss_is_reported_rather_than_guessed(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
cache = gdl.ProbeCache(Path(d) / "p.json", ttl_hours=24)
|
||||
self.assertIsNone(cache.get("https://never-seen/", NOW_TS))
|
||||
|
||||
|
||||
class Seeding(unittest.TestCase):
|
||||
"""The bug that seeded 5 of 2275: matching the wrong shortcode field."""
|
||||
|
||||
def test_posts_are_keyed_by_post_shortcode(self):
|
||||
item = {"shortcode": "childcode", "post_shortcode": "POSTCODE",
|
||||
"num": 2, "media_id": "9"}
|
||||
self.assertEqual(gdl.live_key(item, "posts"), ("POSTCODE", 2))
|
||||
|
||||
def test_stories_are_keyed_by_the_per_item_shortcode(self):
|
||||
item = {"shortcode": "ITEMCODE", "post_shortcode": "reelid",
|
||||
"num": 3, "media_id": "9"}
|
||||
self.assertEqual(gdl.live_key(item, "stories"), ("ITEMCODE", 1))
|
||||
self.assertEqual(gdl.live_key(item, "highlights"), ("ITEMCODE", 1))
|
||||
|
||||
def test_index_existing_normalises_a_missing_index_to_one(self):
|
||||
held = gdl.index_existing([
|
||||
"u/2023-04-19_u - ABC.mp4",
|
||||
"u/2023-04-12_u - DEF - 3.jpg",
|
||||
"u/2023-04-12_u - DEF.txt", # sidecars are not media
|
||||
"u/2023-04-12_u - DEF.json",
|
||||
])
|
||||
self.assertEqual(held, {("ABC", 1), ("DEF", 3)})
|
||||
|
||||
def test_seeding_marks_only_what_is_already_held(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
db = Path(d) / "a.db"
|
||||
live = [
|
||||
{"post_shortcode": "HELD", "shortcode": "x", "num": 1, "media_id": "11"},
|
||||
{"post_shortcode": "NEW", "shortcode": "y", "num": 1, "media_id": "22"},
|
||||
]
|
||||
n = gdl.seed_archive_db(db, {("HELD", 1)}, live, "posts")
|
||||
self.assertEqual(n, 1)
|
||||
import sqlite3
|
||||
rows = {r[0] for r in sqlite3.connect(db).execute(
|
||||
"SELECT entry FROM archive")}
|
||||
self.assertEqual(rows, {"instagram11"})
|
||||
|
||||
|
||||
class Publishing(unittest.TestCase):
|
||||
def test_publish_only_ever_adds(self):
|
||||
cmd = gdl.rsync_command(Path("/stage"), "host:/archives", dry_run=False)
|
||||
self.assertIn("--ignore-existing", cmd)
|
||||
self.assertNotIn("--delete", cmd)
|
||||
|
||||
def test_tooling_files_are_excluded_from_the_archive(self):
|
||||
cmd = " ".join(gdl.rsync_command(Path("/stage"), "/dest", dry_run=True))
|
||||
for pattern in ("gdl-sync*.json", "*.db"):
|
||||
self.assertIn(pattern, cmd)
|
||||
self.assertIn("--dry-run", cmd)
|
||||
|
||||
|
||||
class UrlsFile(unittest.TestCase):
|
||||
def test_reads_every_form_a_person_might_paste(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
p = Path(d) / "urls.txt"
|
||||
p.write_text(
|
||||
"# comment\n"
|
||||
"https://www.instagram.com/a/\n"
|
||||
"https://instagram.com/b\n"
|
||||
"www.instagram.com/c/\n"
|
||||
"d\n"
|
||||
" e # trailing\n"
|
||||
"\n"
|
||||
"https://www.instagram.com/a/\n" # duplicate
|
||||
"https://www.instagram.com/p/ABC123/\n" # a post, not a profile
|
||||
"not a username\n")
|
||||
self.assertEqual(gdl.read_urls_file(p), ["a", "b", "c", "d", "e"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
+55
-1
@@ -73,6 +73,17 @@ export default function App() {
|
||||
*/
|
||||
const initialRouteRef = useRef(parseRoute(window.location.pathname, window.location.search));
|
||||
|
||||
/**
|
||||
* Back-button support for the post view. Every other URL change
|
||||
* (`replaceState`s the tab/archive) is intentionally NOT pushed — only
|
||||
* opening a post gets its own history entry, matching Instagram's own
|
||||
* back-button behaviour: Back closes the post instead of leaving the app.
|
||||
*/
|
||||
const pushedPostRef = useRef(false);
|
||||
/** Set while reacting to a popstate, so the URL-sync effect below does not
|
||||
* try to push/replace/back() again for a change the browser already made. */
|
||||
const suppressNextSyncRef = useRef(false);
|
||||
|
||||
const isMobile = useIsMobile();
|
||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
||||
const profilePicInputRef = useRef<HTMLInputElement>(null);
|
||||
@@ -367,6 +378,13 @@ export default function App() {
|
||||
// loader below is waiting to read.
|
||||
if (!hasInitialLoaded) return;
|
||||
|
||||
// Consumed exactly once per popstate, regardless of what happens below:
|
||||
// a popstate-driven change often already matches the URL (the browser
|
||||
// already moved the pointer), which used to leave this flag stuck true
|
||||
// and silently no-op the NEXT real close/open until something reset it.
|
||||
const wasPopState = suppressNextSyncRef.current;
|
||||
suppressNextSyncRef.current = false;
|
||||
|
||||
const archive = currentArchive?.name ?? (allPosts.length > 0 ? username : null) ?? null;
|
||||
const nextPath = buildPath({
|
||||
archive,
|
||||
@@ -374,12 +392,48 @@ export default function App() {
|
||||
post: selectedPost ? postSlug(selectedPost) : null,
|
||||
});
|
||||
|
||||
if (nextPath !== window.location.pathname + window.location.search) {
|
||||
if (nextPath === window.location.pathname + window.location.search) return;
|
||||
if (wasPopState) return; // the browser already navigated; nothing to add
|
||||
|
||||
console.log(`[Permalink] Updating URL to: ${nextPath}`);
|
||||
|
||||
if (selectedPost && !pushedPostRef.current) {
|
||||
// Opening a post: push, so Back closes it instead of leaving the app.
|
||||
window.history.pushState(null, '', nextPath);
|
||||
pushedPostRef.current = true;
|
||||
} else if (!selectedPost && pushedPostRef.current) {
|
||||
// Closing a post that was pushed for: consume that entry rather than
|
||||
// piling a new one on top of it, so Back still means "one step".
|
||||
pushedPostRef.current = false;
|
||||
window.history.back();
|
||||
} else {
|
||||
window.history.replaceState(null, '', nextPath);
|
||||
}
|
||||
}, [hasInitialLoaded, currentArchive?.name, username, allPosts.length, activeTab, selectedPost?.id]);
|
||||
|
||||
/**
|
||||
* Back/forward support for the post view. Only a post push (above) ever
|
||||
* creates an entry, so this only ever needs to open or close a post —
|
||||
* never re-derive the tab or archive, which stayed on replaceState.
|
||||
*/
|
||||
useEffect(() => {
|
||||
const onPopState = () => {
|
||||
suppressNextSyncRef.current = true;
|
||||
const route = parseRoute(window.location.pathname, window.location.search);
|
||||
const post = route.post ? findPostBySlug(allPosts, route.post) : null;
|
||||
if (post) {
|
||||
setActiveTab(tabForSource(post.source));
|
||||
setSelectedPost(post);
|
||||
pushedPostRef.current = true; // forward navigation can land back here
|
||||
} else {
|
||||
setSelectedPost(null);
|
||||
pushedPostRef.current = false;
|
||||
}
|
||||
};
|
||||
window.addEventListener('popstate', onPopState);
|
||||
return () => window.removeEventListener('popstate', onPopState);
|
||||
}, [allPosts]);
|
||||
|
||||
useEffect(() => {
|
||||
if (hasInitialLoaded) return;
|
||||
|
||||
|
||||
Reference in New Issue
Block a user