Compare commits
58
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6b8410da09 | ||
|
|
de8145447a | ||
|
|
96ea0cc1d0 | ||
|
|
882296b1c0 | ||
|
|
4f8b0021c6 | ||
|
|
dcd8f2ef1d | ||
|
|
5d5dea10c8 | ||
|
|
816fa970b5 | ||
|
|
53b1f80e1d | ||
|
|
89bd5346db | ||
|
|
b153a49bbb | ||
|
|
8b053b4b2e | ||
|
|
d1fa1a8d2f | ||
|
|
b9ece021d4 | ||
|
|
f300b8d9f5 | ||
|
|
e57be521a2 | ||
|
|
792b834cbe | ||
|
|
106d3f6691 | ||
|
|
92a4ada3c2 | ||
|
|
c54f8d5b09 | ||
|
|
877d21ff1f | ||
|
|
0cff91edae | ||
|
|
0b4b20e0ff | ||
|
|
c0b6b6cf3e | ||
|
|
ab7099220a | ||
|
|
e6663657b0 | ||
|
|
24ff2727c8 | ||
|
|
f089e49e84 | ||
|
|
1d86fa3583 | ||
|
|
0ba7a0d9ad | ||
|
|
899e8dfbbb | ||
|
|
8809b7794b | ||
|
|
8ec2c07b3f | ||
|
|
3c50be286e | ||
|
|
7fc31fed94 | ||
|
|
0ed4cf292c | ||
|
|
f51b82e37a | ||
|
|
e539677ec3 | ||
|
|
418997779c | ||
|
|
3896506600 | ||
|
|
bc0f4c230d | ||
|
|
ad2f6ec083 | ||
|
|
bdcedab2da | ||
|
|
c196f40cfd | ||
|
|
2d0501f124 | ||
|
|
be54b42e9e | ||
|
|
5842a773d3 | ||
|
|
eb66250511 | ||
|
|
c34aaae63c | ||
|
|
5bd6534adb | ||
|
|
c86ee00e9b | ||
|
|
ab0001efbc | ||
|
|
afacc61888 | ||
|
|
ee87e9b523 | ||
|
|
0e30c8b8b0 | ||
|
|
c8e3026ffd | ||
|
|
e4c53364c3 | ||
|
|
eb8adf9dc2 |
@@ -15,9 +15,6 @@ InstaArchive Viewer is a React 19 + Vite 6 PWA for browsing archived Instagram d
|
|||||||
- `npm run lint` — type-check only (`tsc --noEmit`)
|
- `npm run lint` — type-check only (`tsc --noEmit`)
|
||||||
- `npm test` / `npm run test:watch` — vitest
|
- `npm test` / `npm run test:watch` — vitest
|
||||||
- `npx vitest run src/lib/archive-patterns.test.ts` — a single test file
|
- `npx vitest run src/lib/archive-patterns.test.ts` — a single test file
|
||||||
- `npm run jd2 -- --archives <dir> --dry-run` — generate JDownloader `.crawljob`
|
|
||||||
files for every profile on disk (see `scripts/jd2-sync.ts` and
|
|
||||||
`docs/jdownloader.md`)
|
|
||||||
|
|
||||||
Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not.
|
Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not.
|
||||||
|
|
||||||
@@ -37,10 +34,10 @@ Loading is unified behind the `ArchiveFile` interface (`src/types/index.ts`, imp
|
|||||||
An archive root holds one directory per profile plus *sidecars* that belong to it:
|
An archive root holds one directory per profile plus *sidecars* that belong to it:
|
||||||
|
|
||||||
```
|
```
|
||||||
0ct0ber19 -> posts (base)
|
4utumn07 -> posts (base)
|
||||||
0ct0ber19 - reels -> reels
|
4utumn07 - reels -> reels
|
||||||
story - 0ct0ber19 -> stories
|
story - 4utumn07 -> stories
|
||||||
story highlights - 0ct0ber19 - Heestory -> highlight "Heestory"
|
story highlights - 4utumn07 - Sunstory -> highlight "Sunstory"
|
||||||
```
|
```
|
||||||
|
|
||||||
`src/lib/archive-grouping.ts` (shared by server and tests) folds these into a single profile with a `sources` list. Sidecars never appear as standalone archives. Each file the server returns carries its `kind`, so the client routes posts / reels / story ring / highlight circles without re-deriving naming rules.
|
`src/lib/archive-grouping.ts` (shared by server and tests) folds these into a single profile with a `sources` list. Sidecars never appear as standalone archives. Each file the server returns carries its `kind`, so the client routes posts / reels / story ring / highlight circles without re-deriving naming rules.
|
||||||
@@ -72,7 +69,7 @@ Three different JSON shapes turn up as `.json`, so they are told apart structura
|
|||||||
| Instaloader `.json.xz` | GraphQL node under `node` / `__typename` |
|
| Instaloader `.json.xz` | GraphQL node under `node` / `__typename` |
|
||||||
| gallery-dl sidecar | flat, `post_shortcode` + `type`, none of the above |
|
| gallery-dl sidecar | flat, `post_shortcode` + `type`, none of the above |
|
||||||
|
|
||||||
The gallery-dl sidecar is the only source that states what a post *is*: its `type` (`post` / `reel` / `story` / `highlight`) is Instagram's own classification, so `post.isReel` set from it beats every fallback in `post-tabs.ts`. This matters — of the 781 items in `official_artms - reels`, the sidecars say only **360 are reels**; the other 421 are ordinary feed videos the clips endpoint returns via `include_feed_video`. Directory-based classification counted all 781.
|
The gallery-dl sidecar is the only source that states what a post *is*: its `type` (`post` / `reel` / `story` / `highlight`) is Instagram's own classification, so `post.isReel` set from it beats every fallback in `post-tabs.ts`. This matters — of the 781 items in `official_band - reels`, the sidecars say only **360 are reels**; the other 421 are ordinary feed videos the clips endpoint returns via `include_feed_video`. Directory-based classification counted all 781.
|
||||||
|
|
||||||
**Dates are ranked, not last-write-wins** (`src/lib/post-dates.ts`): sidecar (what Instagram reported) beats filename (what the fetcher wrote) beats mtime (when the file hit disk, and unrelated to when it was posted). Ties keep the incumbent. Several files describe one post and they are scanned in directory order, not in order of trustworthiness, so without the ranking the date was decided by whichever file came first. Only JDownloader highlights fall to mtime at all — `parseArchiveFilename` flags those via `dateFromMtime`.
|
**Dates are ranked, not last-write-wins** (`src/lib/post-dates.ts`): sidecar (what Instagram reported) beats filename (what the fetcher wrote) beats mtime (when the file hit disk, and unrelated to when it was posted). Ties keep the incumbent. Several files describe one post and they are scanned in directory order, not in order of trustworthiness, so without the ranking the date was decided by whichever file came first. Only JDownloader highlights fall to mtime at all — `parseArchiveFilename` flags those via `dateFromMtime`.
|
||||||
|
|
||||||
@@ -88,15 +85,15 @@ Images over 1MiB are downscaled in a Web Worker via `OffscreenCanvas`. The queue
|
|||||||
|
|
||||||
### Profile tabs (`src/lib/post-tabs.ts`)
|
### Profile tabs (`src/lib/post-tabs.ts`)
|
||||||
|
|
||||||
The grid holds **everything**, reels included, and the Reels tab is a *filtered view* of that same set. Only the Reels tab filters. The tabs were mutually exclusive until v1.7.0, which hid a lot: 1100 of `loonatheworld`'s 3813 posts and 533 of `for.heejin`'s 1225 never appeared in the grid at all.
|
The grid holds **everything**, reels included, and the Reels tab is a *filtered view* of that same set. Only the Reels tab filters. The tabs were mutually exclusive until v1.7.0, which hid a lot: 1100 of `groupfandom`'s 3813 posts and 533 of `for.member`'s 1225 never appeared in the grid at all.
|
||||||
|
|
||||||
This *approximates* Instagram rather than matching it. Instagram's grid includes a reel only if the creator shared it to feed — a per-post choice, measured live on 2026-08-16: `official_artms` had 21 reels in its first 34 grid tiles, `0ct0ber19` just 1 in 214. That flag appears nowhere in an archive (JD2 stores no metadata, and Instaloader's `product_type` says what a post *is*, not whether it was shared to feed), so showing everything is the closest reachable behaviour. Instagram's "N posts" counter equals its grid, which is why the header counts `postsForTab(allPosts, 'posts')` and not `allPosts` — the raw list still holds both copies of a double-fetched post.
|
This *approximates* Instagram rather than matching it. Instagram's grid includes a reel only if the creator shared it to feed — a per-post choice, measured live on 2026-08-16: `official_band` had 21 reels in its first 34 grid tiles, `4utumn07` just 1 in 214. That flag appears nowhere in an archive (JD2 stores no metadata, and Instaloader's `product_type` says what a post *is*, not whether it was shared to feed), so showing everything is the closest reachable behaviour. Instagram's "N posts" counter equals its grid, which is why the header counts `postsForTab(allPosts, 'posts')` and not `allPosts` — the raw list still holds both copies of a double-fetched post.
|
||||||
|
|
||||||
When checking the live site, note that grid reels link to `/reel/<code>/` while ordinary posts link to `/<user>/p/<code>/`. Matching only `/p/` silently drops every reel, which once produced a confident and completely wrong conclusion that Instagram never shows reels in the grid.
|
When checking the live site, note that grid reels link to `/reel/<code>/` while ordinary posts link to `/<user>/p/<code>/`. Matching only `/p/` silently drops every reel, which once produced a confident and completely wrong conclusion that Instagram never shows reels in the grid.
|
||||||
|
|
||||||
Deciding *what is a reel* has no good answer for most archives. Instagram's own marker is `product_type` on the post's GraphQL node (`clips` = reel, `feed` = ordinary feed video, `igtv`, `story`) — `__typename` is `GraphVideo` for all three, and aspect ratio does not separate them either. But:
|
Deciding *what is a reel* has no good answer for most archives. Instagram's own marker is `product_type` on the post's GraphQL node (`clips` = reel, `feed` = ordinary feed video, `igtv`, `story`) — `__typename` is `GraphVideo` for all three, and aspect ratio does not separate them either. But:
|
||||||
|
|
||||||
- Only Instaloader archives carry that metadata, and only newer captures. A survey of `gibiofficial` found `product_type` on 1101 of 5919 sidecars, and just **2** posts marked `clips`.
|
- Only Instaloader archives carry that metadata, and only newer captures. A survey of `hazelofficial` found `product_type` on 1101 of 5919 sidecars, and just **2** posts marked `clips`.
|
||||||
- JDownloader archives carry none at all — media plus a `.txt` holding the bare caption.
|
- JDownloader archives carry none at all — media plus a `.txt` holding the bare caption.
|
||||||
|
|
||||||
So the viewer believes a `- reels` sidecar directory when one exists, and otherwise falls back to treating a lone video as a reel. **The fallback is a guess**: it cannot tell a reel from a feed video or an old IGTV upload, and it misses videos inside carousels.
|
So the viewer believes a `- reels` sidecar directory when one exists, and otherwise falls back to treating a lone video as a reel. **The fallback is a guess**: it cannot tell a reel from a feed video or an old IGTV upload, and it misses videos inside carousels.
|
||||||
|
|||||||
+129
-11
@@ -1,10 +1,10 @@
|
|||||||
# gallery-dl — a CLI replacement for JDownloader2
|
# gallery-dl — a CLI replacement for JDownloader2
|
||||||
|
|
||||||
Status: **design + verified config.** `scripts/gdl-sync.py` is a skeleton; no
|
Status: **in production.** All six ARTMS profiles are synced with
|
||||||
profile has been migrated yet.
|
`scripts/gdl-sync.py`; JD2 is no longer used for them.
|
||||||
|
|
||||||
Everything below was measured against the live site and the real archive on
|
Everything below was measured against the live site and the real archive on
|
||||||
2026-08-16, not inferred from documentation.
|
2026-08-16 and 2026-08-20, not inferred from documentation.
|
||||||
|
|
||||||
## Why gallery-dl and not a hand-rolled script
|
## Why gallery-dl and not a hand-rolled script
|
||||||
|
|
||||||
@@ -20,6 +20,12 @@ The account used for all of the below was suspended the same day this tooling
|
|||||||
was built, for "activity that doesn't follow our Community Standards on spam".
|
was built, for "activity that doesn't follow our Community Standards on spam".
|
||||||
The fetching was not the expensive part. **Verification was.**
|
The fetching was not the expensive part. **Verification was.**
|
||||||
|
|
||||||
|
**It was restored, and synced normally again on 2026-08-20** — a full run
|
||||||
|
across all six profiles with 0 failures and 0 CDN 429s. That is not evidence
|
||||||
|
the limits were imagined; it is one data point on a restored account that has
|
||||||
|
been treated carefully since. Everything below still applies, and the budget is
|
||||||
|
still per session rather than per command.
|
||||||
|
|
||||||
What was actually spent against `instagram.com` in a few hours, from one
|
What was actually spent against `instagram.com` in a few hours, from one
|
||||||
session and one IP:
|
session and one IP:
|
||||||
|
|
||||||
@@ -359,6 +365,71 @@ The one trap, which silently seeds almost nothing if you get it backwards:
|
|||||||
|
|
||||||
`live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
|
`live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
|
||||||
|
|
||||||
|
### The skip-archive saves the CDN, not `instagram.com`
|
||||||
|
|
||||||
|
Worth being exact about, because the two costs land on different surfaces and
|
||||||
|
only one of them bans accounts:
|
||||||
|
|
||||||
|
| what | which surface | scales with |
|
||||||
|
|---|---|---|
|
||||||
|
| downloading media | `scontent-*.cdninstagram.com` | how much is **new** |
|
||||||
|
| enumerating the profile to find it | `instagram.com` | how **big** the profile is |
|
||||||
|
|
||||||
|
The skip-archive suppresses the first. It does nothing about the second, so a
|
||||||
|
2275-post profile costs ~76 pages of pagination every run, forever, whether it
|
||||||
|
has three new posts or none. Seeding (above) saved a *second* full pass, not
|
||||||
|
the first.
|
||||||
|
|
||||||
|
Measured on the 2026-08-20 run, from sidecar write times in staging — free,
|
||||||
|
since the run was paying for the listing anyway:
|
||||||
|
|
||||||
|
```
|
||||||
|
1787248852 2026-08-19 … DcOeoVxkthi new, +0s
|
||||||
|
1787248944 2026-08-18 … DcLpfoJCZtp new, +92s
|
||||||
|
1787249058 2026-08-17 … DcIlGbxCUk0 new, +114s
|
||||||
|
1787249162 2026-07-24 … DbKr1TxlPSX ┐ all one second: nothing
|
||||||
|
1787249162 2026-08-15 … DcD-FdBCYGm ┘ downloaded, sidecars only
|
||||||
|
```
|
||||||
|
|
||||||
|
Three posts took ~100s each; the remaining 2272 were enumeration with nothing
|
||||||
|
to show for it.
|
||||||
|
|
||||||
|
**Pinned posts do not break early abort.** Test case 16 previously claimed
|
||||||
|
`0ct0ber19` returns its 3 pinned posts out of date order — that is true of the
|
||||||
|
*web grid*, but the REST `/posts/` listing came back strictly
|
||||||
|
reverse-chronological, newest first, no hoisting. That matters because
|
||||||
|
front-loaded old posts are the one thing that would make `skip: abort:N`
|
||||||
|
dangerous: it would trip on them and abort before reaching anything new.
|
||||||
|
|
||||||
|
So `skip: abort:N` is viable, and cuts ~420 requests per run to ~40-60:
|
||||||
|
|
||||||
|
| surface | live items | pages | with `abort:50` |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| posts, 6 profiles | 11,248 | ~377 | ~12 |
|
||||||
|
| reels, 6 profiles | 1,080 | ~24 | ~8 |
|
||||||
|
| stories + highlights | — | ~20 | ~20 |
|
||||||
|
|
||||||
|
N counts consecutive skipped **files**, not posts, so it must clear the largest
|
||||||
|
already-held carousel — `DcD-FdBCYGm` alone is 22 media. 50 is comfortable; 5
|
||||||
|
would not be.
|
||||||
|
|
||||||
|
**The tradeoff is edited carousels.** Test case 15 is a post that gained items
|
||||||
|
after we archived it, and only a full enumeration finds those. Suggested
|
||||||
|
policy: `abort:50` for routine runs, a full sweep occasionally.
|
||||||
|
|
||||||
|
Measured the same day, resuming a stopped run with `--abort 50`:
|
||||||
|
|
||||||
|
| source | live items | enumerated |
|
||||||
|
|---|---:|---:|
|
||||||
|
| `cher_ryppo` posts | 2,151 | **7** |
|
||||||
|
| `cher_ryppo` reels | 92 | 53 |
|
||||||
|
|
||||||
|
One page instead of 72, and every new post was still caught. The 7 is roughly
|
||||||
|
3 new posts plus 4 already-held carousels making up the 50 skipped files.
|
||||||
|
Reels need 53 because they are single-media, so 50 consecutive skips really is
|
||||||
|
50 reels — another reminder that N counts files, and that the same N behaves
|
||||||
|
very differently on a carousel-heavy surface than on a reels tab.
|
||||||
|
|
||||||
## Publishing
|
## Publishing
|
||||||
|
|
||||||
The fetch host stages to local disk and rsyncs afterwards. `rsync
|
The fetch host stages to local disk and rsyncs afterwards. `rsync
|
||||||
@@ -372,15 +443,46 @@ never reaches the archive half-written.
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
In use. `withaseul` has been fetched and published to the live archive — 322
|
In use for all six ARTMS profiles.
|
||||||
files added (74 media, 241 `.json`, 7 `.txt`), nothing overwritten or deleted.
|
|
||||||
Of the 74 new media files, **zero** duplicated media already held under a
|
`withaseul` first — 322 files added (74 media, 241 `.json`, 7 `.txt`), nothing
|
||||||
different name, which is the check that says JD2 and gallery-dl naming really
|
overwritten or deleted. Of the 74 new media, **zero** duplicated media already
|
||||||
do converge.
|
held under a different name, which is the check that says JD2 and gallery-dl
|
||||||
|
naming really do converge.
|
||||||
|
|
||||||
|
**2026-08-20**, the first full incremental sync, four days after the previous
|
||||||
|
one. 184 new media, 299 files published, 0 failures and **0 CDN 429s**:
|
||||||
|
|
||||||
|
| profile | posts | reels | stories | files added |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| 0ct0ber19 | 58 | 2 | 4 | +77 |
|
||||||
|
| official_artms | 12 | — | 2 | +85 |
|
||||||
|
| cher_ryppo | 41 | 1 | 8 | +63 |
|
||||||
|
| zindoriyam | 23 | — | 4 | +35 |
|
||||||
|
| kimxxlip | 16 | — | 2 | +23 |
|
||||||
|
| withaseul | 10 | — | — | +16 |
|
||||||
|
|
||||||
|
The 20 story items are the part that could not have been recovered later.
|
||||||
|
|
||||||
|
Two things made it cheap, and both are worth keeping:
|
||||||
|
|
||||||
|
- The archive DB was already seeded from the previous run, so `--min-interval`
|
||||||
|
and the recorded `seeded` state meant **no probe passes at all**. A state
|
||||||
|
file has to exist for this; if one is missing after a manual run, write it
|
||||||
|
rather than letting the tool re-seed 24 sources.
|
||||||
|
- `--abort 50` (see above) cut the remaining listing cost by roughly 85%.
|
||||||
|
|
||||||
|
The run was deliberately **stopped and resumed** halfway to pick up `--abort`.
|
||||||
|
That is safe precisely because of the state file: the 12 finished sources were
|
||||||
|
already marked `fetched`, so the 20h floor skipped them and only the remaining
|
||||||
|
12 re-ran. Stopping a run is cheap now; it was not before.
|
||||||
|
|
||||||
Published files land owned by the SSH user rather than `rslsync`. The viewer
|
Published files land owned by the SSH user rather than `rslsync`. The viewer
|
||||||
reads them fine (world-readable), but Resilio does not own what it syncs; worth
|
reads them fine (world-readable), but Resilio does not own what it syncs; worth
|
||||||
a `chown` if that ever matters.
|
a `chown` if that ever matters. This also makes **`rsync` exit 23**
|
||||||
|
("some files/attrs were not transferred") the *normal* outcome of a publish —
|
||||||
|
it is the failed `chown`, not lost data. Confirm by re-running the same rsync
|
||||||
|
with `--dry-run`: an empty file list means everything arrived.
|
||||||
|
|
||||||
The profiles to fetch live in `artms_account_links.txt` at the archive root,
|
The profiles to fetch live in `artms_account_links.txt` at the archive root,
|
||||||
passed with `--urls-file`.
|
passed with `--urls-file`.
|
||||||
@@ -426,7 +528,23 @@ Re-running with those recovered both videos and produced **0 failures and 0
|
|||||||
|
|
||||||
Without it, gallery-dl logs `Cannot import yt-dlp or youtube-dl` and falls back
|
Without it, gallery-dl logs `Cannot import yt-dlp or youtube-dl` and falls back
|
||||||
to a progressive URL for DASH videos. The fallback mostly works but is what the
|
to a progressive URL for DASH videos. The fallback mostly works but is what the
|
||||||
429s hit hardest. `pipx install yt-dlp` on the fetch host.
|
429s hit hardest.
|
||||||
|
|
||||||
|
**`pipx install yt-dlp` does not work** — it was the advice here until
|
||||||
|
2026-08-20, and it is wrong. It gives yt-dlp its own venv, so the binary lands
|
||||||
|
on `PATH` while gallery-dl, in a *different* venv, still cannot `import yt_dlp`.
|
||||||
|
The symptom is that everything looks installed and the log keeps saying
|
||||||
|
`Cannot import yt-dlp`. gallery-dl needs it importable, not runnable:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
pipx inject gallery-dl yt-dlp
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify by asking gallery-dl's own interpreter, not the shell:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
/home/matt/.local/share/pipx/venvs/gallery-dl/bin/python -c 'import yt_dlp'
|
||||||
|
```
|
||||||
|
|
||||||
## Known quirks
|
## Known quirks
|
||||||
|
|
||||||
@@ -477,7 +595,7 @@ Real subjects, all present in the archive today. See
|
|||||||
| 13 | empty caption | `CrdsY5CrSsO` | media written, `.txt` absent |
|
| 13 | empty caption | `CrdsY5CrSsO` | media written, `.txt` absent |
|
||||||
| 14 | deleted post | `C0TgI7sphfZ` | on disk, absent live — must not be removed |
|
| 14 | deleted post | `C0TgI7sphfZ` | on disk, absent live — must not be removed |
|
||||||
| 15 | edited carousel | `C7zG7-jJMlq` | 18 on disk, 8 live — must not be removed |
|
| 15 | edited carousel | `C7zG7-jJMlq` | 18 on disk, 8 live — must not be removed |
|
||||||
| 16 | pinned posts | `0ct0ber19` | 3 pinned, returned out of date order |
|
| 16 | pinned posts | `0ct0ber19` | REST listing is strictly reverse-chronological; see below |
|
||||||
| 17 | profile avatar | `0ct0ber19.jpg` | base dir, undated |
|
| 17 | profile avatar | `0ct0ber19.jpg` | base dir, undated |
|
||||||
|
|
||||||
Cases 14–16 are reconciliation, not naming: **a sync must never delete**, since
|
Cases 14–16 are reconciliation, not naming: **a sync must never delete**, since
|
||||||
|
|||||||
+14
-1
@@ -1,4 +1,17 @@
|
|||||||
# JDownloader2 — archive fetching quick reference
|
# JDownloader2 — archive fetching
|
||||||
|
|
||||||
|
> **This file lives only on the `tooling` branch.** `main` is published to
|
||||||
|
> GitHub and deliberately carries none of this — not the host details, not the
|
||||||
|
> IPs, and not the account names. `main`'s history was redacted on 2026-08-20;
|
||||||
|
> real names exist only here.
|
||||||
|
>
|
||||||
|
> There is no `npm run jd2` script — `package.json` and `CLAUDE.md` are kept
|
||||||
|
> byte-identical to `main` so that merging `main` into `tooling` never
|
||||||
|
> conflicts. Run the crawljob generator directly:
|
||||||
|
>
|
||||||
|
> ```sh
|
||||||
|
> npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run
|
||||||
|
> ```
|
||||||
|
|
||||||
How content gets into this archive, and why the setup is shaped the way it is.
|
How content gets into this archive, and why the setup is shaped the way it is.
|
||||||
|
|
||||||
|
|||||||
+1
-2
@@ -12,8 +12,7 @@
|
|||||||
"clean": "rm -rf dist",
|
"clean": "rm -rf dist",
|
||||||
"lint": "tsc --noEmit",
|
"lint": "tsc --noEmit",
|
||||||
"test": "vitest run",
|
"test": "vitest run",
|
||||||
"test:watch": "vitest",
|
"test:watch": "vitest"
|
||||||
"jd2": "tsx scripts/jd2-sync.ts"
|
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@tailwindcss/vite": "^4.1.14",
|
"@tailwindcss/vite": "^4.1.14",
|
||||||
|
|||||||
+28
-2
@@ -219,7 +219,7 @@ class ArchiveIndex:
|
|||||||
# --------------------------------------------------------------------------
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
def build_config(rate: str, sleep_request: list[float],
|
def build_config(rate: str, sleep_request: list[float],
|
||||||
sleep: list[float]) -> dict:
|
sleep: list[float], abort: int = 0) -> dict:
|
||||||
"""
|
"""
|
||||||
The config is generated rather than checked in so the safety-critical
|
The config is generated rather than checked in so the safety-critical
|
||||||
options cannot drift out of sync with the docs.
|
options cannot drift out of sync with the docs.
|
||||||
@@ -253,7 +253,26 @@ def build_config(rate: str, sleep_request: list[float],
|
|||||||
|
|
||||||
def post_like(stem: str) -> dict:
|
def post_like(stem: str) -> dict:
|
||||||
"""Naming for surfaces whose unit is a post (posts, reels)."""
|
"""Naming for surfaces whose unit is a post (posts, reels)."""
|
||||||
|
skip: dict = {}
|
||||||
|
if abort:
|
||||||
|
# Stop enumerating once `abort` consecutive files are already in
|
||||||
|
# the skip-archive. The listing pass -- not the downloading -- is
|
||||||
|
# what costs `instagram.com` requests, and it otherwise walks the
|
||||||
|
# whole profile every run to find three new posts.
|
||||||
|
#
|
||||||
|
# Safe here only because the REST listing is strictly
|
||||||
|
# reverse-chronological: the web grid hoists pinned posts to the
|
||||||
|
# front, but this endpoint does not (measured 2026-08-20), so old
|
||||||
|
# posts never appear before new ones.
|
||||||
|
#
|
||||||
|
# Counted in FILES, not posts, so it must clear the largest
|
||||||
|
# already-held carousel -- 22 media for one real post in this
|
||||||
|
# archive. It also means edited carousels (test case 15) stop
|
||||||
|
# being noticed, so a full sweep is still worth running
|
||||||
|
# occasionally.
|
||||||
|
skip["skip"] = f"abort:{abort}"
|
||||||
return {
|
return {
|
||||||
|
**skip,
|
||||||
# `sidecar_shortcode` is set only for carousels, so it is the
|
# `sidecar_shortcode` is set only for carousels, so it is the
|
||||||
# carousel discriminator. First matching condition wins.
|
# carousel discriminator. First matching condition wins.
|
||||||
"filename": {
|
"filename": {
|
||||||
@@ -666,6 +685,12 @@ def main() -> int:
|
|||||||
ap.add_argument("--max-sources", type=int, default=0, metavar="N",
|
ap.add_argument("--max-sources", type=int, default=0, metavar="N",
|
||||||
help="hard ceiling on sources touched in one run "
|
help="hard ceiling on sources touched in one run "
|
||||||
"(0 = no limit)")
|
"(0 = no limit)")
|
||||||
|
ap.add_argument("--abort", type=int, default=0, metavar="N",
|
||||||
|
help="stop enumerating posts/reels after N consecutive "
|
||||||
|
"already-archived FILES (0 = walk everything, the "
|
||||||
|
"default). 50 is a safe routine value; it cuts the "
|
||||||
|
"per-run listing cost by roughly 85%%, at the price "
|
||||||
|
"of no longer noticing edited carousels")
|
||||||
ap.add_argument("--probe-ttl", type=float, default=24.0, metavar="HOURS",
|
ap.add_argument("--probe-ttl", type=float, default=24.0, metavar="HOURS",
|
||||||
help="reuse cached listing results younger than this")
|
help="reuse cached listing results younger than this")
|
||||||
ap.add_argument("--force", action="store_true",
|
ap.add_argument("--force", action="store_true",
|
||||||
@@ -704,7 +729,8 @@ def main() -> int:
|
|||||||
else:
|
else:
|
||||||
selected = [Profile(p) for p in sorted(names)]
|
selected = [Profile(p) for p in sorted(names)]
|
||||||
|
|
||||||
config = build_config(args.rate, list(args.sleep_request), list(args.sleep))
|
config = build_config(args.rate, list(args.sleep_request),
|
||||||
|
list(args.sleep), args.abort)
|
||||||
args.staging.mkdir(parents=True, exist_ok=True)
|
args.staging.mkdir(parents=True, exist_ok=True)
|
||||||
# Deliberately a SIBLING of the staging directory, not inside it: staging is
|
# Deliberately a SIBLING of the staging directory, not inside it: staging is
|
||||||
# rsynced wholesale into the archive, and a dry run caught this file being
|
# rsynced wholesale into the archive, and a dry run caught this file being
|
||||||
|
|||||||
@@ -3,39 +3,39 @@ import { classifyDirectory, groupArchiveDirectories } from './archive-grouping';
|
|||||||
|
|
||||||
describe('classifyDirectory', () => {
|
describe('classifyDirectory', () => {
|
||||||
it('treats a bare profile directory as the base', () => {
|
it('treats a bare profile directory as the base', () => {
|
||||||
expect(classifyDirectory('0ct0ber19')).toEqual({
|
expect(classifyDirectory('4utumn07')).toEqual({
|
||||||
owner: '0ct0ber19',
|
owner: '4utumn07',
|
||||||
source: { kind: 'posts', dir: '0ct0ber19' },
|
source: { kind: 'posts', dir: '4utumn07' },
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
it('recognises a reels sidecar', () => {
|
it('recognises a reels sidecar', () => {
|
||||||
expect(classifyDirectory('0ct0ber19 - reels')).toEqual({
|
expect(classifyDirectory('4utumn07 - reels')).toEqual({
|
||||||
owner: '0ct0ber19',
|
owner: '4utumn07',
|
||||||
source: { kind: 'reels', dir: '0ct0ber19 - reels' },
|
source: { kind: 'reels', dir: '4utumn07 - reels' },
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
it('recognises a stories sidecar', () => {
|
it('recognises a stories sidecar', () => {
|
||||||
expect(classifyDirectory('story - cher_ryppo')).toEqual({
|
expect(classifyDirectory('story - dawn_petal')).toEqual({
|
||||||
owner: 'cher_ryppo',
|
owner: 'dawn_petal',
|
||||||
source: { kind: 'stories', dir: 'story - cher_ryppo' },
|
source: { kind: 'stories', dir: 'story - dawn_petal' },
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
it('splits highlight owner from title', () => {
|
it('splits highlight owner from title', () => {
|
||||||
const { owner, source } = classifyDirectory('story highlights - 0ct0ber19 - Heestory');
|
const { owner, source } = classifyDirectory('story highlights - 4utumn07 - Sunstory');
|
||||||
expect(owner).toBe('0ct0ber19');
|
expect(owner).toBe('4utumn07');
|
||||||
expect(source.kind).toBe('highlight');
|
expect(source.kind).toBe('highlight');
|
||||||
expect(source.title).toBe('Heestory');
|
expect(source.title).toBe('Sunstory');
|
||||||
});
|
});
|
||||||
|
|
||||||
it.each([
|
it.each([
|
||||||
['story highlights - theoldtaylorswiftinsta - 💙2014-1989 era', 'theoldtaylorswiftinsta', '💙2014-1989 era'],
|
['story highlights - theoldlyricmuseinsta - 💙1999-2005 era', 'theoldlyricmuseinsta', '💙1999-2005 era'],
|
||||||
['story highlights - heejin_theworld - [Dall]', 'heejin_theworld', '[Dall]'],
|
['story highlights - member_theworld - [Bracket]', 'member_theworld', '[Bracket]'],
|
||||||
['story highlights - official_artms - Cosmo Schedule', 'official_artms', 'Cosmo Schedule'],
|
['story highlights - official_band - Tour Schedule', 'official_band', 'Tour Schedule'],
|
||||||
['story highlights - 0ct0ber19 - Drawheeing⠀', '0ct0ber19', 'Drawheeing⠀'],
|
['story highlights - 4utumn07 - Sketching⠀', '4utumn07', 'Sketching⠀'],
|
||||||
['story highlights - official_artms - G.C.I', 'official_artms', 'G.C.I'],
|
['story highlights - official_band - A.B.C', 'official_band', 'A.B.C'],
|
||||||
])('handles real-world title %s', (dir, owner, title) => {
|
])('handles real-world title %s', (dir, owner, title) => {
|
||||||
const result = classifyDirectory(dir);
|
const result = classifyDirectory(dir);
|
||||||
expect(result.owner).toBe(owner);
|
expect(result.owner).toBe(owner);
|
||||||
@@ -57,25 +57,25 @@ describe('classifyDirectory', () => {
|
|||||||
|
|
||||||
describe('groupArchiveDirectories', () => {
|
describe('groupArchiveDirectories', () => {
|
||||||
const dirs = [
|
const dirs = [
|
||||||
'0ct0ber19',
|
'4utumn07',
|
||||||
'0ct0ber19 - reels',
|
'4utumn07 - reels',
|
||||||
'story - 0ct0ber19',
|
'story - 4utumn07',
|
||||||
'story highlights - 0ct0ber19 - Heestory',
|
'story highlights - 4utumn07 - Sunstory',
|
||||||
'story highlights - 0ct0ber19 - Drawheeing⠀',
|
'story highlights - 4utumn07 - Sketching⠀',
|
||||||
'carlyraejepsen',
|
'kestrelsings',
|
||||||
];
|
];
|
||||||
|
|
||||||
it('folds sidecars into their base profile', () => {
|
it('folds sidecars into their base profile', () => {
|
||||||
const groups = groupArchiveDirectories(dirs);
|
const groups = groupArchiveDirectories(dirs);
|
||||||
expect([...groups.keys()].sort()).toEqual(['0ct0ber19', 'carlyraejepsen']);
|
expect([...groups.keys()].sort()).toEqual(['4utumn07', 'kestrelsings']);
|
||||||
expect(groups.get('0ct0ber19')).toHaveLength(5);
|
expect(groups.get('4utumn07')).toHaveLength(5);
|
||||||
expect(groups.get('carlyraejepsen')).toHaveLength(1);
|
expect(groups.get('kestrelsings')).toHaveLength(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('orders sources posts, reels, stories, then highlights by title', () => {
|
it('orders sources posts, reels, stories, then highlights by title', () => {
|
||||||
const sources = groupArchiveDirectories(dirs).get('0ct0ber19')!;
|
const sources = groupArchiveDirectories(dirs).get('4utumn07')!;
|
||||||
expect(sources.map(s => s.kind)).toEqual(['posts', 'reels', 'stories', 'highlight', 'highlight']);
|
expect(sources.map(s => s.kind)).toEqual(['posts', 'reels', 'stories', 'highlight', 'highlight']);
|
||||||
expect(sources.slice(3).map(s => s.title)).toEqual(['Drawheeing⠀', 'Heestory']);
|
expect(sources.slice(3).map(s => s.title)).toEqual(['Sketching⠀', 'Sunstory']);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('still groups a sidecar whose base profile is missing', () => {
|
it('still groups a sidecar whose base profile is missing', () => {
|
||||||
|
|||||||
@@ -18,10 +18,10 @@ export interface ArchiveSource {
|
|||||||
/**
|
/**
|
||||||
* Sidecar directories sit next to the profile directory they belong to:
|
* Sidecar directories sit next to the profile directory they belong to:
|
||||||
*
|
*
|
||||||
* 0ct0ber19 -> posts (base)
|
* 4utumn07 -> posts (base)
|
||||||
* 0ct0ber19 - reels -> reels
|
* 4utumn07 - reels -> reels
|
||||||
* story - 0ct0ber19 -> stories
|
* story - 4utumn07 -> stories
|
||||||
* story highlights - 0ct0ber19 - Heestory -> highlight "Heestory"
|
* story highlights - 4utumn07 - Sunstory -> highlight "Sunstory"
|
||||||
*
|
*
|
||||||
* Instagram usernames cannot contain spaces, so matching the username as a
|
* Instagram usernames cannot contain spaces, so matching the username as a
|
||||||
* run of non-space characters reliably separates it from a highlight title
|
* run of non-space characters reliably separates it from a highlight title
|
||||||
|
|||||||
@@ -14,11 +14,11 @@ describe('isSystemDirectory', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it.each([
|
it.each([
|
||||||
'0ct0ber19',
|
'4utumn07',
|
||||||
'0ct0ber19 - reels',
|
'4utumn07 - reels',
|
||||||
'story - cher_ryppo',
|
'story - dawn_petal',
|
||||||
'story highlights - official_artms - G.C.I',
|
'story highlights - official_band - A.B.C',
|
||||||
'story highlights - theoldtaylorswiftinsta - 💙2014-1989 era',
|
'story highlights - theoldlyricmuseinsta - 💙1999-2005 era',
|
||||||
'Heejin_Bubble heejinmedia',
|
'Heejin_Bubble heejinmedia',
|
||||||
'gallery-dl',
|
'gallery-dl',
|
||||||
'posts',
|
'posts',
|
||||||
|
|||||||
@@ -3,10 +3,10 @@ import { canonicalItemId, parseArchiveFilename, scopedPostId } from './archive-p
|
|||||||
|
|
||||||
describe('parseArchiveFilename — Instagram export format', () => {
|
describe('parseArchiveFilename — Instagram export format', () => {
|
||||||
it('parses a single-image post', () => {
|
it('parses a single-image post', () => {
|
||||||
expect(parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM.mp4')).toEqual({
|
expect(parseArchiveFilename('2023-04-19_4utumn07 - CrORBIcJJbM.mp4')).toEqual({
|
||||||
postId: 'CrORBIcJJbM',
|
postId: 'CrORBIcJJbM',
|
||||||
date: '2023-04-19',
|
date: '2023-04-19',
|
||||||
username: '0ct0ber19',
|
username: '4utumn07',
|
||||||
index: 1,
|
index: 1,
|
||||||
ext: 'mp4',
|
ext: 'mp4',
|
||||||
isStory: false,
|
isStory: false,
|
||||||
@@ -15,7 +15,7 @@ describe('parseArchiveFilename — Instagram export format', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('parses a carousel slide index', () => {
|
it('parses a carousel slide index', () => {
|
||||||
const parsed = parseArchiveFilename('2023-04-12_0ct0ber19 - Cq8LrxSJAJE - 3.jpg');
|
const parsed = parseArchiveFilename('2023-04-12_4utumn07 - Cq8LrxSJAJE - 3.jpg');
|
||||||
expect(parsed).toMatchObject({ postId: 'Cq8LrxSJAJE', index: 3, ext: 'jpg' });
|
expect(parsed).toMatchObject({ postId: 'Cq8LrxSJAJE', index: 3, ext: 'jpg' });
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -27,7 +27,7 @@ describe('parseArchiveFilename — Instagram export format', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('parses caption sidecar files', () => {
|
it('parses caption sidecar files', () => {
|
||||||
expect(parseArchiveFilename('2023-04-12_0ct0ber19 - Cq8LrxSJAJE.txt')).toMatchObject({
|
expect(parseArchiveFilename('2023-04-12_4utumn07 - Cq8LrxSJAJE.txt')).toMatchObject({
|
||||||
postId: 'Cq8LrxSJAJE',
|
postId: 'Cq8LrxSJAJE',
|
||||||
ext: 'txt',
|
ext: 'txt',
|
||||||
});
|
});
|
||||||
@@ -39,8 +39,8 @@ describe('parseArchiveFilename — Instagram export format', () => {
|
|||||||
|
|
||||||
it('parses the story sidecar layout (date_user - N - shortcode)', () => {
|
it('parses the story sidecar layout (date_user - N - shortcode)', () => {
|
||||||
// Files in `story - <user>` carry a per-day ordinal before the shortcode.
|
// Files in `story - <user>` carry a per-day ordinal before the shortcode.
|
||||||
const parsed = parseArchiveFilename('2025-10-26_0ct0ber19 - 2 - DQRuDx9iW5Q.jpg', 'stories');
|
const parsed = parseArchiveFilename('2025-10-26_4utumn07 - 2 - DQRuDx9iW5Q.jpg', 'stories');
|
||||||
expect(parsed).toMatchObject({ date: '2025-10-26', username: '0ct0ber19', ext: 'jpg' });
|
expect(parsed).toMatchObject({ date: '2025-10-26', username: '4utumn07', ext: 'jpg' });
|
||||||
expect(parsed!.postId).toContain('DQRuDx9iW5Q');
|
expect(parsed!.postId).toContain('DQRuDx9iW5Q');
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -71,9 +71,9 @@ describe('parseArchiveFilename — Instaloader format', () => {
|
|||||||
|
|
||||||
describe('parseArchiveFilename — story highlights', () => {
|
describe('parseArchiveFilename — story highlights', () => {
|
||||||
it('parses the dateless highlight layout', () => {
|
it('parses the dateless highlight layout', () => {
|
||||||
expect(parseArchiveFilename('0ct0ber19 - C5dQPEYpd9W.mp4', 'highlight')).toMatchObject({
|
expect(parseArchiveFilename('4utumn07 - C5dQPEYpd9W.mp4', 'highlight')).toMatchObject({
|
||||||
postId: 'C5dQPEYpd9W',
|
postId: 'C5dQPEYpd9W',
|
||||||
username: '0ct0ber19',
|
username: '4utumn07',
|
||||||
ext: 'mp4',
|
ext: 'mp4',
|
||||||
isStory: false,
|
isStory: false,
|
||||||
});
|
});
|
||||||
@@ -95,7 +95,7 @@ describe('parseArchiveFilename — story highlights', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
describe('parseArchiveFilename — non-matching files', () => {
|
describe('parseArchiveFilename — non-matching files', () => {
|
||||||
it.each(['0ct0ber19.jpg', 'profile_pic.jpg', 'README.md', 'no-separator.png'])(
|
it.each(['4utumn07.jpg', 'profile_pic.jpg', 'README.md', 'no-separator.png'])(
|
||||||
'returns null for %s',
|
'returns null for %s',
|
||||||
name => expect(parseArchiveFilename(name)).toBeNull(),
|
name => expect(parseArchiveFilename(name)).toBeNull(),
|
||||||
);
|
);
|
||||||
@@ -107,8 +107,8 @@ describe('scopedPostId', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('namespaces sidecar ids by directory', () => {
|
it('namespaces sidecar ids by directory', () => {
|
||||||
expect(scopedPostId('C5dQ', 'highlight', 'story highlights - u - Heestory'))
|
expect(scopedPostId('C5dQ', 'highlight', 'story highlights - u - Sunstory'))
|
||||||
.toBe('story highlights - u - Heestory/C5dQ');
|
.toBe('story highlights - u - Sunstory/C5dQ');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('keeps the same shortcode distinct across sources', () => {
|
it('keeps the same shortcode distinct across sources', () => {
|
||||||
@@ -125,8 +125,8 @@ describe('scopedPostId', () => {
|
|||||||
*/
|
*/
|
||||||
describe('gallery-dl / JDownloader naming interop', () => {
|
describe('gallery-dl / JDownloader naming interop', () => {
|
||||||
it('treats a single-media post the same with or without an index', () => {
|
it('treats a single-media post the same with or without an index', () => {
|
||||||
const jd2 = parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM.mp4')!;
|
const jd2 = parseArchiveFilename('2023-04-19_4utumn07 - CrORBIcJJbM.mp4')!;
|
||||||
const gdl = parseArchiveFilename('2023-04-19_0ct0ber19 - CrORBIcJJbM - 1.mp4')!;
|
const gdl = parseArchiveFilename('2023-04-19_4utumn07 - CrORBIcJJbM - 1.mp4')!;
|
||||||
expect(jd2.postId).toBe(gdl.postId);
|
expect(jd2.postId).toBe(gdl.postId);
|
||||||
expect(jd2.index).toBe(gdl.index);
|
expect(jd2.index).toBe(gdl.index);
|
||||||
expect(jd2.index).toBe(1);
|
expect(jd2.index).toBe(1);
|
||||||
@@ -136,21 +136,21 @@ describe('gallery-dl / JDownloader naming interop', () => {
|
|||||||
// JD2 pads to the width of the media count (10+ items -> "01"), and
|
// JD2 pads to the width of the media count (10+ items -> "01"), and
|
||||||
// gallery-dl's count can be one higher, so the same post may be padded
|
// gallery-dl's count can be one higher, so the same post may be padded
|
||||||
// by one tool and not the other.
|
// by one tool and not the other.
|
||||||
expect(parseArchiveFilename('2024-04-17_0ct0ber19 - C53YPQzp7Wj - 09.jpg')!.index).toBe(9);
|
expect(parseArchiveFilename('2024-04-17_4utumn07 - C53YPQzp7Wj - 09.jpg')!.index).toBe(9);
|
||||||
expect(parseArchiveFilename('2024-04-17_0ct0ber19 - C53YPQzp7Wj - 9.jpg')!.index).toBe(9);
|
expect(parseArchiveFilename('2024-04-17_4utumn07 - C53YPQzp7Wj - 9.jpg')!.index).toBe(9);
|
||||||
expect(parseArchiveFilename('2023-11-03_0ct0ber19 - CzM8Uf6B6H_ - 01.jpg')!.index).toBe(1);
|
expect(parseArchiveFilename('2023-11-03_4utumn07 - CzM8Uf6B6H_ - 01.jpg')!.index).toBe(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('reads a gallery-dl story name, which carries a per-item shortcode', () => {
|
it('reads a gallery-dl story name, which carries a per-item shortcode', () => {
|
||||||
const p = parseArchiveFilename('2026-08-16_official_artms - DcF9OyhBJ1H.jpg', 'stories')!;
|
const p = parseArchiveFilename('2026-08-16_official_band - DcF9OyhBJ1H.jpg', 'stories')!;
|
||||||
expect(p.postId).toBe('DcF9OyhBJ1H');
|
expect(p.postId).toBe('DcF9OyhBJ1H');
|
||||||
expect(p.date).toBe('2026-08-16');
|
expect(p.date).toBe('2026-08-16');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('gives a dated highlight a real date instead of the mtime fallback', () => {
|
it('gives a dated highlight a real date instead of the mtime fallback', () => {
|
||||||
const mtime = Date.parse('2026-08-17T00:00:00Z');
|
const mtime = Date.parse('2026-08-17T00:00:00Z');
|
||||||
const undated = parseArchiveFilename('0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
|
const undated = parseArchiveFilename('4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
|
||||||
const dated = parseArchiveFilename('2024-08-04_0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
|
const dated = parseArchiveFilename('2024-08-04_4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
|
||||||
// Same item either way, so re-fetching cannot split it into two posts.
|
// Same item either way, so re-fetching cannot split it into two posts.
|
||||||
expect(dated.postId).toBe(undated.postId);
|
expect(dated.postId).toBe(undated.postId);
|
||||||
expect(undated.date).toBe('2026-08-17');
|
expect(undated.date).toBe('2026-08-17');
|
||||||
@@ -167,13 +167,13 @@ describe('dateFromMtime', () => {
|
|||||||
const mtime = Date.parse('2026-08-17T00:00:00Z');
|
const mtime = Date.parse('2026-08-17T00:00:00Z');
|
||||||
|
|
||||||
it('flags an undated highlight name as mtime-dated', () => {
|
it('flags an undated highlight name as mtime-dated', () => {
|
||||||
const p = parseArchiveFilename('0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
|
const p = parseArchiveFilename('4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
|
||||||
expect(p.date).toBe('2026-08-17');
|
expect(p.date).toBe('2026-08-17');
|
||||||
expect(p.dateFromMtime).toBe(true);
|
expect(p.dateFromMtime).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('does not flag a highlight that carries its own date', () => {
|
it('does not flag a highlight that carries its own date', () => {
|
||||||
const p = parseArchiveFilename('2024-08-04_0ct0ber19 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
|
const p = parseArchiveFilename('2024-08-04_4utumn07 - C-IImhvpFuk.jpg', 'highlight', mtime)!;
|
||||||
expect(p.date).toBe('2024-08-04');
|
expect(p.date).toBe('2024-08-04');
|
||||||
expect(p.dateFromMtime).toBe(false);
|
expect(p.dateFromMtime).toBe(false);
|
||||||
});
|
});
|
||||||
@@ -184,7 +184,7 @@ describe('dateFromMtime', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('leaves the date empty rather than guessing when no mtime is given', () => {
|
it('leaves the date empty rather than guessing when no mtime is given', () => {
|
||||||
const p = parseArchiveFilename('0ct0ber19 - C-IImhvpFuk.jpg', 'highlight')!;
|
const p = parseArchiveFilename('4utumn07 - C-IImhvpFuk.jpg', 'highlight')!;
|
||||||
expect(p.date).toBe('');
|
expect(p.date).toBe('');
|
||||||
expect(p.dateFromMtime).toBe(false);
|
expect(p.dateFromMtime).toBe(false);
|
||||||
});
|
});
|
||||||
@@ -196,9 +196,9 @@ describe('dateFromMtime', () => {
|
|||||||
*/
|
*/
|
||||||
describe('canonicalItemId', () => {
|
describe('canonicalItemId', () => {
|
||||||
it('collapses the two highlight naming conventions onto one id', () => {
|
it('collapses the two highlight naming conventions onto one id', () => {
|
||||||
const dir = 'story highlights - 0ct0ber19 - Heestory';
|
const dir = 'story highlights - 4utumn07 - Sunstory';
|
||||||
const undated = parseArchiveFilename('0ct0ber19 - C5dQPEYpd9W.mp4', 'highlight', 1)!;
|
const undated = parseArchiveFilename('4utumn07 - C5dQPEYpd9W.mp4', 'highlight', 1)!;
|
||||||
const dated = parseArchiveFilename('2024-04-07_0ct0ber19 - 01 - C5dQPEYpd9W.mp4', 'highlight')!;
|
const dated = parseArchiveFilename('2024-04-07_4utumn07 - 01 - C5dQPEYpd9W.mp4', 'highlight')!;
|
||||||
expect(scopedPostId(dated.postId, 'highlight', dir))
|
expect(scopedPostId(dated.postId, 'highlight', dir))
|
||||||
.toBe(scopedPostId(undated.postId, 'highlight', dir));
|
.toBe(scopedPostId(undated.postId, 'highlight', dir));
|
||||||
});
|
});
|
||||||
@@ -216,7 +216,7 @@ describe('canonicalItemId', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('leaves a shortcode that merely starts with digits alone', () => {
|
it('leaves a shortcode that merely starts with digits alone', () => {
|
||||||
expect(canonicalItemId('0ct0ber19')).toBe('0ct0ber19');
|
expect(canonicalItemId('4utumn07')).toBe('4utumn07');
|
||||||
expect(canonicalItemId('C5dQPEYpd9W')).toBe('C5dQPEYpd9W');
|
expect(canonicalItemId('C5dQPEYpd9W')).toBe('C5dQPEYpd9W');
|
||||||
expect(canonicalItemId('12345')).toBe('12345');
|
expect(canonicalItemId('12345')).toBe('12345');
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -7,13 +7,13 @@ import {
|
|||||||
const REEL: GalleryDlSidecar = {
|
const REEL: GalleryDlSidecar = {
|
||||||
post_shortcode: 'Db-lNCoib9m', post_id: '3962768346034323302', type: 'reel',
|
post_shortcode: 'Db-lNCoib9m', post_id: '3962768346034323302', type: 'reel',
|
||||||
date: '2026-08-13 11:00:44', post_date: '2026-08-13 11:00:44',
|
date: '2026-08-13 11:00:44', post_date: '2026-08-13 11:00:44',
|
||||||
username: 'official_artms', fullname: 'Official ARTMS',
|
username: 'official_band', fullname: 'Official ARTMS',
|
||||||
description: 'Dancing in the spotlight', count: 1, likes: 22914,
|
description: 'Dancing in the spotlight', count: 1, likes: 22914,
|
||||||
};
|
};
|
||||||
const FEED_VIDEO: GalleryDlSidecar = { ...REEL, post_shortcode: 'DbdG9L9jU4m', type: 'post', count: 2 };
|
const FEED_VIDEO: GalleryDlSidecar = { ...REEL, post_shortcode: 'DbdG9L9jU4m', type: 'post', count: 2 };
|
||||||
const HIGHLIGHT: GalleryDlSidecar = {
|
const HIGHLIGHT: GalleryDlSidecar = {
|
||||||
post_shortcode: 'BATVdRZi_3', post_id: '18099435932626935', type: 'highlight',
|
post_shortcode: 'BATVdRZi_3', post_id: '18099435932626935', type: 'highlight',
|
||||||
date: '2026-08-08 16:22:09', username: 'official_artms', count: 154,
|
date: '2026-08-08 16:22:09', username: 'official_band', count: 154,
|
||||||
};
|
};
|
||||||
|
|
||||||
describe('isGalleryDlSidecar', () => {
|
describe('isGalleryDlSidecar', () => {
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ import { Tab } from './routing';
|
|||||||
/**
|
/**
|
||||||
* The shortcode shared by every copy of a post, regardless of which source
|
* The shortcode shared by every copy of a post, regardless of which source
|
||||||
* directory it came from. Sidecar ids are directory-scoped
|
* directory it came from. Sidecar ids are directory-scoped
|
||||||
* (`0ct0ber19 - reels/Cq8LrxSJAJE`); the trailing segment is the shortcode.
|
* (`4utumn07 - reels/Cq8LrxSJAJE`); the trailing segment is the shortcode.
|
||||||
*/
|
*/
|
||||||
const shortcode = (post: Post): string => post.id.split('/').pop() ?? post.id;
|
const shortcode = (post: Post): string => post.id.split('/').pop() ?? post.id;
|
||||||
|
|
||||||
|
|||||||
+16
-16
@@ -12,21 +12,21 @@ describe('parseRoute', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('reads a profile', () => {
|
it('reads a profile', () => {
|
||||||
expect(parseRoute('/0ct0ber19/')).toEqual({ archive: '0ct0ber19', tab: 'posts', post: null });
|
expect(parseRoute('/4utumn07/')).toEqual({ archive: '4utumn07', tab: 'posts', post: null });
|
||||||
});
|
});
|
||||||
|
|
||||||
it('reads a profile without a trailing slash', () => {
|
it('reads a profile without a trailing slash', () => {
|
||||||
expect(parseRoute('/0ct0ber19')).toEqual({ archive: '0ct0ber19', tab: 'posts', post: null });
|
expect(parseRoute('/4utumn07')).toEqual({ archive: '4utumn07', tab: 'posts', post: null });
|
||||||
});
|
});
|
||||||
|
|
||||||
it('reads a tab', () => {
|
it('reads a tab', () => {
|
||||||
expect(parseRoute('/0ct0ber19/reels/').tab).toBe('reels');
|
expect(parseRoute('/4utumn07/reels/').tab).toBe('reels');
|
||||||
expect(parseRoute('/0ct0ber19/saved/').tab).toBe('saved');
|
expect(parseRoute('/4utumn07/saved/').tab).toBe('saved');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('reads a post in Instagram form', () => {
|
it('reads a post in Instagram form', () => {
|
||||||
expect(parseRoute('/0ct0ber19/p/Db5tIoRCcvm/')).toEqual({
|
expect(parseRoute('/4utumn07/p/Db5tIoRCcvm/')).toEqual({
|
||||||
archive: '0ct0ber19', tab: 'posts', post: 'Db5tIoRCcvm',
|
archive: '4utumn07', tab: 'posts', post: 'Db5tIoRCcvm',
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -41,8 +41,8 @@ describe('parseRoute', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('still understands the legacy query form', () => {
|
it('still understands the legacy query form', () => {
|
||||||
expect(parseRoute('/', '?a=0ct0ber19&t=reels&p=ABC')).toEqual({
|
expect(parseRoute('/', '?a=4utumn07&t=reels&p=ABC')).toEqual({
|
||||||
archive: '0ct0ber19', tab: 'reels', post: 'ABC',
|
archive: '4utumn07', tab: 'reels', post: 'ABC',
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -54,9 +54,9 @@ describe('parseRoute', () => {
|
|||||||
describe('buildPath', () => {
|
describe('buildPath', () => {
|
||||||
it.each([
|
it.each([
|
||||||
[{ archive: null, tab: 'posts', post: null }, '/'],
|
[{ archive: null, tab: 'posts', post: null }, '/'],
|
||||||
[{ archive: '0ct0ber19', tab: 'posts', post: null }, '/0ct0ber19/'],
|
[{ archive: '4utumn07', tab: 'posts', post: null }, '/4utumn07/'],
|
||||||
[{ archive: '0ct0ber19', tab: 'reels', post: null }, '/0ct0ber19/reels/'],
|
[{ archive: '4utumn07', tab: 'reels', post: null }, '/4utumn07/reels/'],
|
||||||
[{ archive: '0ct0ber19', tab: 'posts', post: 'Db5tIoRCcvm' }, '/0ct0ber19/p/Db5tIoRCcvm/'],
|
[{ archive: '4utumn07', tab: 'posts', post: 'Db5tIoRCcvm' }, '/4utumn07/p/Db5tIoRCcvm/'],
|
||||||
] as const)('builds %j', (route, expected) => {
|
] as const)('builds %j', (route, expected) => {
|
||||||
expect(buildPath(route as any)).toBe(expected);
|
expect(buildPath(route as any)).toBe(expected);
|
||||||
});
|
});
|
||||||
@@ -71,8 +71,8 @@ describe('buildPath', () => {
|
|||||||
|
|
||||||
it('round-trips through parseRoute', () => {
|
it('round-trips through parseRoute', () => {
|
||||||
for (const route of [
|
for (const route of [
|
||||||
{ archive: '0ct0ber19', tab: 'posts' as const, post: null },
|
{ archive: '4utumn07', tab: 'posts' as const, post: null },
|
||||||
{ archive: '0ct0ber19', tab: 'reels' as const, post: null },
|
{ archive: '4utumn07', tab: 'reels' as const, post: null },
|
||||||
{ archive: 'Heejin_Bubble heejinmedia', tab: 'posts' as const, post: null },
|
{ archive: 'Heejin_Bubble heejinmedia', tab: 'posts' as const, post: null },
|
||||||
]) {
|
]) {
|
||||||
expect(parseRoute(buildPath(route))).toEqual(route);
|
expect(parseRoute(buildPath(route))).toEqual(route);
|
||||||
@@ -86,12 +86,12 @@ describe('postSlug / findPostBySlug', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('strips the sidecar directory from the slug', () => {
|
it('strips the sidecar directory from the slug', () => {
|
||||||
expect(postSlug(post('story highlights - u - Heestory/C5dQPEYpd9W'))).toBe('C5dQPEYpd9W');
|
expect(postSlug(post('story highlights - u - Sunstory/C5dQPEYpd9W'))).toBe('C5dQPEYpd9W');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('resolves a slug back to its post', () => {
|
it('resolves a slug back to its post', () => {
|
||||||
const posts = [post('AAA'), post('0ct0ber19 - reels/BBB', 'reels')];
|
const posts = [post('AAA'), post('4utumn07 - reels/BBB', 'reels')];
|
||||||
expect(findPostBySlug(posts, 'BBB')?.id).toBe('0ct0ber19 - reels/BBB');
|
expect(findPostBySlug(posts, 'BBB')?.id).toBe('4utumn07 - reels/BBB');
|
||||||
expect(findPostBySlug(posts, 'AAA')?.id).toBe('AAA');
|
expect(findPostBySlug(posts, 'AAA')?.id).toBe('AAA');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user