docs: record the first full incremental sync, and what made it cheap

All six ARTMS profiles, four days after the previous run: 184 new media, 299
files published, 0 failures and 0 CDN 429s. The header still claimed the
script was a skeleton with nothing migrated, which stopped being true a while
ago.

Notes the two things that made it cheap — an already-seeded archive DB, so no
probe passes ran at all, and --abort 50 — and that the run was deliberately
stopped and resumed midway to pick up the new flag. That is only safe because
the state file had already marked the finished sources as fetched, so the 20h
floor skipped them. Stopping a run used to mean repeating it.

Also records that rsync exit 23 is the NORMAL outcome of a publish: the SSH
user cannot chown to rslsync, so attrs fail while data arrives intact. The
check is a --dry-run re-run returning an empty file list, which is what
verified this one (+299 files, matching rsync exactly).

The account was restored; noting that without pretending it licenses more
traffic.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF

.gitignore

@@ -9,5 +9,3 @@ coverage/
 !.env.example
 _sample-archives
 _gemini-plans
-__pycache__/
-*.pyc

docs/gallery-dl.md

@@ -1,10 +1,10 @@
 # gallery-dl — a CLI replacement for JDownloader2
 
-Status: **design + verified config.** `scripts/gdl-sync.py` is a skeleton; no
-profile has been migrated yet.
+Status: **in production.** All six ARTMS profiles are synced with
+`scripts/gdl-sync.py`; JD2 is no longer used for them.
 
 Everything below was measured against the live site and the real archive on
-2026-08-16, not inferred from documentation.
+2026-08-16 and 2026-08-20, not inferred from documentation.
 
 ## Why gallery-dl and not a hand-rolled script
 
@@ -20,6 +20,12 @@ The account used for all of the below was suspended the same day this tooling
 was built, for "activity that doesn't follow our Community Standards on spam".
 The fetching was not the expensive part. **Verification was.**
 
+**It was restored, and synced normally again on 2026-08-20** — a full run
+across all six profiles with 0 failures and 0 CDN 429s. That is not evidence
+the limits were imagined; it is one data point on a restored account that has
+been treated carefully since. Everything below still applies, and the budget is
+still per session rather than per command.
+
 What was actually spent against `instagram.com` in a few hours, from one
 session and one IP:
 
@@ -359,6 +365,71 @@ The one trap, which silently seeds almost nothing if you get it backwards:
 
 `live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
 
+### The skip-archive saves the CDN, not `instagram.com`
+
+Worth being exact about, because the two costs land on different surfaces and
+only one of them bans accounts:
+
+| what | which surface | scales with |
+|---|---|---|
+| downloading media | `scontent-*.cdninstagram.com` | how much is **new** |
+| enumerating the profile to find it | `instagram.com` | how **big** the profile is |
+
+The skip-archive suppresses the first. It does nothing about the second, so a
+2275-post profile costs ~76 pages of pagination every run, forever, whether it
+has three new posts or none. Seeding (above) saved a *second* full pass, not
+the first.
+
+Measured on the 2026-08-20 run, from sidecar write times in staging — free,
+since the run was paying for the listing anyway:
+
+```
+1787248852   2026-08-19 … DcOeoVxkthi     new, +0s
+1787248944   2026-08-18 … DcLpfoJCZtp     new, +92s
+1787249058   2026-08-17 … DcIlGbxCUk0     new, +114s
+1787249162   2026-07-24 … DbKr1TxlPSX     ┐ all one second: nothing
+1787249162   2026-08-15 … DcD-FdBCYGm     ┘ downloaded, sidecars only
+```
+
+Three posts took ~100s each; the remaining 2272 were enumeration with nothing
+to show for it.
+
+**Pinned posts do not break early abort.** Test case 16 previously claimed
+`0ct0ber19` returns its 3 pinned posts out of date order — that is true of the
+*web grid*, but the REST `/posts/` listing came back strictly
+reverse-chronological, newest first, no hoisting. That matters because
+front-loaded old posts are the one thing that would make `skip: abort:N`
+dangerous: it would trip on them and abort before reaching anything new.
+
+So `skip: abort:N` is viable, and cuts ~420 requests per run to ~40-60:
+
+| surface | live items | pages | with `abort:50` |
+|---|---:|---:|---:|
+| posts, 6 profiles | 11,248 | ~377 | ~12 |
+| reels, 6 profiles | 1,080 | ~24 | ~8 |
+| stories + highlights | — | ~20 | ~20 |
+
+N counts consecutive skipped **files**, not posts, so it must clear the largest
+already-held carousel — `DcD-FdBCYGm` alone is 22 media. 50 is comfortable; 5
+would not be.
+
+**The tradeoff is edited carousels.** Test case 15 is a post that gained items
+after we archived it, and only a full enumeration finds those. Suggested
+policy: `abort:50` for routine runs, a full sweep occasionally.
+
+Measured the same day, resuming a stopped run with `--abort 50`:
+
+| source | live items | enumerated |
+|---|---:|---:|
+| `cher_ryppo` posts | 2,151 | **7** |
+| `cher_ryppo` reels | 92 | 53 |
+
+One page instead of 72, and every new post was still caught. The 7 is roughly
+3 new posts plus 4 already-held carousels making up the 50 skipped files.
+Reels need 53 because they are single-media, so 50 consecutive skips really is
+50 reels — another reminder that N counts files, and that the same N behaves
+very differently on a carousel-heavy surface than on a reels tab.
+
 ## Publishing
 
 The fetch host stages to local disk and rsyncs afterwards. `rsync
@@ -372,15 +443,46 @@ never reaches the archive half-written.
 
 ## Status
 
-In use. `withaseul` has been fetched and published to the live archive — 322
-files added (74 media, 241 `.json`, 7 `.txt`), nothing overwritten or deleted.
-Of the 74 new media files, **zero** duplicated media already held under a
-different name, which is the check that says JD2 and gallery-dl naming really
-do converge.
+In use for all six ARTMS profiles.
+
+`withaseul` first — 322 files added (74 media, 241 `.json`, 7 `.txt`), nothing
+overwritten or deleted. Of the 74 new media, **zero** duplicated media already
+held under a different name, which is the check that says JD2 and gallery-dl
+naming really do converge.
+
+**2026-08-20**, the first full incremental sync, four days after the previous
+one. 184 new media, 299 files published, 0 failures and **0 CDN 429s**:
+
+| profile | posts | reels | stories | files added |
+|---|---:|---:|---:|---:|
+| 0ct0ber19 | 58 | 2 | 4 | +77 |
+| official_artms | 12 | — | 2 | +85 |
+| cher_ryppo | 41 | 1 | 8 | +63 |
+| zindoriyam | 23 | — | 4 | +35 |
+| kimxxlip | 16 | — | 2 | +23 |
+| withaseul | 10 | — | — | +16 |
+
+The 20 story items are the part that could not have been recovered later.
+
+Two things made it cheap, and both are worth keeping:
+
+- The archive DB was already seeded from the previous run, so `--min-interval`
+  and the recorded `seeded` state meant **no probe passes at all**. A state
+  file has to exist for this; if one is missing after a manual run, write it
+  rather than letting the tool re-seed 24 sources.
+- `--abort 50` (see above) cut the remaining listing cost by roughly 85%.
+
+The run was deliberately **stopped and resumed** halfway to pick up `--abort`.
+That is safe precisely because of the state file: the 12 finished sources were
+already marked `fetched`, so the 20h floor skipped them and only the remaining
+12 re-ran. Stopping a run is cheap now; it was not before.
 
 Published files land owned by the SSH user rather than `rslsync`. The viewer
 reads them fine (world-readable), but Resilio does not own what it syncs; worth
-a `chown` if that ever matters.
+a `chown` if that ever matters. This also makes **`rsync` exit 23**
+("some files/attrs were not transferred") the *normal* outcome of a publish —
+it is the failed `chown`, not lost data. Confirm by re-running the same rsync
+with `--dry-run`: an empty file list means everything arrived.
 
 The profiles to fetch live in `artms_account_links.txt` at the archive root,
 passed with `--urls-file`.
@@ -426,7 +528,23 @@ Re-running with those recovered both videos and produced **0 failures and 0
 
 Without it, gallery-dl logs `Cannot import yt-dlp or youtube-dl` and falls back
 to a progressive URL for DASH videos. The fallback mostly works but is what the
-429s hit hardest. `pipx install yt-dlp` on the fetch host.
+429s hit hardest.
+
+**`pipx install yt-dlp` does not work** — it was the advice here until
+2026-08-20, and it is wrong. It gives yt-dlp its own venv, so the binary lands
+on `PATH` while gallery-dl, in a *different* venv, still cannot `import yt_dlp`.
+The symptom is that everything looks installed and the log keeps saying
+`Cannot import yt-dlp`. gallery-dl needs it importable, not runnable:
+
+```sh
+pipx inject gallery-dl yt-dlp
+```
+
+Verify by asking gallery-dl's own interpreter, not the shell:
+
+```sh
+/home/matt/.local/share/pipx/venvs/gallery-dl/bin/python -c 'import yt_dlp'
+```
 
 ## Known quirks
 
@@ -477,7 +595,7 @@ Real subjects, all present in the archive today. See
 | 13 | empty caption | `CrdsY5CrSsO` | media written, `.txt` absent |
 | 14 | deleted post | `C0TgI7sphfZ` | on disk, absent live — must not be removed |
 | 15 | edited carousel | `C7zG7-jJMlq` | 18 on disk, 8 live — must not be removed |
-| 16 | pinned posts | `0ct0ber19` | 3 pinned, returned out of date order |
+| 16 | pinned posts | `0ct0ber19` | REST listing is strictly reverse-chronological; see below |
 | 17 | profile avatar | `0ct0ber19.jpg` | base dir, undated |
 
 Cases 14–16 are reconciliation, not naming: **a sync must never delete**, since

scripts/gdl-sync.py

@@ -219,7 +219,7 @@ class ArchiveIndex:
 # --------------------------------------------------------------------------
 
 def build_config(rate: str, sleep_request: list[float],
-                 sleep: list[float]) -> dict:
+                 sleep: list[float], abort: int = 0) -> dict:
     """
     The config is generated rather than checked in so the safety-critical
     options cannot drift out of sync with the docs.
@@ -253,7 +253,26 @@ def build_config(rate: str, sleep_request: list[float],
 
     def post_like(stem: str) -> dict:
         """Naming for surfaces whose unit is a post (posts, reels)."""
+        skip: dict = {}
+        if abort:
+            # Stop enumerating once `abort` consecutive files are already in
+            # the skip-archive. The listing pass -- not the downloading -- is
+            # what costs `instagram.com` requests, and it otherwise walks the
+            # whole profile every run to find three new posts.
+            #
+            # Safe here only because the REST listing is strictly
+            # reverse-chronological: the web grid hoists pinned posts to the
+            # front, but this endpoint does not (measured 2026-08-20), so old
+            # posts never appear before new ones.
+            #
+            # Counted in FILES, not posts, so it must clear the largest
+            # already-held carousel -- 22 media for one real post in this
+            # archive. It also means edited carousels (test case 15) stop
+            # being noticed, so a full sweep is still worth running
+            # occasionally.
+            skip["skip"] = f"abort:{abort}"
         return {
+            **skip,
             # `sidecar_shortcode` is set only for carousels, so it is the
             # carousel discriminator. First matching condition wins.
             "filename": {
@@ -666,6 +685,12 @@ def main() -> int:
     ap.add_argument("--max-sources", type=int, default=0, metavar="N",
                     help="hard ceiling on sources touched in one run "
                          "(0 = no limit)")
+    ap.add_argument("--abort", type=int, default=0, metavar="N",
+                    help="stop enumerating posts/reels after N consecutive "
+                         "already-archived FILES (0 = walk everything, the "
+                         "default). 50 is a safe routine value; it cuts the "
+                         "per-run listing cost by roughly 85%%, at the price "
+                         "of no longer noticing edited carousels")
     ap.add_argument("--probe-ttl", type=float, default=24.0, metavar="HOURS",
                     help="reuse cached listing results younger than this")
     ap.add_argument("--force", action="store_true",
@@ -704,7 +729,8 @@ def main() -> int:
     else:
         selected = [Profile(p) for p in sorted(names)]
 
-    config = build_config(args.rate, list(args.sleep_request), list(args.sleep))
+    config = build_config(args.rate, list(args.sleep_request),
+                          list(args.sleep), args.abort)
     args.staging.mkdir(parents=True, exist_ok=True)
     # Deliberately a SIBLING of the staging directory, not inside it: staging is
     # rsynced wholesale into the archive, and a dry run caught this file being