diff --git a/docs/gallery-dl.md b/docs/gallery-dl.md index 011c3d7..d496d2e 100644 --- a/docs/gallery-dl.md +++ b/docs/gallery-dl.md @@ -1,10 +1,10 @@ # gallery-dl — a CLI replacement for JDownloader2 -Status: **design + verified config.** `scripts/gdl-sync.py` is a skeleton; no -profile has been migrated yet. +Status: **in production.** All six ARTMS profiles are synced with +`scripts/gdl-sync.py`; JD2 is no longer used for them. Everything below was measured against the live site and the real archive on -2026-08-16, not inferred from documentation. +2026-08-16 and 2026-08-20, not inferred from documentation. ## Why gallery-dl and not a hand-rolled script @@ -20,6 +20,12 @@ The account used for all of the below was suspended the same day this tooling was built, for "activity that doesn't follow our Community Standards on spam". The fetching was not the expensive part. **Verification was.** +**It was restored, and synced normally again on 2026-08-20** — a full run +across all six profiles with 0 failures and 0 CDN 429s. That is not evidence +the limits were imagined; it is one data point on a restored account that has +been treated carefully since. Everything below still applies, and the budget is +still per session rather than per command. + What was actually spent against `instagram.com` in a few hours, from one session and one IP: @@ -437,15 +443,46 @@ never reaches the archive half-written. ## Status -In use. `withaseul` has been fetched and published to the live archive — 322 -files added (74 media, 241 `.json`, 7 `.txt`), nothing overwritten or deleted. -Of the 74 new media files, **zero** duplicated media already held under a -different name, which is the check that says JD2 and gallery-dl naming really -do converge. +In use for all six ARTMS profiles. + +`withaseul` first — 322 files added (74 media, 241 `.json`, 7 `.txt`), nothing +overwritten or deleted. Of the 74 new media, **zero** duplicated media already +held under a different name, which is the check that says JD2 and gallery-dl +naming really do converge. + +**2026-08-20**, the first full incremental sync, four days after the previous +one. 184 new media, 299 files published, 0 failures and **0 CDN 429s**: + +| profile | posts | reels | stories | files added | +|---|---:|---:|---:|---:| +| 0ct0ber19 | 58 | 2 | 4 | +77 | +| official_artms | 12 | — | 2 | +85 | +| cher_ryppo | 41 | 1 | 8 | +63 | +| zindoriyam | 23 | — | 4 | +35 | +| kimxxlip | 16 | — | 2 | +23 | +| withaseul | 10 | — | — | +16 | + +The 20 story items are the part that could not have been recovered later. + +Two things made it cheap, and both are worth keeping: + +- The archive DB was already seeded from the previous run, so `--min-interval` + and the recorded `seeded` state meant **no probe passes at all**. A state + file has to exist for this; if one is missing after a manual run, write it + rather than letting the tool re-seed 24 sources. +- `--abort 50` (see above) cut the remaining listing cost by roughly 85%. + +The run was deliberately **stopped and resumed** halfway to pick up `--abort`. +That is safe precisely because of the state file: the 12 finished sources were +already marked `fetched`, so the 20h floor skipped them and only the remaining +12 re-ran. Stopping a run is cheap now; it was not before. Published files land owned by the SSH user rather than `rslsync`. The viewer reads them fine (world-readable), but Resilio does not own what it syncs; worth -a `chown` if that ever matters. +a `chown` if that ever matters. This also makes **`rsync` exit 23** +("some files/attrs were not transferred") the *normal* outcome of a publish — +it is the failed `chown`, not lost data. Confirm by re-running the same rsync +with `--dry-run`: an empty file list means everything arrived. The profiles to fetch live in `artms_account_links.txt` at the archive root, passed with `--urls-file`.