feat: publish fetched media by rsync, and back off from the CDN's 429s

Wires up the staging -> rsync step and replaces --archives with --index
(a listing source: local path or the viewer's API), --staging and
--publish, so the fetch host needs no copy of the archive.

rsync runs --ignore-existing with no --delete. That is a safety property
rather than an optimisation: the archive deliberately outlives Instagram,
so publishing must only ever add. It runs once at the end so a profile
that fails midway never reaches the archive half-written.

Exercised end-to-end against withaseul across all four surfaces,
publishing to a scratch directory. Seeding worked as designed (915 of 984
post items and 28 of 34 reel items already held), stories and highlights
returned no results cleanly, and the collab-reel case landed correctly:
"withaseul - reels" holds files owned by cher_ryppo, 0ct0ber19 and
official_artms, each with the owner in the filename and the crawl scope
as the directory.

The first run drew '429 Too Many Requests' from the CDN at 3M with 1-3s
sleeps and lost two videos. That is the tolerant surface complaining, so
the defaults are now 1M, 6-10s between requests, 3-6s between downloads,
sleep-429 of 120s and 8 retries. Re-running recovered both videos with
zero failures and zero 429s. Installing yt-dlp on the fetch host also
matters: without it DASH videos fall back to a progressive URL, which is
what the rate limiting hit hardest.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-16 21:37:38 -04:00
co-authored by Claude Opus 5
parent 4fee8b1dfe
commit 3a34e4359e
2 changed files with 224 additions and 41 deletions
+54
View File
@@ -251,6 +251,60 @@ The one trap, which silently seeds almost nothing if you get it backwards:
`live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
## Publishing
The fetch host stages to local disk and rsyncs afterwards. `rsync
--ignore-existing` is not an optimisation but the safety property: the archive
deliberately outlives Instagram, so publishing must only ever **add**. No
`--delete`, and nothing already present is overwritten — including sidecars,
which are rewritten every run and would otherwise churn the synced share.
Publishing happens once at the end of a run, so a profile that fails midway
never reaches the archive half-written.
## Verified run
`withaseul`, all four surfaces, staged locally and published to a scratch
directory (never the live archive):
```
==> withaseul / posts seeded 915 of 984 live items
==> withaseul / reels seeded 28 of 34 live items
==> withaseul / stories no results (none active)
==> withaseul / highlights no results
```
Output landed correctly, including the collab-reel case — `withaseul - reels`
contains 53 files owned by `withaseul`, 10 by `cher_ryppo`, 3 by `0ct0ber19`
and 2 by `official_artms`, all with the owner in the filename and the crawl
scope as the directory.
### The CDN rate-limits, and the first run tripped it
At `rate: 3M` with `sleep: [1.0, 3.0]`, `scontent-*.cdninstagram.com` returned
**`429 Too Many Requests`** and two videos were lost (gallery-dl retried, then
gave up with exit 4). This is the *tolerant* surface complaining, which is a
clear signal the pacing was too aggressive.
Defaults are now:
| option | value |
|---|---|
| `--rate` | `1M` |
| `--sleep-request` | 610 s |
| `--sleep` | 36 s |
| `sleep-429` | 120 s |
| `retries` (extractor and downloader) | 8 |
Re-running with those recovered both videos and produced **0 failures and 0
429s**. Do not raise them for speed; an archive sync has no deadline.
### yt-dlp is worth installing
Without it, gallery-dl logs `Cannot import yt-dlp or youtube-dl` and falls back
to a progressive URL for DASH videos. The fallback mostly works but is what the
429s hit hardest. `pipx install yt-dlp` on the fetch host.
## Known quirks
- **`count` is not the emitted file count.** For 135 of 214 posts it was exactly