feat: publish fetched media by rsync, and back off from the CDN's 429s
Wires up the staging -> rsync step and replaces --archives with --index (a listing source: local path or the viewer's API), --staging and --publish, so the fetch host needs no copy of the archive. rsync runs --ignore-existing with no --delete. That is a safety property rather than an optimisation: the archive deliberately outlives Instagram, so publishing must only ever add. It runs once at the end so a profile that fails midway never reaches the archive half-written. Exercised end-to-end against withaseul across all four surfaces, publishing to a scratch directory. Seeding worked as designed (915 of 984 post items and 28 of 34 reel items already held), stories and highlights returned no results cleanly, and the collab-reel case landed correctly: "withaseul - reels" holds files owned by cher_ryppo, 0ct0ber19 and official_artms, each with the owner in the filename and the crawl scope as the directory. The first run drew '429 Too Many Requests' from the CDN at 3M with 1-3s sleeps and lost two videos. That is the tolerant surface complaining, so the defaults are now 1M, 6-10s between requests, 3-6s between downloads, sleep-429 of 120s and 8 retries. Re-running recovered both videos with zero failures and zero 429s. Installing yt-dlp on the fetch host also matters: without it DASH videos fall back to a progressive URL, which is what the rate limiting hit hardest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -251,6 +251,60 @@ The one trap, which silently seeds almost nothing if you get it backwards:
|
||||
|
||||
`live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
|
||||
|
||||
## Publishing
|
||||
|
||||
The fetch host stages to local disk and rsyncs afterwards. `rsync
|
||||
--ignore-existing` is not an optimisation but the safety property: the archive
|
||||
deliberately outlives Instagram, so publishing must only ever **add**. No
|
||||
`--delete`, and nothing already present is overwritten — including sidecars,
|
||||
which are rewritten every run and would otherwise churn the synced share.
|
||||
|
||||
Publishing happens once at the end of a run, so a profile that fails midway
|
||||
never reaches the archive half-written.
|
||||
|
||||
## Verified run
|
||||
|
||||
`withaseul`, all four surfaces, staged locally and published to a scratch
|
||||
directory (never the live archive):
|
||||
|
||||
```
|
||||
==> withaseul / posts seeded 915 of 984 live items
|
||||
==> withaseul / reels seeded 28 of 34 live items
|
||||
==> withaseul / stories no results (none active)
|
||||
==> withaseul / highlights no results
|
||||
```
|
||||
|
||||
Output landed correctly, including the collab-reel case — `withaseul - reels`
|
||||
contains 53 files owned by `withaseul`, 10 by `cher_ryppo`, 3 by `0ct0ber19`
|
||||
and 2 by `official_artms`, all with the owner in the filename and the crawl
|
||||
scope as the directory.
|
||||
|
||||
### The CDN rate-limits, and the first run tripped it
|
||||
|
||||
At `rate: 3M` with `sleep: [1.0, 3.0]`, `scontent-*.cdninstagram.com` returned
|
||||
**`429 Too Many Requests`** and two videos were lost (gallery-dl retried, then
|
||||
gave up with exit 4). This is the *tolerant* surface complaining, which is a
|
||||
clear signal the pacing was too aggressive.
|
||||
|
||||
Defaults are now:
|
||||
|
||||
| option | value |
|
||||
|---|---|
|
||||
| `--rate` | `1M` |
|
||||
| `--sleep-request` | 6–10 s |
|
||||
| `--sleep` | 3–6 s |
|
||||
| `sleep-429` | 120 s |
|
||||
| `retries` (extractor and downloader) | 8 |
|
||||
|
||||
Re-running with those recovered both videos and produced **0 failures and 0
|
||||
429s**. Do not raise them for speed; an archive sync has no deadline.
|
||||
|
||||
### yt-dlp is worth installing
|
||||
|
||||
Without it, gallery-dl logs `Cannot import yt-dlp or youtube-dl` and falls back
|
||||
to a progressive URL for DASH videos. The fallback mostly works but is what the
|
||||
429s hit hardest. `pipx install yt-dlp` on the fetch host.
|
||||
|
||||
## Known quirks
|
||||
|
||||
- **`count` is not the emitted file count.** For 135 of 214 posts it was exactly
|
||||
|
||||
Reference in New Issue
Block a user