feat: give the sync a memory, so it stops paying for the same listing twice
Nothing in this tool had any memory: every invocation started from zero and would happily re-enumerate a profile it had listed minutes earlier. That is what suspended the account -- the listing passes, not the downloads -- and an aborted run re-enumerating five profiles on restart was a large part of the bill. Three changes, in order of how much they save: - Seeding is now a one-time bootstrap per source. After the first successful sync the archive DB records everything gallery-dl has seen, so the source is never probed again. A second full sync costs roughly half what the first did. - Stories never seed at all. A story cannot be in the archive before it is fetched, so there is nothing to seed from, and probing would double the cost of the cheapest surface we have. - A source fetched within --min-interval (20h) is refused, and listing results are cached for --probe-ttl (24h), so a restart mid-run is free rather than a repeat. --force overrides both. --only replaces --no-stories and takes any subset of the surfaces, which is what makes a daily stories-only run possible: one source per profile, no seeding, a handful of requests. Everything else stays monthly. Tested with stdlib unittest -- no new dependencies, and it runs anywhere the sync does. The cases include the aborted-restart scenario, which now plans zero work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -270,6 +270,48 @@ viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
|
||||
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
|
||||
only matters when testing by hand.
|
||||
|
||||
## Cadence, and the budget that enforces it
|
||||
|
||||
**Monthly for everything, daily for stories only.** Stories expire in 24h and
|
||||
cannot be backfilled, so they are the one surface where missing a day means
|
||||
losing the content permanently. Everything else can wait — the skip-archive
|
||||
means an infrequent full sync costs barely more than a frequent one, because it
|
||||
only fetches what is new.
|
||||
|
||||
```
|
||||
# monthly, everything
|
||||
gdl-sync.py --index <viewer-url> --staging ~/gdl/staging \
|
||||
--publish <user>@<nas>:<archives> --archive-db ~/gdl/artms.db \
|
||||
--urls-file artms_account_links.txt --execute
|
||||
|
||||
# daily, stories only -- one request per profile
|
||||
gdl-sync.py ... --only stories --execute
|
||||
```
|
||||
|
||||
A stories-only run is one source per profile and **never seeds**, because a
|
||||
story cannot be in the archive before it is fetched; probing would double the
|
||||
cost of the cheapest surface for no benefit. Six profiles is a handful of
|
||||
requests.
|
||||
|
||||
When scheduling it, **randomise the minute and avoid the hour boundary**. A job
|
||||
that fires at exactly 09:00 every day is a machine; one that fires somewhere in
|
||||
a window looks like someone opening the app.
|
||||
|
||||
The tool now refuses to repeat itself:
|
||||
|
||||
| flag | default | what it prevents |
|
||||
|---|---|---|
|
||||
| `--min-interval` | 20h | re-fetching a source touched recently — the aborted-restart case that re-enumerated five profiles |
|
||||
| `--probe-ttl` | 24h | paying for a listing pass twice within a run cycle |
|
||||
| `--max-sources` | off | a runaway list touching more than intended |
|
||||
| `--force` | off | (escape hatch: ignores both guards) |
|
||||
|
||||
State lives beside the archive DB as `<db>.state.json`, recording per source
|
||||
when it was seeded and last fetched. **Seeding is a one-time bootstrap**: after
|
||||
the first successful sync the archive DB records everything gallery-dl has
|
||||
seen, so the source is never probed again. That is the single biggest saving
|
||||
here — a second full sync costs roughly half what the first did.
|
||||
|
||||
## Incremental sync — why the fetch host needs no copy of the archive
|
||||
|
||||
gallery-dl can skip already-held media two ways, and the difference decides
|
||||
|
||||
Reference in New Issue
Block a user