feat: give the sync a memory, so it stops paying for the same listing twice

Nothing in this tool had any memory: every invocation started from zero
and would happily re-enumerate a profile it had listed minutes earlier.
That is what suspended the account -- the listing passes, not the
downloads -- and an aborted run re-enumerating five profiles on restart
was a large part of the bill.

Three changes, in order of how much they save:

- Seeding is now a one-time bootstrap per source. After the first
  successful sync the archive DB records everything gallery-dl has seen,
  so the source is never probed again. A second full sync costs roughly
  half what the first did.
- Stories never seed at all. A story cannot be in the archive before it
  is fetched, so there is nothing to seed from, and probing would double
  the cost of the cheapest surface we have.
- A source fetched within --min-interval (20h) is refused, and listing
  results are cached for --probe-ttl (24h), so a restart mid-run is free
  rather than a repeat. --force overrides both.

--only replaces --no-stories and takes any subset of the surfaces, which
is what makes a daily stories-only run possible: one source per profile,
no seeding, a handful of requests. Everything else stays monthly.

Tested with stdlib unittest -- no new dependencies, and it runs anywhere
the sync does. The cases include the aborted-restart scenario, which now
plans zero work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-17 13:53:18 -04:00
co-authored by Claude Opus 5
parent 7a085d2272
commit 4ce5cf048e
5 changed files with 476 additions and 26 deletions
+42
View File
@@ -270,6 +270,48 @@ viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
only matters when testing by hand.
## Cadence, and the budget that enforces it
**Monthly for everything, daily for stories only.** Stories expire in 24h and
cannot be backfilled, so they are the one surface where missing a day means
losing the content permanently. Everything else can wait — the skip-archive
means an infrequent full sync costs barely more than a frequent one, because it
only fetches what is new.
```
# monthly, everything
gdl-sync.py --index <viewer-url> --staging ~/gdl/staging \
--publish <user>@<nas>:<archives> --archive-db ~/gdl/artms.db \
--urls-file artms_account_links.txt --execute
# daily, stories only -- one request per profile
gdl-sync.py ... --only stories --execute
```
A stories-only run is one source per profile and **never seeds**, because a
story cannot be in the archive before it is fetched; probing would double the
cost of the cheapest surface for no benefit. Six profiles is a handful of
requests.
When scheduling it, **randomise the minute and avoid the hour boundary**. A job
that fires at exactly 09:00 every day is a machine; one that fires somewhere in
a window looks like someone opening the app.
The tool now refuses to repeat itself:
| flag | default | what it prevents |
|---|---|---|
| `--min-interval` | 20h | re-fetching a source touched recently — the aborted-restart case that re-enumerated five profiles |
| `--probe-ttl` | 24h | paying for a listing pass twice within a run cycle |
| `--max-sources` | off | a runaway list touching more than intended |
| `--force` | off | (escape hatch: ignores both guards) |
State lives beside the archive DB as `<db>.state.json`, recording per source
when it was seeded and last fetched. **Seeding is a one-time bootstrap**: after
the first successful sync the archive DB records everything gallery-dl has
seen, so the source is never probed again. That is the single biggest saving
here — a second full sync costs roughly half what the first did.
## Incremental sync — why the fetch host needs no copy of the archive
gallery-dl can skip already-held media two ways, and the difference decides