feat: schedule the sync with systemd timers, and let the fetch host publish
mattellite's key is now in the NAS's authorized_keys for agentapi, so
gdl-sync.py publishes straight to the archive. That was the last manual step:
the 2026-08-20 run published locally and rsynced by hand with sshpass, which
no scheduled job could do. Verified end to end -- key auth, then a dry-run
rsync from the fetch host into the archive root.
Three timers, installed and enabled: stories daily, full monthly, sweep
quarterly. A templated gdl-sync@.service takes the mode as its instance name.
Settings that are load-bearing rather than decoration:
RandomizedDelaySec=45m a job firing at exactly 09:00 daily is obviously a
machine; list-timers now shows 09:36, not 09:00
Persistent=true catch up a run the host slept through -- cron just
skips, and a skipped stories run is content gone
TimeoutStartSec=infinity a sweep runs for hours at this pacing; the 90s
default would kill it mid-fetch
Two things caught while installing, both documented: month names are invalid
in OnCalendar's date field (Jan,Apr,Jul,Oct-07 is rejected -- use numerics,
and check with systemd-analyze calendar), and `systemctl --user` over
non-interactive ssh needs XDG_RUNTIME_DIR or it cannot find the bus.
Smoke-tested by starting the stories service: it spent zero Instagram requests
because the 20h floor skipped all six sources, which is exactly what a job
firing twice should do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
Claude Opus 5
+56
-32
@@ -63,6 +63,8 @@ every deletion because of it.
|
||||
| path | what it is |
|
||||
|---|---|
|
||||
| `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
|
||||
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `full` \| `sweep` |
|
||||
| `scripts/systemd/` | the timers actually installed on the fetch host |
|
||||
| `scripts/test_gdl_sync.py` | its tests |
|
||||
| `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
|
||||
| `docs/gallery-dl.md` | the measurements behind every option in the fetcher — **read before changing pacing** |
|
||||
@@ -134,38 +136,63 @@ The skip-archive means an infrequent `full` costs barely more than a frequent
|
||||
one — it only fetches what is new. Frequency buys freshness, not completeness,
|
||||
except for stories.
|
||||
|
||||
## Cron
|
||||
## Scheduling — installed on `mattellite`
|
||||
|
||||
systemd **user** timers, running as `matt`, with lingering enabled so they fire
|
||||
without a login session:
|
||||
|
||||
```sh
|
||||
crontab -e
|
||||
loginctl show-user matt --property=Linger # Linger=yes
|
||||
systemctl --user list-timers 'gdl-sync@*'
|
||||
```
|
||||
|
||||
| unit | schedule | next fire (as installed) |
|
||||
|---|---|---|
|
||||
| `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working |
|
||||
| `gdl-sync@full.timer` | 3rd of each month, 04:00 | 04:37:44 |
|
||||
| `gdl-sync@sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
|
||||
|
||||
Unit files are version-controlled in `scripts/systemd/` and installed to
|
||||
`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
|
||||
the mode as its instance name and runs `gdl-cron.sh %i`.
|
||||
|
||||
Three settings are load-bearing:
|
||||
|
||||
- **`RandomizedDelaySec=45m`** — a job firing at exactly 09:00 daily is
|
||||
obviously a machine, and the entire safety model is about not looking like
|
||||
one. This is why the table above shows 09:36 rather than 09:00.
|
||||
- **`Persistent=true`** — catch up a run missed because the host was off.
|
||||
cron silently skips, and a skipped `stories` run is content gone for good.
|
||||
- **`TimeoutStartSec=infinity`** — a sweep can run for hours at this pacing.
|
||||
The default 90s would kill it mid-fetch.
|
||||
|
||||
Operating them:
|
||||
|
||||
```sh
|
||||
export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed over non-interactive ssh
|
||||
systemctl --user start gdl-sync@stories.service # run one now
|
||||
systemctl --user status gdl-sync@full.timer
|
||||
journalctl --user -u 'gdl-sync@*' -n 50
|
||||
systemctl --user disable --now gdl-sync@sweep.timer # stop one
|
||||
```
|
||||
|
||||
`systemctl --user` fails with "Failed to connect to bus" over ssh unless
|
||||
`XDG_RUNTIME_DIR` is set. Note also that **month names are not valid in
|
||||
`OnCalendar`'s date field** — `Jan,Apr,Jul,Oct-07` is rejected outright, hence
|
||||
`*-01,04,07,10-07`. Check any change with `systemd-analyze calendar '<expr>'`
|
||||
before installing it.
|
||||
|
||||
### cron, if you ever prefer it
|
||||
|
||||
```cron
|
||||
# Randomise the start: a job firing at exactly 09:00 every day is obviously a
|
||||
# machine. `shuf` rather than $RANDOM, because cron runs /bin/sh, not bash --
|
||||
# and avoid `%` in a crontab line, where it means "newline" unless escaped.
|
||||
17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
|
||||
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full
|
||||
17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
|
||||
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full
|
||||
11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep
|
||||
```
|
||||
|
||||
The wrapper exits non-zero when the sync does, so cron will mail you on
|
||||
failure; it also tees every run to `~/gdl/logs/` and keeps the last 30.
|
||||
|
||||
**systemd is the better fit if you want it**, because the randomisation is
|
||||
built in rather than a `sleep`:
|
||||
|
||||
```ini
|
||||
# ~/.config/systemd/user/gdl-stories.timer
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 09:00:00
|
||||
RandomizedDelaySec=45m
|
||||
Persistent=true # catches up after the host is off, which cron will not
|
||||
```
|
||||
|
||||
with `loginctl enable-linger matt` so user timers run without a session.
|
||||
`Persistent=true` matters here: a laptop asleep at 09:00 silently skips a cron
|
||||
job, and a skipped stories run is content gone for good.
|
||||
cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a
|
||||
crontab line means newline unless escaped, so avoid it entirely. cron has no
|
||||
equivalent of `Persistent=true`.
|
||||
|
||||
## Outstanding
|
||||
|
||||
@@ -174,14 +201,11 @@ broken; these are decisions not yet made and cleanups not yet done.
|
||||
|
||||
### Fetching
|
||||
|
||||
- **The fetch host cannot yet publish to the NAS.** `gdl-sync.py --publish`
|
||||
accepts `user@host:/path` and shells out to plain `rsync`, but `mattellite`
|
||||
has no key on the NAS (`agentapi@10.20.28.200: Permission denied`). The
|
||||
2026-08-20 run worked around it by publishing to a local directory and
|
||||
rsyncing to the NAS by hand with `sshpass` from the workstation. **Until
|
||||
mattellite's `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys`, any
|
||||
cron job will fetch successfully and then fail at the publish step.** That is
|
||||
the single thing standing between this and being fully unattended.
|
||||
- `mattellite`'s `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys` for
|
||||
`agentapi` (added 2026-08-20, alongside the workstation's existing key), so
|
||||
the fetch host publishes straight to the archive and no `sshpass` step is
|
||||
needed. **That key is what makes the timers work** — remove it and every
|
||||
scheduled run will fetch successfully and then fail at publish.
|
||||
|
||||
- **5.3 GB of stale staging on `mattellite`** — `~/gdl/staging` and `~/gdl/out`
|
||||
(2.2 GB each, from the 2026-08-17 run) and `~/gdl/staging-0820` /
|
||||
|
||||
Reference in New Issue
Block a user