diff --git a/TOOLING.md b/TOOLING.md index 85c890f..54e4bbf 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -63,6 +63,8 @@ every deletion because of it. | path | what it is | |---|---| | `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles | +| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `full` \| `sweep` | +| `scripts/systemd/` | the timers actually installed on the fetch host | | `scripts/test_gdl_sync.py` | its tests | | `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere | | `docs/gallery-dl.md` | the measurements behind every option in the fetcher — **read before changing pacing** | @@ -134,38 +136,63 @@ The skip-archive means an infrequent `full` costs barely more than a frequent one — it only fetches what is new. Frequency buys freshness, not completeness, except for stories. -## Cron +## Scheduling — installed on `mattellite` + +systemd **user** timers, running as `matt`, with lingering enabled so they fire +without a login session: ```sh -crontab -e +loginctl show-user matt --property=Linger # Linger=yes +systemctl --user list-timers 'gdl-sync@*' ``` +| unit | schedule | next fire (as installed) | +|---|---|---| +| `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working | +| `gdl-sync@full.timer` | 3rd of each month, 04:00 | 04:37:44 | +| `gdl-sync@sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 | + +Unit files are version-controlled in `scripts/systemd/` and installed to +`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes +the mode as its instance name and runs `gdl-cron.sh %i`. + +Three settings are load-bearing: + +- **`RandomizedDelaySec=45m`** — a job firing at exactly 09:00 daily is + obviously a machine, and the entire safety model is about not looking like + one. This is why the table above shows 09:36 rather than 09:00. +- **`Persistent=true`** — catch up a run missed because the host was off. + cron silently skips, and a skipped `stories` run is content gone for good. +- **`TimeoutStartSec=infinity`** — a sweep can run for hours at this pacing. + The default 90s would kill it mid-fetch. + +Operating them: + +```sh +export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed over non-interactive ssh +systemctl --user start gdl-sync@stories.service # run one now +systemctl --user status gdl-sync@full.timer +journalctl --user -u 'gdl-sync@*' -n 50 +systemctl --user disable --now gdl-sync@sweep.timer # stop one +``` + +`systemctl --user` fails with "Failed to connect to bus" over ssh unless +`XDG_RUNTIME_DIR` is set. Note also that **month names are not valid in +`OnCalendar`'s date field** — `Jan,Apr,Jul,Oct-07` is rejected outright, hence +`*-01,04,07,10-07`. Check any change with `systemd-analyze calendar ''` +before installing it. + +### cron, if you ever prefer it + ```cron -# Randomise the start: a job firing at exactly 09:00 every day is obviously a -# machine. `shuf` rather than $RANDOM, because cron runs /bin/sh, not bash -- -# and avoid `%` in a crontab line, where it means "newline" unless escaped. -17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories -43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full +17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories +43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full 11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep ``` -The wrapper exits non-zero when the sync does, so cron will mail you on -failure; it also tees every run to `~/gdl/logs/` and keeps the last 30. - -**systemd is the better fit if you want it**, because the randomisation is -built in rather than a `sleep`: - -```ini -# ~/.config/systemd/user/gdl-stories.timer -[Timer] -OnCalendar=*-*-* 09:00:00 -RandomizedDelaySec=45m -Persistent=true # catches up after the host is off, which cron will not -``` - -with `loginctl enable-linger matt` so user timers run without a session. -`Persistent=true` matters here: a laptop asleep at 09:00 silently skips a cron -job, and a skipped stories run is content gone for good. +cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a +crontab line means newline unless escaped, so avoid it entirely. cron has no +equivalent of `Persistent=true`. ## Outstanding @@ -174,14 +201,11 @@ broken; these are decisions not yet made and cleanups not yet done. ### Fetching -- **The fetch host cannot yet publish to the NAS.** `gdl-sync.py --publish` - accepts `user@host:/path` and shells out to plain `rsync`, but `mattellite` - has no key on the NAS (`agentapi@10.20.28.200: Permission denied`). The - 2026-08-20 run worked around it by publishing to a local directory and - rsyncing to the NAS by hand with `sshpass` from the workstation. **Until - mattellite's `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys`, any - cron job will fetch successfully and then fail at the publish step.** That is - the single thing standing between this and being fully unattended. +- `mattellite`'s `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys` for + `agentapi` (added 2026-08-20, alongside the workstation's existing key), so + the fetch host publishes straight to the archive and no `sshpass` step is + needed. **That key is what makes the timers work** — remove it and every + scheduled run will fetch successfully and then fail at publish. - **5.3 GB of stale staging on `mattellite`** — `~/gdl/staging` and `~/gdl/out` (2.2 GB each, from the 2026-08-17 run) and `~/gdl/staging-0820` / diff --git a/scripts/systemd/gdl-sync@.service b/scripts/systemd/gdl-sync@.service new file mode 100644 index 0000000..829b0c4 --- /dev/null +++ b/scripts/systemd/gdl-sync@.service @@ -0,0 +1,16 @@ +[Unit] +# One templated service for all three modes; the instance name (%i) is the +# mode: stories, full or sweep. +Description=Instagram archive sync (%i) +Documentation=file:%h/gdl/TOOLING.md +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=%h/gdl/gdl-cron.sh %i +# A sync has no deadline and the pacing is deliberately slow; a full sweep can +# run for hours. Never let systemd kill one midway -- a half-published run is +# the one state the publish step is designed to avoid. +TimeoutStartSec=infinity +Nice=10 diff --git a/scripts/systemd/gdl-sync@full.timer b/scripts/systemd/gdl-sync@full.timer new file mode 100644 index 0000000..9f998a4 --- /dev/null +++ b/scripts/systemd/gdl-sync@full.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Monthly Instagram archive sync (all surfaces, --abort 50) + +[Timer] +OnCalendar=*-*-03 04:00:00 +RandomizedDelaySec=45m +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/scripts/systemd/gdl-sync@stories.timer b/scripts/systemd/gdl-sync@stories.timer new file mode 100644 index 0000000..b0906ee --- /dev/null +++ b/scripts/systemd/gdl-sync@stories.timer @@ -0,0 +1,15 @@ +[Unit] +Description=Daily Instagram stories sync +# Stories expire in 24h and cannot be backfilled. This is the only timer whose +# missed run means content is gone for good, which is what Persistent= is for. + +[Timer] +OnCalendar=*-*-* 09:00:00 +# Not a fixed time: a job firing at exactly 09:00 every day is obviously a +# machine, and the whole safety model is about not looking like one. +RandomizedDelaySec=45m +# Catch up after the host was asleep or off. cron would silently skip. +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/scripts/systemd/gdl-sync@sweep.timer b/scripts/systemd/gdl-sync@sweep.timer new file mode 100644 index 0000000..348a005 --- /dev/null +++ b/scripts/systemd/gdl-sync@sweep.timer @@ -0,0 +1,13 @@ +[Unit] +Description=Quarterly full Instagram archive sweep (no abort) +# The only run that enumerates every profile to the end, and so the only one +# that notices carousels edited after we archived them (test case 15). + +[Timer] +# Month names are not valid in OnCalendar's date field -- numeric only. +OnCalendar=*-01,04,07,10-07 04:00:00 +RandomizedDelaySec=45m +Persistent=true + +[Install] +WantedBy=timers.target