diff --git a/TOOLING.md b/TOOLING.md index 530669d..85c890f 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -93,6 +93,80 @@ attempt put tooling notes in `CLAUDE.md` and a `jd2` script in `package.json`; because `main` had *deleted* those lines, every merge re-applied the deletion. Keep branch-specific documentation in this file, which `main` does not have. +## Running it by hand + +Everything happens on **`mattellite`** — the fetch host whose public IP matches +the browser the cookie came from. Running it anywhere else is what +session-hijack detection looks for. + +```sh +ssh mattellite +~/gdl/gdl-cron.sh full # or: stories | sweep +``` + +That is the whole thing: it wipes staging, fetches, and publishes straight to +the NAS. To drive `gdl-sync.py` directly instead — always `--dry-run` first, +which prints the plan and the exact rsync that would touch the archive: + +```sh +cd ~/gdl +PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \ + --index https://instaarchive.ergosteur.com \ + --staging ~/gdl/staging-manual \ + --publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \ + --archive-db ~/gdl/artms.db \ + --urls-file ~/gdl/artms_account_links.txt \ + --abort 50 --dry-run # swap for --execute when the plan looks right +``` + +`PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not +on cron's PATH and not on a non-login shell's either. + +### The three modes + +| mode | cadence | cost | why | +|---|---|---|---| +| `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped | +| `full` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating once it reaches content already held | +| `sweep` | quarterly | ~420 requests | no abort; the **only** run that notices carousels edited after we archived them (test case 15) | + +The skip-archive means an infrequent `full` costs barely more than a frequent +one — it only fetches what is new. Frequency buys freshness, not completeness, +except for stories. + +## Cron + +```sh +crontab -e +``` + +```cron +# Randomise the start: a job firing at exactly 09:00 every day is obviously a +# machine. `shuf` rather than $RANDOM, because cron runs /bin/sh, not bash -- +# and avoid `%` in a crontab line, where it means "newline" unless escaped. +17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories +43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full +11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep +``` + +The wrapper exits non-zero when the sync does, so cron will mail you on +failure; it also tees every run to `~/gdl/logs/` and keeps the last 30. + +**systemd is the better fit if you want it**, because the randomisation is +built in rather than a `sleep`: + +```ini +# ~/.config/systemd/user/gdl-stories.timer +[Timer] +OnCalendar=*-*-* 09:00:00 +RandomizedDelaySec=45m +Persistent=true # catches up after the host is off, which cron will not +``` + +with `loginctl enable-linger matt` so user timers run without a session. +`Persistent=true` matters here: a laptop asleep at 09:00 silently skips a cron +job, and a skipped stories run is content gone for good. + ## Outstanding State as of 2026-08-20, after the sync run and the repo split. Nothing here is @@ -100,6 +174,15 @@ broken; these are decisions not yet made and cleanups not yet done. ### Fetching +- **The fetch host cannot yet publish to the NAS.** `gdl-sync.py --publish` + accepts `user@host:/path` and shells out to plain `rsync`, but `mattellite` + has no key on the NAS (`agentapi@10.20.28.200: Permission denied`). The + 2026-08-20 run worked around it by publishing to a local directory and + rsyncing to the NAS by hand with `sshpass` from the workstation. **Until + mattellite's `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys`, any + cron job will fetch successfully and then fail at the publish step.** That is + the single thing standing between this and being fully unattended. + - **5.3 GB of stale staging on `mattellite`** — `~/gdl/staging` and `~/gdl/out` (2.2 GB each, from the 2026-08-17 run) and `~/gdl/staging-0820` / `~/gdl/out-0820` (446 MB each, from 2026-08-20). Every file in all four was diff --git a/scripts/gdl-cron.sh b/scripts/gdl-cron.sh new file mode 100755 index 0000000..443e445 --- /dev/null +++ b/scripts/gdl-cron.sh @@ -0,0 +1,55 @@ +#!/bin/sh +# Unattended wrapper around gdl-sync.py. One argument: the run mode. +# +# stories daily ~6 requests; the only surface that cannot be backfilled +# full monthly every surface, --abort 50 (stops at already-held content) +# sweep quarterly every surface, no abort -- the only run that notices +# carousels edited after we archived them +# +# Exits non-zero if the sync does, so cron mails you. Everything is logged. +set -eu + +MODE="${1:?usage: gdl-cron.sh stories|full|sweep}" + +GDL_HOME="${GDL_HOME:-$HOME/gdl}" +INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}" +PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}" +STAGING="$GDL_HOME/staging-$MODE" +LOG="$GDL_HOME/logs/$MODE-$(date +%Y%m%d-%H%M%S).log" + +PATH="$HOME/.local/bin:$PATH"; export PATH + +case "$MODE" in + stories) ARGS="--only stories" ;; + full) ARGS="--only posts,reels,stories,highlights --abort 50" ;; + sweep) ARGS="--only posts,reels,stories,highlights" ;; + *) echo "unknown mode: $MODE" >&2; exit 2 ;; +esac + +mkdir -p "$GDL_HOME/logs" + +# Staging is wiped every run ON PURPOSE. What we already hold is decided by the +# skip-archive (--download-archive), never by which files happen to be sitting +# in staging, so starting empty is correct -- and it keeps the publish rsync +# to just the new files instead of re-walking gigabytes each time. +rm -rf "$STAGING" + +{ + echo "=== $MODE run $(date -Is) ===" + # shellcheck disable=SC2086 + "$GDL_HOME/gdl-sync.py" \ + --index "$INDEX" \ + --staging "$STAGING" \ + --publish "$PUBLISH" \ + --archive-db "$GDL_HOME/artms.db" \ + --urls-file "$GDL_HOME/artms_account_links.txt" \ + $ARGS --execute + status=$? + echo "=== exit $status at $(date -Is) ===" + exit $status +} 2>&1 | tee -a "$LOG" + +# Keep the log directory from growing without bound. +ls -1t "$GDL_HOME/logs" | tail -n +30 | while read -r old; do + rm -f "$GDL_HOME/logs/$old" +done