feat: add an unattended wrapper, and document running this without an agent

gdl-sync.py always needed a fistful of paths on the command line, and the 2026-08-20 run published to a local directory and then rsynced to the NAS by hand -- fine interactively, impossible from cron. gdl-cron.sh takes one argument (stories|full|sweep) and does the whole thing. Staging is wiped every run deliberately: what we already hold is decided by the skip-archive, never by what happens to be sitting in staging, so starting empty is correct and keeps the publish to just the new files. Documents the three cadences and what each is for -- notably that only `sweep` notices carousels edited after we archived them, and that only `stories` loses content if it is skipped, since stories cannot be backfilled. Cron notes the two things that would silently break it: cron runs /bin/sh so $RANDOM does not exist (use shuf), and `%` in a crontab line means newline. systemd is offered as the better option because RandomizedDelaySec and Persistent=true are built in -- a host asleep at 09:00 silently skips a cron job, and a skipped stories run is content gone for good. The publish step is still blocked on mattellite having no key on the NAS; that is recorded under Outstanding as the one thing between this and unattended. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
2026-08-20 19:59:57 -04:00
parent ff0d0f9f78
commit ef99389cf4
2 changed files with 138 additions and 0 deletions
@@ -93,6 +93,80 @@ attempt put tooling notes in `CLAUDE.md` and a `jd2` script in `package.json`;
 because `main` had *deleted* those lines, every merge re-applied the deletion.
 Keep branch-specific documentation in this file, which `main` does not have.

+## Running it by hand
+
+Everything happens on **`mattellite`** — the fetch host whose public IP matches
+the browser the cookie came from. Running it anywhere else is what
+session-hijack detection looks for.
+
+```sh
+ssh mattellite
+~/gdl/gdl-cron.sh full        # or: stories | sweep
+```
+
+That is the whole thing: it wipes staging, fetches, and publishes straight to
+the NAS. To drive `gdl-sync.py` directly instead — always `--dry-run` first,
+which prints the plan and the exact rsync that would touch the archive:
+
+```sh
+cd ~/gdl
+PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
+    --index    https://instaarchive.ergosteur.com \
+    --staging  ~/gdl/staging-manual \
+    --publish  agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
+    --archive-db ~/gdl/artms.db \
+    --urls-file  ~/gdl/artms_account_links.txt \
+    --abort 50 --dry-run          # swap for --execute when the plan looks right
+```
+
+`PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not
+on cron's PATH and not on a non-login shell's either.
+
+### The three modes
+
+| mode | cadence | cost | why |
+|---|---|---|---|
+| `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped |
+| `full` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating once it reaches content already held |
+| `sweep` | quarterly | ~420 requests | no abort; the **only** run that notices carousels edited after we archived them (test case 15) |
+
+The skip-archive means an infrequent `full` costs barely more than a frequent
+one — it only fetches what is new. Frequency buys freshness, not completeness,
+except for stories.
+
+## Cron
+
+```sh
+crontab -e
+```
+
+```cron
+# Randomise the start: a job firing at exactly 09:00 every day is obviously a
+# machine. `shuf` rather than $RANDOM, because cron runs /bin/sh, not bash --
+# and avoid `%` in a crontab line, where it means "newline" unless escaped.
+17 9  *  *  *      sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
+43 4  3  *  *      sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full
+11 4  7  1,4,7,10 *  sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep
+```
+
+The wrapper exits non-zero when the sync does, so cron will mail you on
+failure; it also tees every run to `~/gdl/logs/` and keeps the last 30.
+
+**systemd is the better fit if you want it**, because the randomisation is
+built in rather than a `sleep`:
+
+```ini
+# ~/.config/systemd/user/gdl-stories.timer
+[Timer]
+OnCalendar=*-*-* 09:00:00
+RandomizedDelaySec=45m
+Persistent=true          # catches up after the host is off, which cron will not
+```
+
+with `loginctl enable-linger matt` so user timers run without a session.
+`Persistent=true` matters here: a laptop asleep at 09:00 silently skips a cron
+job, and a skipped stories run is content gone for good.
+
 ## Outstanding

 State as of 2026-08-20, after the sync run and the repo split. Nothing here is
@@ -100,6 +174,15 @@ broken; these are decisions not yet made and cleanups not yet done.

 ### Fetching

+- **The fetch host cannot yet publish to the NAS.** `gdl-sync.py --publish`
+  accepts `user@host:/path` and shells out to plain `rsync`, but `mattellite`
+  has no key on the NAS (`agentapi@10.20.28.200: Permission denied`). The
+  2026-08-20 run worked around it by publishing to a local directory and
+  rsyncing to the NAS by hand with `sshpass` from the workstation. **Until
+  mattellite's `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys`, any
+  cron job will fetch successfully and then fail at the publish step.** That is
+  the single thing standing between this and being fully unattended.
+
 - **5.3 GB of stale staging on `mattellite`** — `~/gdl/staging` and `~/gdl/out`
  (2.2 GB each, from the 2026-08-17 run) and `~/gdl/staging-0820` /
  `~/gdl/out-0820` (446 MB each, from 2026-08-20). Every file in all four was
@@ -0,0 +1,55 @@
+#!/bin/sh
+# Unattended wrapper around gdl-sync.py. One argument: the run mode.
+#
+#   stories   daily      ~6 requests; the only surface that cannot be backfilled
+#   full      monthly    every surface, --abort 50 (stops at already-held content)
+#   sweep     quarterly  every surface, no abort -- the only run that notices
+#                        carousels edited after we archived them
+#
+# Exits non-zero if the sync does, so cron mails you. Everything is logged.
+set -eu
+
+MODE="${1:?usage: gdl-cron.sh stories|full|sweep}"
+
+GDL_HOME="${GDL_HOME:-$HOME/gdl}"
+INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
+PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
+STAGING="$GDL_HOME/staging-$MODE"
+LOG="$GDL_HOME/logs/$MODE-$(date +%Y%m%d-%H%M%S).log"
+
+PATH="$HOME/.local/bin:$PATH"; export PATH
+
+case "$MODE" in
+  stories) ARGS="--only stories" ;;
+  full)    ARGS="--only posts,reels,stories,highlights --abort 50" ;;
+  sweep)   ARGS="--only posts,reels,stories,highlights" ;;
+  *) echo "unknown mode: $MODE" >&2; exit 2 ;;
+esac
+
+mkdir -p "$GDL_HOME/logs"
+
+# Staging is wiped every run ON PURPOSE. What we already hold is decided by the
+# skip-archive (--download-archive), never by which files happen to be sitting
+# in staging, so starting empty is correct -- and it keeps the publish rsync
+# to just the new files instead of re-walking gigabytes each time.
+rm -rf "$STAGING"
+
+{
+  echo "=== $MODE run $(date -Is) ==="
+  # shellcheck disable=SC2086
+  "$GDL_HOME/gdl-sync.py" \
+      --index "$INDEX" \
+      --staging "$STAGING" \
+      --publish "$PUBLISH" \
+      --archive-db "$GDL_HOME/artms.db" \
+      --urls-file "$GDL_HOME/artms_account_links.txt" \
+      $ARGS --execute
+  status=$?
+  echo "=== exit $status at $(date -Is) ==="
+  exit $status
+} 2>&1 | tee -a "$LOG"
+
+# Keep the log directory from growing without bound.
+ls -1t "$GDL_HOME/logs" | tail -n +30 | while read -r old; do
+  rm -f "$GDL_HOME/logs/$old"
+done