feat: add an unattended wrapper, and document running this without an agent
gdl-sync.py always needed a fistful of paths on the command line, and the 2026-08-20 run published to a local directory and then rsynced to the NAS by hand -- fine interactively, impossible from cron. gdl-cron.sh takes one argument (stories|full|sweep) and does the whole thing. Staging is wiped every run deliberately: what we already hold is decided by the skip-archive, never by what happens to be sitting in staging, so starting empty is correct and keeps the publish to just the new files. Documents the three cadences and what each is for -- notably that only `sweep` notices carousels edited after we archived them, and that only `stories` loses content if it is skipped, since stories cannot be backfilled. Cron notes the two things that would silently break it: cron runs /bin/sh so $RANDOM does not exist (use shuf), and `%` in a crontab line means newline. systemd is offered as the better option because RandomizedDelaySec and Persistent=true are built in -- a host asleep at 09:00 silently skips a cron job, and a skipped stories run is content gone for good. The publish step is still blocked on mattellite having no key on the NAS; that is recorded under Outstanding as the one thing between this and unattended. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
This commit is contained in:
+83
@@ -93,6 +93,80 @@ attempt put tooling notes in `CLAUDE.md` and a `jd2` script in `package.json`;
|
||||
because `main` had *deleted* those lines, every merge re-applied the deletion.
|
||||
Keep branch-specific documentation in this file, which `main` does not have.
|
||||
|
||||
## Running it by hand
|
||||
|
||||
Everything happens on **`mattellite`** — the fetch host whose public IP matches
|
||||
the browser the cookie came from. Running it anywhere else is what
|
||||
session-hijack detection looks for.
|
||||
|
||||
```sh
|
||||
ssh mattellite
|
||||
~/gdl/gdl-cron.sh full # or: stories | sweep
|
||||
```
|
||||
|
||||
That is the whole thing: it wipes staging, fetches, and publishes straight to
|
||||
the NAS. To drive `gdl-sync.py` directly instead — always `--dry-run` first,
|
||||
which prints the plan and the exact rsync that would touch the archive:
|
||||
|
||||
```sh
|
||||
cd ~/gdl
|
||||
PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
|
||||
--index https://instaarchive.ergosteur.com \
|
||||
--staging ~/gdl/staging-manual \
|
||||
--publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
|
||||
--archive-db ~/gdl/artms.db \
|
||||
--urls-file ~/gdl/artms_account_links.txt \
|
||||
--abort 50 --dry-run # swap for --execute when the plan looks right
|
||||
```
|
||||
|
||||
`PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not
|
||||
on cron's PATH and not on a non-login shell's either.
|
||||
|
||||
### The three modes
|
||||
|
||||
| mode | cadence | cost | why |
|
||||
|---|---|---|---|
|
||||
| `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped |
|
||||
| `full` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating once it reaches content already held |
|
||||
| `sweep` | quarterly | ~420 requests | no abort; the **only** run that notices carousels edited after we archived them (test case 15) |
|
||||
|
||||
The skip-archive means an infrequent `full` costs barely more than a frequent
|
||||
one — it only fetches what is new. Frequency buys freshness, not completeness,
|
||||
except for stories.
|
||||
|
||||
## Cron
|
||||
|
||||
```sh
|
||||
crontab -e
|
||||
```
|
||||
|
||||
```cron
|
||||
# Randomise the start: a job firing at exactly 09:00 every day is obviously a
|
||||
# machine. `shuf` rather than $RANDOM, because cron runs /bin/sh, not bash --
|
||||
# and avoid `%` in a crontab line, where it means "newline" unless escaped.
|
||||
17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
|
||||
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full
|
||||
11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep
|
||||
```
|
||||
|
||||
The wrapper exits non-zero when the sync does, so cron will mail you on
|
||||
failure; it also tees every run to `~/gdl/logs/` and keeps the last 30.
|
||||
|
||||
**systemd is the better fit if you want it**, because the randomisation is
|
||||
built in rather than a `sleep`:
|
||||
|
||||
```ini
|
||||
# ~/.config/systemd/user/gdl-stories.timer
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 09:00:00
|
||||
RandomizedDelaySec=45m
|
||||
Persistent=true # catches up after the host is off, which cron will not
|
||||
```
|
||||
|
||||
with `loginctl enable-linger matt` so user timers run without a session.
|
||||
`Persistent=true` matters here: a laptop asleep at 09:00 silently skips a cron
|
||||
job, and a skipped stories run is content gone for good.
|
||||
|
||||
## Outstanding
|
||||
|
||||
State as of 2026-08-20, after the sync run and the repo split. Nothing here is
|
||||
@@ -100,6 +174,15 @@ broken; these are decisions not yet made and cleanups not yet done.
|
||||
|
||||
### Fetching
|
||||
|
||||
- **The fetch host cannot yet publish to the NAS.** `gdl-sync.py --publish`
|
||||
accepts `user@host:/path` and shells out to plain `rsync`, but `mattellite`
|
||||
has no key on the NAS (`agentapi@10.20.28.200: Permission denied`). The
|
||||
2026-08-20 run worked around it by publishing to a local directory and
|
||||
rsyncing to the NAS by hand with `sshpass` from the workstation. **Until
|
||||
mattellite's `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys`, any
|
||||
cron job will fetch successfully and then fail at the publish step.** That is
|
||||
the single thing standing between this and being fully unattended.
|
||||
|
||||
- **5.3 GB of stale staging on `mattellite`** — `~/gdl/staging` and `~/gdl/out`
|
||||
(2.2 GB each, from the 2026-08-17 run) and `~/gdl/staging-0820` /
|
||||
`~/gdl/out-0820` (446 MB each, from 2026-08-20). Every file in all four was
|
||||
|
||||
Reference in New Issue
Block a user