diff --git a/TOOLING.md b/TOOLING.md index 54e4bbf..b83cd9b 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -194,6 +194,86 @@ cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a crontab line means newline unless escaped, so avoid it entirely. cron has no equivalent of `Persistent=true`. +## Verifying a run + +The first unattended run is **2026-08-21, around 09:36** (09:00 plus the +randomised delay). Nothing below costs an Instagram request — every check is +against the journal, local logs, or our own viewer's API. + +```sh +# 1. did it run, and did it exit 0? +ssh mattellite +export XDG_RUNTIME_DIR=/run/user/$(id -u) +systemctl --user list-timers 'gdl-sync@*' # LAST/PASSED columns +journalctl --user -u 'gdl-sync@stories.service' --since yesterday --no-pager + +# 2. what did it actually fetch? (sidecars vastly outnumber media -- count media) +ls -1t ~/gdl/logs | head -3 +grep -E '^(==>| FAILED|done;)' ~/gdl/logs/stories-*.log | tail -20 +grep -ic 429 ~/gdl/logs/stories-*.log # MUST be 0 -- see below +``` + +**A `429` from `scontent-*.cdninstagram.com` ends the session, it is not a +pacing knob to tune.** The warning order last time was CDN 429 → `400` on the +highlights endpoint → suspension. If a run logs one, disable the timers and +stop for the day: + +```sh +systemctl --user disable --now gdl-sync@stories.timer gdl-sync@full.timer gdl-sync@sweep.timer +``` + +Then confirm the archive actually grew, from the workstation: + +```sh +# 3. did the publish land? compare against yesterday's counts +for u in 0ct0ber19 kimxxlip withaseul cher_ryppo zindoriyam official_artms; do + n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))') + printf '%-18s %s\n' "$u" "$n" +done +``` + +Counts after the 2026-08-20 run, to diff against: + +| profile | files | +|---|---:| +| 0ct0ber19 | 3151 | +| official_artms | 6714 | +| cher_ryppo | 3031 | +| kimxxlip | 3019 | +| zindoriyam | 2253 | +| withaseul | 1760 | + +A stories-only run adds few files and often **none** — profiles frequently have +no active story. "0 new" is a normal result, not a failure. `fileCount` in +`/api/archives` is stale by design; use the per-profile `/files` listing. + +## What changed on 2026-08-20 + +One session, three separate pieces of work. Recorded because the reasons are +not recoverable from the diffs. + +**The sync run.** First incremental fetch in four days: 184 new media, 299 +files published, 0 failures, 0 CDN 429s. 20 story items, which are the part +that could not have been recovered later. Cost about half what it would have, +because the archive DB was already seeded and the state file was primed by hand +so no probe passes ran. + +**`--abort 50`.** The skip-archive suppresses *downloads*, which spends the +CDN; it does nothing about the *listing pass*, which spends `instagram.com` and +scales with how big a profile is rather than how much is new. Measured from +sidecar write times mid-run: 3 new posts took ~100s each, the other 2272 were +written in one second. Enumerating `cher_ryppo` fell from 2151 posts to 7. + +**The repo split.** `main` is public and now carries none of the fetching +tooling, no host details, and no real account names — its entire history was +rewritten, the GitHub repo deleted and recreated to clear force-push residue, +and 22 container images pruned from ghcr because the server bundle had been +shipping source comments naming real accounts. This branch holds everything +that was removed. See the caution at the top. + +**Automation.** mattellite got a key on the NAS, closing the last manual step, +and three systemd timers now run the sync unattended. + ## Outstanding State as of 2026-08-20, after the sync run and the repo split. Nothing here is @@ -201,6 +281,10 @@ broken; these are decisions not yet made and cleanups not yet done. ### Fetching +- **The first unattended run has not happened yet** — 2026-08-21 ~09:36. Until + it has, the timers are unproven in the one condition that matters: firing + with nobody watching. Check it with "Verifying a run" above; the 20h floor + means a manual run beforehand would make the automatic one a no-op. - `mattellite`'s `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys` for `agentapi` (added 2026-08-20, alongside the workstation's existing key), so the fetch host publishes straight to the archive and no `sshpass` step is