ergosteurandClaude Opus 5
652747e0f1
mattellite's key is now in the NAS's authorized_keys for agentapi, so
gdl-sync.py publishes straight to the archive. That was the last manual step:
the 2026-08-20 run published locally and rsynced by hand with sshpass, which
no scheduled job could do. Verified end to end -- key auth, then a dry-run
rsync from the fetch host into the archive root.
Three timers, installed and enabled: stories daily, full monthly, sweep
quarterly. A templated gdl-sync@.service takes the mode as its instance name.
Settings that are load-bearing rather than decoration:
RandomizedDelaySec=45m a job firing at exactly 09:00 daily is obviously a
machine; list-timers now shows 09:36, not 09:00
Persistent=true catch up a run the host slept through -- cron just
skips, and a skipped stories run is content gone
TimeoutStartSec=infinity a sweep runs for hours at this pacing; the 90s
default would kill it mid-fetch
Two things caught while installing, both documented: month names are invalid
in OnCalendar's date field (Jan,Apr,Jul,Oct-07 is rejected -- use numerics,
and check with systemd-analyze calendar), and `systemctl --user` over
non-interactive ssh needs XDG_RUNTIME_DIR or it cannot find the bus.
Smoke-tested by starting the stories service: it spent zero Instagram requests
because the 20h floor skipped all six sources, which is exactly what a job
firing twice should do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF
255 lines
11 KiB
Markdown
255 lines
11 KiB
Markdown
# Tooling branch
|
|
|
|
> [!CAUTION]
|
|
> ## This branch must not be pushed to GitHub
|
|
>
|
|
> `tooling` is the only branch that still contains the archive-fetching
|
|
> scripts and their docs, and those name things `main` was rewritten to
|
|
> remove:
|
|
>
|
|
> - the fetch host's **public IP** (`docs/gallery-dl.md`)
|
|
> - the browser profile the session cookie is read from
|
|
> - the NAS archive path
|
|
> - the **list of Instagram accounts being archived**
|
|
>
|
|
> On 2026-08-20 `main`'s entire history was rewritten with `git filter-repo`,
|
|
> the GitHub repo was deleted and recreated, and 22 container images were
|
|
> pruned from ghcr — all to get exactly this material out of public view.
|
|
> **One push of this branch to GitHub undoes all of it.**
|
|
>
|
|
> A second cleanup would be harder than the first: after a force-push the old
|
|
> commits stayed reachable by raw SHA, and only deleting the repository
|
|
> outright removed them.
|
|
|
|
## Remotes
|
|
|
|
| remote | what goes there |
|
|
|---|---|
|
|
| `origin` → gitea | everything: `main`, `tooling`, tags, backups |
|
|
| `github` | **`main` and the current release tag only** — it exists to run the CI/CD image build |
|
|
|
|
The other 22 release tags stay on gitea. Pushing them all to GitHub triggers
|
|
one container build per tag, because each tag carries its own workflow file.
|
|
|
|
## Guards — recreate these after a fresh clone
|
|
|
|
Neither guard is versioned, so a new clone has **no protection at all**:
|
|
|
|
```sh
|
|
git config remote.github.push refs/heads/main:refs/heads/main
|
|
|
|
cat > .git/hooks/pre-push <<'HOOK'
|
|
#!/bin/sh
|
|
remote_url="$2"
|
|
case "$remote_url" in *github.com*) ;; *) exit 0 ;; esac
|
|
while read -r _ _ remote_ref _; do
|
|
[ -z "$remote_ref" ] && continue
|
|
case "$remote_ref" in
|
|
refs/heads/main|refs/tags/*) ;;
|
|
*) echo "pre-push: refusing to push '$remote_ref' to GitHub." >&2; exit 1 ;;
|
|
esac
|
|
done
|
|
exit 0
|
|
HOOK
|
|
chmod +x .git/hooks/pre-push
|
|
```
|
|
|
|
Decide on the **remote** ref, not the local one: a delete push sends
|
|
`(delete)` as the local ref, and an earlier version of this hook rejected
|
|
every deletion because of it.
|
|
|
|
## What lives here
|
|
|
|
| path | what it is |
|
|
|---|---|
|
|
| `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
|
|
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `full` \| `sweep` |
|
|
| `scripts/systemd/` | the timers actually installed on the fetch host |
|
|
| `scripts/test_gdl_sync.py` | its tests |
|
|
| `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
|
|
| `docs/gallery-dl.md` | the measurements behind every option in the fetcher — **read before changing pacing** |
|
|
| `docs/jdownloader.md` | the older JD2 flow |
|
|
| `docs/artms-instagram-accounts.txt` | the profile list passed to `--urls-file` |
|
|
|
|
## Commands
|
|
|
|
```sh
|
|
# fetch: always --dry-run first; it prints the plan and the publish step
|
|
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \
|
|
--staging <dir> --publish <user>@<nas>:<archives> \
|
|
--archive-db <db> --urls-file artms_account_links.txt --abort 50 --dry-run
|
|
|
|
# crawljobs (no npm script — package.json is kept identical to main)
|
|
npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run
|
|
```
|
|
|
|
Run the fetcher from the host whose public IP matches the browser the cookie
|
|
came from. `--abort 50` is the routine setting; omit it for a full sweep that
|
|
also catches edited carousels.
|
|
|
|
## Why there is no CLAUDE.md entry for any of this
|
|
|
|
`CLAUDE.md`, `README.md` and `package.json` are kept **byte-identical** to
|
|
`main` so that merging `main` into `tooling` never conflicts. The earlier
|
|
attempt put tooling notes in `CLAUDE.md` and a `jd2` script in `package.json`;
|
|
because `main` had *deleted* those lines, every merge re-applied the deletion.
|
|
Keep branch-specific documentation in this file, which `main` does not have.
|
|
|
|
## Running it by hand
|
|
|
|
Everything happens on **`mattellite`** — the fetch host whose public IP matches
|
|
the browser the cookie came from. Running it anywhere else is what
|
|
session-hijack detection looks for.
|
|
|
|
```sh
|
|
ssh mattellite
|
|
~/gdl/gdl-cron.sh full # or: stories | sweep
|
|
```
|
|
|
|
That is the whole thing: it wipes staging, fetches, and publishes straight to
|
|
the NAS. To drive `gdl-sync.py` directly instead — always `--dry-run` first,
|
|
which prints the plan and the exact rsync that would touch the archive:
|
|
|
|
```sh
|
|
cd ~/gdl
|
|
PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
|
|
--index https://instaarchive.ergosteur.com \
|
|
--staging ~/gdl/staging-manual \
|
|
--publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
|
|
--archive-db ~/gdl/artms.db \
|
|
--urls-file ~/gdl/artms_account_links.txt \
|
|
--abort 50 --dry-run # swap for --execute when the plan looks right
|
|
```
|
|
|
|
`PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not
|
|
on cron's PATH and not on a non-login shell's either.
|
|
|
|
### The three modes
|
|
|
|
| mode | cadence | cost | why |
|
|
|---|---|---|---|
|
|
| `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped |
|
|
| `full` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating once it reaches content already held |
|
|
| `sweep` | quarterly | ~420 requests | no abort; the **only** run that notices carousels edited after we archived them (test case 15) |
|
|
|
|
The skip-archive means an infrequent `full` costs barely more than a frequent
|
|
one — it only fetches what is new. Frequency buys freshness, not completeness,
|
|
except for stories.
|
|
|
|
## Scheduling — installed on `mattellite`
|
|
|
|
systemd **user** timers, running as `matt`, with lingering enabled so they fire
|
|
without a login session:
|
|
|
|
```sh
|
|
loginctl show-user matt --property=Linger # Linger=yes
|
|
systemctl --user list-timers 'gdl-sync@*'
|
|
```
|
|
|
|
| unit | schedule | next fire (as installed) |
|
|
|---|---|---|
|
|
| `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working |
|
|
| `gdl-sync@full.timer` | 3rd of each month, 04:00 | 04:37:44 |
|
|
| `gdl-sync@sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
|
|
|
|
Unit files are version-controlled in `scripts/systemd/` and installed to
|
|
`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
|
|
the mode as its instance name and runs `gdl-cron.sh %i`.
|
|
|
|
Three settings are load-bearing:
|
|
|
|
- **`RandomizedDelaySec=45m`** — a job firing at exactly 09:00 daily is
|
|
obviously a machine, and the entire safety model is about not looking like
|
|
one. This is why the table above shows 09:36 rather than 09:00.
|
|
- **`Persistent=true`** — catch up a run missed because the host was off.
|
|
cron silently skips, and a skipped `stories` run is content gone for good.
|
|
- **`TimeoutStartSec=infinity`** — a sweep can run for hours at this pacing.
|
|
The default 90s would kill it mid-fetch.
|
|
|
|
Operating them:
|
|
|
|
```sh
|
|
export XDG_RUNTIME_DIR=/run/user/$(id -u) # needed over non-interactive ssh
|
|
systemctl --user start gdl-sync@stories.service # run one now
|
|
systemctl --user status gdl-sync@full.timer
|
|
journalctl --user -u 'gdl-sync@*' -n 50
|
|
systemctl --user disable --now gdl-sync@sweep.timer # stop one
|
|
```
|
|
|
|
`systemctl --user` fails with "Failed to connect to bus" over ssh unless
|
|
`XDG_RUNTIME_DIR` is set. Note also that **month names are not valid in
|
|
`OnCalendar`'s date field** — `Jan,Apr,Jul,Oct-07` is rejected outright, hence
|
|
`*-01,04,07,10-07`. Check any change with `systemd-analyze calendar '<expr>'`
|
|
before installing it.
|
|
|
|
### cron, if you ever prefer it
|
|
|
|
```cron
|
|
17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
|
|
43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full
|
|
11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep
|
|
```
|
|
|
|
cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a
|
|
crontab line means newline unless escaped, so avoid it entirely. cron has no
|
|
equivalent of `Persistent=true`.
|
|
|
|
## Outstanding
|
|
|
|
State as of 2026-08-20, after the sync run and the repo split. Nothing here is
|
|
broken; these are decisions not yet made and cleanups not yet done.
|
|
|
|
### Fetching
|
|
|
|
- `mattellite`'s `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys` for
|
|
`agentapi` (added 2026-08-20, alongside the workstation's existing key), so
|
|
the fetch host publishes straight to the archive and no `sshpass` step is
|
|
needed. **That key is what makes the timers work** — remove it and every
|
|
scheduled run will fetch successfully and then fail at publish.
|
|
|
|
- **5.3 GB of stale staging on `mattellite`** — `~/gdl/staging` and `~/gdl/out`
|
|
(2.2 GB each, from the 2026-08-17 run) and `~/gdl/staging-0820` /
|
|
`~/gdl/out-0820` (446 MB each, from 2026-08-20). Every file in all four was
|
|
verified present in the live archive, so they are safe to delete. 46 GB free,
|
|
so there is no urgency — but nothing will clean them up on its own.
|
|
- **No daily stories run is scheduled.** Stories expire in 24h and cannot be
|
|
backfilled, so this is the *only* surface where waiting loses content
|
|
permanently. `--only stories` never seeds and costs roughly six requests for
|
|
all six profiles. When scheduling it, randomise the minute and avoid the hour
|
|
boundary: a job firing at exactly 09:00 daily is obviously a machine.
|
|
- **`--abort 50` is opt-in and nothing uses it yet.** It is the right setting
|
|
for routine runs — it cut a 2151-post profile to 7 enumerated posts — but it
|
|
stops noticing **edited carousels** (test case 15), which only a full
|
|
enumeration finds. A full-sweep cadence has not been decided; quarterly was
|
|
suggested and never agreed.
|
|
- **The `seeded` flags in `<db>.state.json` were hand-written**, reconstructed
|
|
from the 2026-08-17 log rather than derived from the archive DB. They assert
|
|
"the skip-archive already knows this source". If `artms.db` is ever rebuilt,
|
|
moved or lost, **clear the state file too** — otherwise those sources will
|
|
never re-seed and a fetch into empty staging re-downloads everything.
|
|
- **`~/gdl/gdl-sync.py` on the fetch host is a copy, not a checkout.** It
|
|
currently matches this branch (`96e5694e…`), but nothing keeps them in sync;
|
|
`scp` it after any change and re-check the hash.
|
|
- The 2026-08-20 run is split across two logs — `artms-run3.log` (12 sources,
|
|
no abort) and `artms-run4.log` (12 sources, `--abort 50`) — because it was
|
|
stopped midway to pick up the new flag.
|
|
|
|
### Repo and infrastructure
|
|
|
|
- **The `pre-rewrite-*` branches on gitea hold the unredacted history** — real
|
|
account names, the fetch host's IP, and the tooling, as it was before the
|
|
rewrite. They are deliberate backups. Decide whether they expire; the
|
|
`pre-push` hook does cover them (it allows only `main` and tags to GitHub).
|
|
- **The `pre-rewrite-full.bundle` backup is in a session scratchpad** and will
|
|
be deleted with it. If a durable backup outside gitea is wanted, move it now.
|
|
- **Only one container image exists.** 22 versions were pruned, so rolling back
|
|
to an older release means checking out its tag from gitea and pushing that
|
|
tag to GitHub to rebuild it — the old images are gone, not archived.
|
|
- **CI warns that the Node 20 actions are deprecated.** `actions/checkout@v4`,
|
|
`docker/login-action@v3`, `docker/metadata-action@v5` and
|
|
`docker/build-push-action@v5` are being forced onto Node 24. They work today;
|
|
bump when convenient.
|
|
- **`review-fixes`** on gitea is a stale v1.3.0-era branch, never merged,
|
|
published only because the whole local repo was pushed. Probably deletable.
|
|
- GitHub Actions run history was lost when the repo was recreated. Cosmetic.
|