instaarchive-viewer/TOOLING.md

Tooling branch

Caution

This branch must not be pushed to GitHub

tooling is the only branch that still contains the archive-fetching scripts and their docs, and those name things main was rewritten to remove:

• the fetch host's public IP (docs/gallery-dl.md)
• the browser profile the session cookie is read from
• the NAS archive path
• the list of Instagram accounts being archived

On 2026-08-20 main's entire history was rewritten with git filter-repo, the GitHub repo was deleted and recreated, and 22 container images were pruned from ghcr — all to get exactly this material out of public view. One push of this branch to GitHub undoes all of it.

A second cleanup would be harder than the first: after a force-push the old commits stayed reachable by raw SHA, and only deleting the repository outright removed them.

Remotes

remote	what goes there
origin → gitea	everything: main, tooling, tags, backups
github	main and the current release tag only — it exists to run the CI/CD image build

The other 22 release tags stay on gitea. Pushing them all to GitHub triggers one container build per tag, because each tag carries its own workflow file.

Guards — recreate these after a fresh clone

Neither guard is versioned, so a new clone has no protection at all:

git config remote.github.push refs/heads/main:refs/heads/main

cat > .git/hooks/pre-push <<'HOOK'
#!/bin/sh
remote_url="$2"
case "$remote_url" in *github.com*) ;; *) exit 0 ;; esac
while read -r _ _ remote_ref _; do
  [ -z "$remote_ref" ] && continue
  case "$remote_ref" in
    refs/heads/main|refs/tags/*) ;;
    *) echo "pre-push: refusing to push '$remote_ref' to GitHub." >&2; exit 1 ;;
  esac
done
exit 0
HOOK
chmod +x .git/hooks/pre-push

Decide on the remote ref, not the local one: a delete push sends (delete) as the local ref, and an earlier version of this hook rejected every deletion because of it.

What lives here

path	what it is
scripts/gdl-sync.py	the gallery-dl fetcher; replaced JD2 for the ARTMS profiles
scripts/gdl-cron.sh	unattended wrapper: stories | full | sweep
scripts/systemd/	the timers actually installed on the fetch host
scripts/test_gdl_sync.py	its tests
scripts/jd2-sync.ts	JDownloader .crawljob generator, still used elsewhere
docs/gallery-dl.md	the measurements behind every option in the fetcher — read before changing pacing
docs/jdownloader.md	the older JD2 flow
docs/artms-instagram-accounts.txt	the profile list passed to --urls-file

Commands

# fetch: always --dry-run first; it prints the plan and the publish step
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \
    --staging <dir> --publish <user>@<nas>:<archives> \
    --archive-db <db> --urls-file artms_account_links.txt --abort 50 --dry-run

# crawljobs (no npm script — package.json is kept identical to main)
npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run

Run the fetcher from the host whose public IP matches the browser the cookie came from. --abort 50 is the routine setting; omit it for a full sweep that also catches edited carousels.

Why there is no CLAUDE.md entry for any of this

CLAUDE.md, README.md and package.json are kept byte-identical to main so that merging main into tooling never conflicts. The earlier attempt put tooling notes in CLAUDE.md and a jd2 script in package.json; because main had deleted those lines, every merge re-applied the deletion. Keep branch-specific documentation in this file, which main does not have.

Running it by hand

Everything happens on mattellite — the fetch host whose public IP matches the browser the cookie came from. Running it anywhere else is what session-hijack detection looks for.

ssh mattellite
~/gdl/gdl-cron.sh full        # or: stories | sweep

That is the whole thing: it wipes staging, fetches, and publishes straight to the NAS. To drive gdl-sync.py directly instead — always --dry-run first, which prints the plan and the exact rsync that would touch the archive:

cd ~/gdl
PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
    --index    https://instaarchive.ergosteur.com \
    --staging  ~/gdl/staging-manual \
    --publish  agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
    --archive-db ~/gdl/artms.db \
    --urls-file  ~/gdl/artms_account_links.txt \
    --abort 50 --dry-run          # swap for --execute when the plan looks right

PATH matters: gallery-dl is a pipx install in ~/.local/bin, which is not on cron's PATH and not on a non-login shell's either.

The three modes

mode	cadence	cost	why
stories	daily	~6 requests	stories expire in 24h and cannot be backfilled; this is the only run that loses content if skipped
full	monthly	~40-60 requests	every surface, --abort 50 — stops enumerating once it reaches content already held
sweep	quarterly	~420 requests	no abort; the only run that notices carousels edited after we archived them (test case 15)

The skip-archive means an infrequent full costs barely more than a frequent one — it only fetches what is new. Frequency buys freshness, not completeness, except for stories.

Scheduling — installed on mattellite

systemd user timers, running as matt, with lingering enabled so they fire without a login session:

loginctl show-user matt --property=Linger      # Linger=yes
systemctl --user list-timers 'gdl-sync@*'

unit	schedule	next fire (as installed)
gdl-sync@stories.timer	daily 09:00	09:36:45 — the delay is the randomisation working
gdl-sync@full.timer	3rd of each month, 04:00	04:37:44
gdl-sync@sweep.timer	7th of Jan/Apr/Jul/Oct, 04:00	04:42:39

Unit files are version-controlled in scripts/systemd/ and installed to ~/.config/systemd/user/. One templated service, gdl-sync@.service, takes the mode as its instance name and runs gdl-cron.sh %i.

Three settings are load-bearing:

• RandomizedDelaySec=45m — a job firing at exactly 09:00 daily is obviously a machine, and the entire safety model is about not looking like one. This is why the table above shows 09:36 rather than 09:00.
• Persistent=true — catch up a run missed because the host was off. cron silently skips, and a skipped stories run is content gone for good.
• TimeoutStartSec=infinity — a sweep can run for hours at this pacing. The default 90s would kill it mid-fetch.

Operating them:

export XDG_RUNTIME_DIR=/run/user/$(id -u)      # needed over non-interactive ssh
systemctl --user start   gdl-sync@stories.service   # run one now
systemctl --user status  gdl-sync@full.timer
journalctl --user -u 'gdl-sync@*' -n 50
systemctl --user disable --now gdl-sync@sweep.timer  # stop one

systemctl --user fails with "Failed to connect to bus" over ssh unless XDG_RUNTIME_DIR is set. Note also that month names are not valid in OnCalendar's date field — Jan,Apr,Jul,Oct-07 is rejected outright, hence *-01,04,07,10-07. Check any change with systemd-analyze calendar '<expr>' before installing it.

cron, if you ever prefer it

17 9  *  *  *        sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
43 4  3  *  *        sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full
11 4  7  1,4,7,10 *  sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep

cron runs /bin/sh, so $RANDOM does not exist — hence shuf. And % in a crontab line means newline unless escaped, so avoid it entirely. cron has no equivalent of Persistent=true.

Outstanding

State as of 2026-08-20, after the sync run and the repo split. Nothing here is broken; these are decisions not yet made and cleanups not yet done.

Fetching

• mattellite's ~/.ssh/id_ed25519.pub is in the NAS's authorized_keys for agentapi (added 2026-08-20, alongside the workstation's existing key), so the fetch host publishes straight to the archive and no sshpass step is needed. That key is what makes the timers work — remove it and every scheduled run will fetch successfully and then fail at publish.

• 5.3 GB of stale staging on mattellite — ~/gdl/staging and ~/gdl/out (2.2 GB each, from the 2026-08-17 run) and ~/gdl/staging-0820 / ~/gdl/out-0820 (446 MB each, from 2026-08-20). Every file in all four was verified present in the live archive, so they are safe to delete. 46 GB free, so there is no urgency — but nothing will clean them up on its own.

• No daily stories run is scheduled. Stories expire in 24h and cannot be backfilled, so this is the only surface where waiting loses content permanently. --only stories never seeds and costs roughly six requests for all six profiles. When scheduling it, randomise the minute and avoid the hour boundary: a job firing at exactly 09:00 daily is obviously a machine.

• --abort 50 is opt-in and nothing uses it yet. It is the right setting for routine runs — it cut a 2151-post profile to 7 enumerated posts — but it stops noticing edited carousels (test case 15), which only a full enumeration finds. A full-sweep cadence has not been decided; quarterly was suggested and never agreed.

• The seeded flags in <db>.state.json were hand-written, reconstructed from the 2026-08-17 log rather than derived from the archive DB. They assert "the skip-archive already knows this source". If artms.db is ever rebuilt, moved or lost, clear the state file too — otherwise those sources will never re-seed and a fetch into empty staging re-downloads everything.

• ~/gdl/gdl-sync.py on the fetch host is a copy, not a checkout. It currently matches this branch (96e5694e…), but nothing keeps them in sync; scp it after any change and re-check the hash.

• The 2026-08-20 run is split across two logs — artms-run3.log (12 sources, no abort) and artms-run4.log (12 sources, --abort 50) — because it was stopped midway to pick up the new flag.

Repo and infrastructure

• The pre-rewrite-* branches on gitea hold the unredacted history — real account names, the fetch host's IP, and the tooling, as it was before the rewrite. They are deliberate backups. Decide whether they expire; the pre-push hook does cover them (it allows only main and tags to GitHub).
• The pre-rewrite-full.bundle backup is in a session scratchpad and will be deleted with it. If a durable backup outside gitea is wanted, move it now.
• Only one container image exists. 22 versions were pruned, so rolling back to an older release means checking out its tag from gitea and pushing that tag to GitHub to rebuild it — the old images are gone, not archived.
• CI warns that the Node 20 actions are deprecated. actions/checkout@v4, docker/login-action@v3, docker/metadata-action@v5 and docker/build-push-action@v5 are being forced onto Node 24. They work today; bump when convenient.
• review-fixes on gitea is a stale v1.3.0-era branch, never merged, published only because the whole local repo was pushed. Probably deletable.
• GitHub Actions run history was lost when the repo was recreated. Cosmetic.