instaarchive-viewer/TOOLING.md

Tooling branch

Caution

This branch must not be pushed to GitHub

tooling is the only branch that still contains the archive-fetching scripts and their docs, and those name things main was rewritten to remove:

• the fetch host's public IP (docs/gallery-dl.md)
• the browser profile the session cookie is read from
• the NAS archive path
• the list of Instagram accounts being archived

On 2026-08-20 main's entire history was rewritten with git filter-repo, the GitHub repo was deleted and recreated, and 22 container images were pruned from ghcr — all to get exactly this material out of public view. One push of this branch to GitHub undoes all of it.

A second cleanup would be harder than the first: after a force-push the old commits stayed reachable by raw SHA, and only deleting the repository outright removed them.

Remotes

remote	what goes there
origin → gitea	everything: main, tooling, tags, backups
github	main and the current release tag only — it exists to run the CI/CD image build

The other 22 release tags stay on gitea. Pushing them all to GitHub triggers one container build per tag, because each tag carries its own workflow file.

Guards — recreate these after a fresh clone

Neither guard is versioned, so a new clone has no protection at all:

git config remote.github.push refs/heads/main:refs/heads/main

cat > .git/hooks/pre-push <<'HOOK'
#!/bin/sh
remote_url="$2"
case "$remote_url" in *github.com*) ;; *) exit 0 ;; esac
while read -r _ _ remote_ref _; do
  [ -z "$remote_ref" ] && continue
  case "$remote_ref" in
    refs/heads/main|refs/tags/*) ;;
    *) echo "pre-push: refusing to push '$remote_ref' to GitHub." >&2; exit 1 ;;
  esac
done
exit 0
HOOK
chmod +x .git/hooks/pre-push

Decide on the remote ref, not the local one: a delete push sends (delete) as the local ref, and an earlier version of this hook rejected every deletion because of it.

What lives here

path	what it is
scripts/gdl-sync.py	the gallery-dl fetcher; replaced JD2 for the ARTMS profiles
scripts/test_gdl_sync.py	its tests
scripts/jd2-sync.ts	JDownloader .crawljob generator, still used elsewhere
docs/gallery-dl.md	the measurements behind every option in the fetcher — read before changing pacing
docs/jdownloader.md	the older JD2 flow
docs/artms-instagram-accounts.txt	the profile list passed to --urls-file

Commands

# fetch: always --dry-run first; it prints the plan and the publish step
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \
    --staging <dir> --publish <user>@<nas>:<archives> \
    --archive-db <db> --urls-file artms_account_links.txt --abort 50 --dry-run

# crawljobs (no npm script — package.json is kept identical to main)
npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run

Run the fetcher from the host whose public IP matches the browser the cookie came from. --abort 50 is the routine setting; omit it for a full sweep that also catches edited carousels.

Why there is no CLAUDE.md entry for any of this

CLAUDE.md, README.md and package.json are kept byte-identical to main so that merging main into tooling never conflicts. The earlier attempt put tooling notes in CLAUDE.md and a jd2 script in package.json; because main had deleted those lines, every merge re-applied the deletion. Keep branch-specific documentation in this file, which main does not have.

Running it by hand

Everything happens on mattellite — the fetch host whose public IP matches the browser the cookie came from. Running it anywhere else is what session-hijack detection looks for.

ssh mattellite
~/gdl/gdl-cron.sh full        # or: stories | sweep

That is the whole thing: it wipes staging, fetches, and publishes straight to the NAS. To drive gdl-sync.py directly instead — always --dry-run first, which prints the plan and the exact rsync that would touch the archive:

cd ~/gdl
PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
    --index    https://instaarchive.ergosteur.com \
    --staging  ~/gdl/staging-manual \
    --publish  agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
    --archive-db ~/gdl/artms.db \
    --urls-file  ~/gdl/artms_account_links.txt \
    --abort 50 --dry-run          # swap for --execute when the plan looks right

PATH matters: gallery-dl is a pipx install in ~/.local/bin, which is not on cron's PATH and not on a non-login shell's either.

The three modes

mode	cadence	cost	why
stories	daily	~6 requests	stories expire in 24h and cannot be backfilled; this is the only run that loses content if skipped
full	monthly	~40-60 requests	every surface, --abort 50 — stops enumerating once it reaches content already held
sweep	quarterly	~420 requests	no abort; the only run that notices carousels edited after we archived them (test case 15)

The skip-archive means an infrequent full costs barely more than a frequent one — it only fetches what is new. Frequency buys freshness, not completeness, except for stories.

Cron

crontab -e

# Randomise the start: a job firing at exactly 09:00 every day is obviously a
# machine. `shuf` rather than $RANDOM, because cron runs /bin/sh, not bash --
# and avoid `%` in a crontab line, where it means "newline" unless escaped.
17 9  *  *  *      sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
43 4  3  *  *      sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full
11 4  7  1,4,7,10 *  sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep

The wrapper exits non-zero when the sync does, so cron will mail you on failure; it also tees every run to ~/gdl/logs/ and keeps the last 30.

systemd is the better fit if you want it, because the randomisation is built in rather than a sleep:

# ~/.config/systemd/user/gdl-stories.timer
[Timer]
OnCalendar=*-*-* 09:00:00
RandomizedDelaySec=45m
Persistent=true          # catches up after the host is off, which cron will not

with loginctl enable-linger matt so user timers run without a session. Persistent=true matters here: a laptop asleep at 09:00 silently skips a cron job, and a skipped stories run is content gone for good.

Outstanding

State as of 2026-08-20, after the sync run and the repo split. Nothing here is broken; these are decisions not yet made and cleanups not yet done.

Fetching

• The fetch host cannot yet publish to the NAS. gdl-sync.py --publish accepts user@host:/path and shells out to plain rsync, but mattellite has no key on the NAS (agentapi@10.20.28.200: Permission denied). The 2026-08-20 run worked around it by publishing to a local directory and rsyncing to the NAS by hand with sshpass from the workstation. Until mattellite's ~/.ssh/id_ed25519.pub is in the NAS's authorized_keys, any cron job will fetch successfully and then fail at the publish step. That is the single thing standing between this and being fully unattended.

• 5.3 GB of stale staging on mattellite — ~/gdl/staging and ~/gdl/out (2.2 GB each, from the 2026-08-17 run) and ~/gdl/staging-0820 / ~/gdl/out-0820 (446 MB each, from 2026-08-20). Every file in all four was verified present in the live archive, so they are safe to delete. 46 GB free, so there is no urgency — but nothing will clean them up on its own.

• No daily stories run is scheduled. Stories expire in 24h and cannot be backfilled, so this is the only surface where waiting loses content permanently. --only stories never seeds and costs roughly six requests for all six profiles. When scheduling it, randomise the minute and avoid the hour boundary: a job firing at exactly 09:00 daily is obviously a machine.

• --abort 50 is opt-in and nothing uses it yet. It is the right setting for routine runs — it cut a 2151-post profile to 7 enumerated posts — but it stops noticing edited carousels (test case 15), which only a full enumeration finds. A full-sweep cadence has not been decided; quarterly was suggested and never agreed.

• The seeded flags in <db>.state.json were hand-written, reconstructed from the 2026-08-17 log rather than derived from the archive DB. They assert "the skip-archive already knows this source". If artms.db is ever rebuilt, moved or lost, clear the state file too — otherwise those sources will never re-seed and a fetch into empty staging re-downloads everything.

• ~/gdl/gdl-sync.py on the fetch host is a copy, not a checkout. It currently matches this branch (96e5694e…), but nothing keeps them in sync; scp it after any change and re-check the hash.

• The 2026-08-20 run is split across two logs — artms-run3.log (12 sources, no abort) and artms-run4.log (12 sources, --abort 50) — because it was stopped midway to pick up the new flag.

Repo and infrastructure

• The pre-rewrite-* branches on gitea hold the unredacted history — real account names, the fetch host's IP, and the tooling, as it was before the rewrite. They are deliberate backups. Decide whether they expire; the pre-push hook does cover them (it allows only main and tags to GitHub).
• The pre-rewrite-full.bundle backup is in a session scratchpad and will be deleted with it. If a durable backup outside gitea is wanted, move it now.
• Only one container image exists. 22 versions were pruned, so rolling back to an older release means checking out its tag from gitea and pushing that tag to GitHub to rebuild it — the old images are gone, not archived.
• CI warns that the Node 20 actions are deprecated. actions/checkout@v4, docker/login-action@v3, docker/metadata-action@v5 and docker/build-push-action@v5 are being forced onto Node 24. They work today; bump when convenient.
• review-fixes on gitea is a stale v1.3.0-era branch, never merged, published only because the whole local repo was pushed. Probably deletable.
• GitHub Actions run history was lost when the repo was recreated. Cosmetic.