# Tooling branch > [!CAUTION] > ## This branch must not be pushed to GitHub > > `tooling` is the only branch that still contains the archive-fetching > scripts and their docs, and those name things `main` was rewritten to > remove: > > - the fetch host's **public IP** (`docs/gallery-dl.md`) > - the browser profile the session cookie is read from > - the NAS archive path > - the **list of Instagram accounts being archived** > > On 2026-08-20 `main`'s entire history was rewritten with `git filter-repo`, > the GitHub repo was deleted and recreated, and 22 container images were > pruned from ghcr — all to get exactly this material out of public view. > **One push of this branch to GitHub undoes all of it.** > > A second cleanup would be harder than the first: after a force-push the old > commits stayed reachable by raw SHA, and only deleting the repository > outright removed them. ## Remotes | remote | what goes there | |---|---| | `origin` → gitea | everything: `main`, `tooling`, tags, backups | | `github` | **`main` and the current release tag only** — it exists to run the CI/CD image build | The other 22 release tags stay on gitea. Pushing them all to GitHub triggers one container build per tag, because each tag carries its own workflow file. ## Guards — recreate these after a fresh clone Neither guard is versioned, so a new clone has **no protection at all**: ```sh git config remote.github.push refs/heads/main:refs/heads/main cat > .git/hooks/pre-push <<'HOOK' #!/bin/sh remote_url="$2" case "$remote_url" in *github.com*) ;; *) exit 0 ;; esac while read -r _ _ remote_ref _; do [ -z "$remote_ref" ] && continue case "$remote_ref" in refs/heads/main|refs/tags/*) ;; *) echo "pre-push: refusing to push '$remote_ref' to GitHub." >&2; exit 1 ;; esac done exit 0 HOOK chmod +x .git/hooks/pre-push ``` Decide on the **remote** ref, not the local one: a delete push sends `(delete)` as the local ref, and an earlier version of this hook rejected every deletion because of it. ## What lives here | path | what it is | |---|---| | `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles | | `scripts/test_gdl_sync.py` | its tests | | `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere | | `docs/gallery-dl.md` | the measurements behind every option in the fetcher — **read before changing pacing** | | `docs/jdownloader.md` | the older JD2 flow | | `docs/artms-instagram-accounts.txt` | the profile list passed to `--urls-file` | ## Commands ```sh # fetch: always --dry-run first; it prints the plan and the publish step ./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \ --staging --publish @: \ --archive-db --urls-file artms_account_links.txt --abort 50 --dry-run # crawljobs (no npm script — package.json is kept identical to main) npx tsx scripts/jd2-sync.ts --archives --dry-run ``` Run the fetcher from the host whose public IP matches the browser the cookie came from. `--abort 50` is the routine setting; omit it for a full sweep that also catches edited carousels. ## Why there is no CLAUDE.md entry for any of this `CLAUDE.md`, `README.md` and `package.json` are kept **byte-identical** to `main` so that merging `main` into `tooling` never conflicts. The earlier attempt put tooling notes in `CLAUDE.md` and a `jd2` script in `package.json`; because `main` had *deleted* those lines, every merge re-applied the deletion. Keep branch-specific documentation in this file, which `main` does not have. ## Running it by hand Everything happens on **`mattellite`** — the fetch host whose public IP matches the browser the cookie came from. Running it anywhere else is what session-hijack detection looks for. ```sh ssh mattellite ~/gdl/gdl-cron.sh full # or: stories | sweep ``` That is the whole thing: it wipes staging, fetches, and publishes straight to the NAS. To drive `gdl-sync.py` directly instead — always `--dry-run` first, which prints the plan and the exact rsync that would touch the archive: ```sh cd ~/gdl PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \ --index https://instaarchive.ergosteur.com \ --staging ~/gdl/staging-manual \ --publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \ --archive-db ~/gdl/artms.db \ --urls-file ~/gdl/artms_account_links.txt \ --abort 50 --dry-run # swap for --execute when the plan looks right ``` `PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not on cron's PATH and not on a non-login shell's either. ### The three modes | mode | cadence | cost | why | |---|---|---|---| | `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped | | `full` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating once it reaches content already held | | `sweep` | quarterly | ~420 requests | no abort; the **only** run that notices carousels edited after we archived them (test case 15) | The skip-archive means an infrequent `full` costs barely more than a frequent one — it only fetches what is new. Frequency buys freshness, not completeness, except for stories. ## Cron ```sh crontab -e ``` ```cron # Randomise the start: a job firing at exactly 09:00 every day is obviously a # machine. `shuf` rather than $RANDOM, because cron runs /bin/sh, not bash -- # and avoid `%` in a crontab line, where it means "newline" unless escaped. 17 9 * * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories 43 4 3 * * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full 11 4 7 1,4,7,10 * sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh sweep ``` The wrapper exits non-zero when the sync does, so cron will mail you on failure; it also tees every run to `~/gdl/logs/` and keeps the last 30. **systemd is the better fit if you want it**, because the randomisation is built in rather than a `sleep`: ```ini # ~/.config/systemd/user/gdl-stories.timer [Timer] OnCalendar=*-*-* 09:00:00 RandomizedDelaySec=45m Persistent=true # catches up after the host is off, which cron will not ``` with `loginctl enable-linger matt` so user timers run without a session. `Persistent=true` matters here: a laptop asleep at 09:00 silently skips a cron job, and a skipped stories run is content gone for good. ## Outstanding State as of 2026-08-20, after the sync run and the repo split. Nothing here is broken; these are decisions not yet made and cleanups not yet done. ### Fetching - **The fetch host cannot yet publish to the NAS.** `gdl-sync.py --publish` accepts `user@host:/path` and shells out to plain `rsync`, but `mattellite` has no key on the NAS (`agentapi@10.20.28.200: Permission denied`). The 2026-08-20 run worked around it by publishing to a local directory and rsyncing to the NAS by hand with `sshpass` from the workstation. **Until mattellite's `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys`, any cron job will fetch successfully and then fail at the publish step.** That is the single thing standing between this and being fully unattended. - **5.3 GB of stale staging on `mattellite`** — `~/gdl/staging` and `~/gdl/out` (2.2 GB each, from the 2026-08-17 run) and `~/gdl/staging-0820` / `~/gdl/out-0820` (446 MB each, from 2026-08-20). Every file in all four was verified present in the live archive, so they are safe to delete. 46 GB free, so there is no urgency — but nothing will clean them up on its own. - **No daily stories run is scheduled.** Stories expire in 24h and cannot be backfilled, so this is the *only* surface where waiting loses content permanently. `--only stories` never seeds and costs roughly six requests for all six profiles. When scheduling it, randomise the minute and avoid the hour boundary: a job firing at exactly 09:00 daily is obviously a machine. - **`--abort 50` is opt-in and nothing uses it yet.** It is the right setting for routine runs — it cut a 2151-post profile to 7 enumerated posts — but it stops noticing **edited carousels** (test case 15), which only a full enumeration finds. A full-sweep cadence has not been decided; quarterly was suggested and never agreed. - **The `seeded` flags in `.state.json` were hand-written**, reconstructed from the 2026-08-17 log rather than derived from the archive DB. They assert "the skip-archive already knows this source". If `artms.db` is ever rebuilt, moved or lost, **clear the state file too** — otherwise those sources will never re-seed and a fetch into empty staging re-downloads everything. - **`~/gdl/gdl-sync.py` on the fetch host is a copy, not a checkout.** It currently matches this branch (`96e5694e…`), but nothing keeps them in sync; `scp` it after any change and re-check the hash. - The 2026-08-20 run is split across two logs — `artms-run3.log` (12 sources, no abort) and `artms-run4.log` (12 sources, `--abort 50`) — because it was stopped midway to pick up the new flag. ### Repo and infrastructure - **The `pre-rewrite-*` branches on gitea hold the unredacted history** — real account names, the fetch host's IP, and the tooling, as it was before the rewrite. They are deliberate backups. Decide whether they expire; the `pre-push` hook does cover them (it allows only `main` and tags to GitHub). - **The `pre-rewrite-full.bundle` backup is in a session scratchpad** and will be deleted with it. If a durable backup outside gitea is wanted, move it now. - **Only one container image exists.** 22 versions were pruned, so rolling back to an older release means checking out its tag from gitea and pushing that tag to GitHub to rebuild it — the old images are gone, not archived. - **CI warns that the Node 20 actions are deprecated.** `actions/checkout@v4`, `docker/login-action@v3`, `docker/metadata-action@v5` and `docker/build-push-action@v5` are being forced onto Node 24. They work today; bump when convenient. - **`review-fixes`** on gitea is a stale v1.3.0-era branch, never merged, published only because the whole local repo was pushed. Probably deletable. - GitHub Actions run history was lost when the repo was recreated. Cosmetic.