instaarchive-viewer/TOOLING.md

Tooling branch

Caution

This branch must not be pushed to GitHub

tooling is the only branch that still contains the archive-fetching scripts and their docs, and those name things main was rewritten to remove:

• the fetch host's public IP (docs/gallery-dl.md)
• the browser profile the session cookie is read from
• the NAS archive path
• the list of Instagram accounts being archived

On 2026-08-20 main's entire history was rewritten with git filter-repo, the GitHub repo was deleted and recreated, and 22 container images were pruned from ghcr — all to get exactly this material out of public view. One push of this branch to GitHub undoes all of it.

A second cleanup would be harder than the first: after a force-push the old commits stayed reachable by raw SHA, and only deleting the repository outright removed them.

Quick reference — getting new content onto the site

For the whole thing by hand, step by step (including the DH2300 / cosmo_web hop), see MANUAL-RUN.md.

The three systemd timers (see "Scheduling" below) run these unattended; run them by hand only to get content sooner. Everything runs on mattellite; it fetches, publishes to the NAS itself, and the site picks it up with no deploy or restart.

ssh mattellite

~/gdl/run-stories.sh      # ~1 min.    DO THIS OFTEN - stories die in 24h
~/gdl/run-profiles.sh     # ~5 min.    posts, reels, highlights, stories
~/gdl/run-full-sweep.sh   # rarely, ~420 requests. Read its own comment first.

Fixed-mode entry points around gdl-cron.sh, so the mode never has to be remembered or typed — each just runs its own thing with the right flags already set. run-profiles.sh covers every surface, so it is the one to run if you only run one, but it is no substitute for run-stories.sh: a story posted and expired between two run-profiles.sh runs is simply gone. Both are safe to run back-to-back; the --min-interval floor makes a repeat a no-op rather than a re-fetch.

run-reels-all.sh (browser-scrolling) is no longer needed for routine use as of 2026-09-14 — see "Reels: the API is blocked" below, now fixed upstream. run-profiles.sh's own reels surface covers it again.

Then check https://instaarchive.ergosteur.com — new posts appear without a redeploy. The archive index re-reads a directory when its mtime changes.

Reading the output. These two look alarming and are fine:

• rsync error: ... (code 23) / done; 1 step(s) failed — the chown to rslsync failing because the ssh user is not root. Files landed.
• No results for a profile — it has no active story right now.

These mean stop, and are covered in "When a run fails":

• 429 from scontent-*.cdninstagram.com
• 400 Bad Request on /api/v1/feed/reels_media/ — the account is behind a scraping-warning interstitial; check the browser before anything else.

Long runs: setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 & (or run-full-sweep.sh / run-reels-all.sh) and tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1). Nothing reaches the archive until a run finishes, so killing one midway is safe.

2026-10-14 — the Instagram session died unnoticed; guards added

What happened. Between 2026-10-06 09:12 and 2026-10-07 04:36 the dedicated Chrome's Instagram session died. Nothing alerted: the 10-07 full sweep logged HTTP redirect to login page on 17 of 18 sources (and "failed"), but every stories run after it fetched "No results" for all six profiles and still exited 0 — about a week of stories, which cannot be backfilled, was lost. Two compounding causes:

• No scheduled run had ever pinged Healthchecks. gdl-sync@.service ran gdl-cron.sh %i directly (written 08-20), but HC_URL is exported by the run-*.sh wrappers (added 09-14), which the unit bypassed. gdl-cron.sh skips the ping silently when HC_URL is empty. Fixed: the unit now runs run-%i.sh. (run-full-sweep.sh has no check URL, so that mode still does not ping.)
• An open Chrome tab keeps showing its logged-in UI after the session is dead — the SPA only notices on its next failed request. Looking at the window proves nothing; reload the tab, or read the cookie DB. The cookie jar had csrftoken, ds_user_id, mid… but no sessionid.

Guards now in gdl-cron.sh (tests: scripts/test_session_guards.py):

exit	from	meaning
76	check-session.py, before any request	no unexpired sessionid in the Chrome profile gdl reads. The run aborts, logs why, pings /76 (red check). Same DB, same profile as gdl, so it sees exactly what gdl would. Prints a WARNING (log only, no alert) when under 7 days of session remain
77	stories-streak.py, after a stories run	every fetched stories source returned "No results" on 3 consecutive days (GDL_EMPTY_STREAK_FAIL to change). Catches a session Instagram stopped honouring while the cookie is still present, which the preflight cannot see. One day per calendar date, so manual re-runs do not inflate it; state in ~/gdl/stories-empty-streak
75	existing	every stories source skipped by --min-interval

To recover from exit 76 the user must log in to Instagram in the Chrome on mattellite (VNC, display :1; neither password nor 2FA is something to script). Then ~/gdl/check-session.py should say session ok ... expires <date>, and a manual ~/gdl/run-stories.sh should find stories again.

Verbose output. A healthy run looked hung: pacing is 12–20 s, the config sets output: null (no per-file lines), and the live-listing probe captured output until it finished. gdl-sync.py now prints [n/N] user / kind (time), a ... fetching: 4m00s elapsed, 12 new file(s) so far (latest: …) heartbeat every 60 s, and done in 6m12s, 14 new file(s). It only counts files already written; no extra requests, pacing unchanged.

Archive resolution check (read-only scan of header dimensions for the six scraped profiles, plus 8 single-post live metadata fetches, paced):

• Logged-out gdl does not download smaller files — it fails with a login redirect, so no logged-out media ever entered the archive.
• Archived image sizes match what a logged-in fetch offers (1440×1800, 3277×4096, 2976×3720, and the genuinely small ones such as 400×400 or 742×928 sitting next to 1440 px siblings in one carousel). No image-upgrade pass is needed.
• Videos: ~1,200 of ~1,300 archived videos are 720p. A 2026 reel offers a 1080×1920 VP9 stream (video-only DASH, plus separate AAC); a 2024 post tops out at 1276×720, same as the archive. A video upgrade would mostly benefit 2025–26 posts (~700), at one request each.
• iOS 18.7 Safari plays VP9 + Opus .webm but not VP9 + AAC .mp4 (error code 4; canPlayType wrongly says "probably"). A video upgrade must therefore output .webm — video stream copied, audio re-encoded to Opus 96k. Not built yet. Open points before building: the viewer mints every video blob as video/mp4 (src/lib/archive-cache.ts, src/hooks/useArchiveScanner.ts) and must derive the MIME type from the extension; superseded 720p files should be kept for review in a directory outside the Resilio-synced, viewer-scanned archives/ tree; and the request budget (suggested: ≤40 posts/day, behind the preflight).

Account swapped (same day). The scraping account is now erg05teur (also signed in on the owner's phone). It is a fresh account, so the "replacement account" rules in docs/gallery-dl.md apply — and were not fully followed on day one: the resolution check and an unplanned catch-up run (below) spent roughly 90 requests within hours of the login. The owner chose to leave the scheduled timers as they are rather than pause them; the preflight (exit 76) and the all-empty-stories streak (exit 77) are the safety net. Worth watching: the weekly profiles timer's first fire (~2026-10-23) is a much larger spend than the daily stories runs.

Host brought in line with the repo (same day). Every tracked script and unit on mattellite is now byte-identical to scripts/, including the weekly profiles timer (it had been left monthly since 08-22). Installing it fired an unplanned catch-up profiles run (see "Scheduling"); that run honoured the 20 h floor (20 of 24 sources skipped) and fetched the four reels sources, ~15 min each at the usual pacing.

2026-09-14 — a tracked account renamed on Instagram: 0ct0ber19 → heejin

HeeJin's handle changed and the old one is now fully dead — instagram.com/ 0ct0ber19/ returns "Sorry, this page isn't available", no redirect. Verified @heejin is her (verified badge, ARTMS affiliation, followed by official_artms, same profile picture) before touching anything.

Directory naming is forced from our own tracked profile name, not derived from Instagram's live username (see gdl_command() in gdl-sync.py) — so updating the URL alone would have started a brand-new heejin/ directory tree alongside the existing 0ct0ber19/ one, splitting one person into two profiles. Fixed by migrating the existing archive instead of leaving it stale:

• 0ct0ber19 → heejin, 0ct0ber19 - reels → heejin - reels, story - 0ct0ber19 → story - heejin, and both her highlight directories — 3,353 files total, done directly on the NAS (ana-nas.lan.1qaz.ca, /volume1/rslsync/sync/Instagram-archive/archives), not via gdl-sync.py.
• Only files whose embedded {username} token was exactly 0ct0ber19 were renamed — matched as _0ct0ber19 - (dated stem) or ^0ct0ber19 - (the older undated highlight naming) or the bare 0ct0ber19.<ext> profile picture, never a substring match. 7 files inside her own directories were correctly left untouched: real collab posts (chuuo3o, findyourmapp) physically stored there but truly owned by someone else — see "Where a shared reel lands" above for why that happens at all.
• Verified by content after each rename(), not exit code — this volume can report a rename as failed when it actually succeeded (see instaarchive-deployment in project memory). Every directory came back with zero missing targets and zero leftover old names.
• docs/artms-instagram-accounts.txt and ~/gdl/artms_account_links.txt updated to the new URL. The live site picked up the rename immediately, no cache-bust needed — the archive index invalidates on directory mtime, which a rename changes.
• Not changed: the skip-archive DB (artms.db) keys purely on instagram_<media_id>, no username involved, so it needed nothing. The per-source state.json timestamps are keyed by URL, though, so heejin's entries look "never seeded" to gdl-sync.py now — the next run against her will do one seeding listing pass to catch up, same as any newly added profile. Cheap, and expected.

If this happens again for another tracked account: confirm the new handle is really the same person before touching anything (verified badge + a mutual signal like "Followed by official_artms" + matching profile picture is usually enough), then this section is the template.

Safety net: a global gallery-dl config

~/.config/gallery-dl/config.json on mattellite exists so that a plain gallery-dl <url> typed by hand — for a quick manual check, outside gdl-sync.py entirely — still gets the hand-paced caution settings instead of gallery-dl's own faster defaults. It is loaded automatically; nothing needs to reference it. gdl-sync.py's own --sleep-request/--sleep/--rate flags still override it as normal — this is only a floor for when nobody passed any.

{
    "extractor": {
        "instagram": {
            "api": "rest",
            "cookies": ["chrome", "/home/matt/.config/google-chrome-devtools"],
            "sleep-request": [12.0, 20.0],
            "sleep": [5.0, 10.0],
            "sleep-429": 120.0,
            "retries": 8,
            "videos": true
        }
    },
    "downloader": {
        "http": {
            "rate": "500K",
            "retries": 8
        }
    }
}

Every value here mirrors gdl-cron.sh's own hand-paced defaults (see its SLEEP_REQUEST/SLEEP/RATE comments) and build_config() in gdl-sync.py — api: rest matters most: the graphql backend issues one request PER POST for every video and carousel, the pattern that got this account banned once already. cookies here is the config-file equivalent of --cookies-from-browser, so a bare gallery-dl <url> is already authenticated as the archive account, not anonymous.

Verified with gallery_dl.config.load() + config.get(...) (zero live requests) — every value above loads correctly with no --config flag passed.

This file is not tracked in the repo — like artms.db and the state files, it is host-local runtime config, and it embeds the same real Chrome-profile path already documented above. Recreate it by hand (or from this section) after a fresh mattellite setup.

Healthchecks.io

run-stories.sh and run-profiles.sh each ping a healthchecks.io check — instagram-archiver-stories / instagram-archiver-posts — via HC_URL, plumbed through to gdl-cron.sh. A start ping fires before the sync, a finish ping fires after with the run's own log attached as the body (visible in the healthchecks.io dashboard, so a failure is diagnosable without SSHing in). The trailing URL segment is the exit code: 0 means success, anything else fails the check — including the stories: 0 to sync → exit 75 remap above, so a fully-skipped stories run alerts, not silently passes.

Unset HC_URL (e.g. HC_URL= ./run-stories.sh) to run without pinging. run-full-sweep.sh and the reels scripts have no check configured yet.

The systemd unit must run the wrappers, not gdl-cron.sh directly — HC_URL lives in run-<mode>.sh, and gdl-cron.sh skips the ping silently when it is empty. Until 2026-10-14 the unit bypassed the wrappers, so no scheduled run had ever pinged. The preflight (exit 76) and the all-empty streak (exit 77) are described in "2026-10-14" above; both reach the dashboard as failures through the same trailing-status-code convention.

This is also why gdl-sync.py's publish() no longer treats the routine chown-permission warning as a failure (2026-09-14). The archive share is never owned by the ssh user, so rsync's own chown step fails on every run — exit 23, "some files/attrs were not transferred" — even though every file lands; "Reading the output" above has documented this as cosmetic since this file was created. Before this fix the exit code didn't know that, so gdl-sync.py returned 1 on every clean run too — harmless when a human was reading the log anyway, but it would have made every healthcheck ping report a normal run as a failure. Only that specific permission-warning shape is masked; a real transfer failure sharing the same exit code (23) still fails loudly, verified with both a real captured rsync failure and unit tests.

Remotes

remote	what goes there
origin → gitea	everything: main, tooling, tags, backups
github	main and the current release tag only — it exists to run the CI/CD image build

The other 22 release tags stay on gitea. Pushing them all to GitHub triggers one container build per tag, because each tag carries its own workflow file.

2026-10-22 — gallery-dl auto-upgrade; posts listing went empty

Scheduled and manual runs were failing quietly from early October: after the session was restored, posts listings for heejin/kimxxlip returned No results in ~3 s (a caught-up profile skips; it does not say that), while stories worked and the posts opened fine in the browser. The host was on gallery-dl 1.32.14; 1.32.17 (2026-10-17) lists "instagram: update user & posts query parameters". After upgrading, the profile lookup failed instead (KeyError: 'data', Requested user could not be found). That is upstream gh#9829 -- Instagram changed the query again on ~10-21; a fix is merged upstream but not in a release as of 10-22. Single-post fetches (/p/<code>/) still work, so the gap is the listing, which the browser covers -- see "Posts grid by browser" below.

scripts/gdl-upgrade.sh (pipx upgrade --include-injected gallery-dl, so yt-dlp too) now runs:

• at the start of every gdl-cron.sh run, after the session preflight and before any request; logs upgrade: up to date or upgrade: CHANGED a -> b, and never fails the run;
• weekly from gdl-upgrade.timer (systemd user timer, installed on mattellite with the others -- systemd rather than cron, to match the rest of the host).

It costs no Instagram requests. The trade-off: a bad upstream release is picked up automatically; roll back with pipx install --force gallery-dl==<old> and re-inject yt-dlp.

Posts grid by browser: posts-sync.sh

The reels workaround from August, generalised. reels-scrape.py is now instagram-chrome-scrape.py with --tab reels|posts, and reels-sync.sh's body is now chrome-sync.sh (GDL_TAB=reels|posts); reels-sync.sh and the new posts-sync.sh are two-line wrappers that set the tab, so every older command still works.

ssh mattellite
~/gdl/posts-sync.sh heejin          # scroll /heejin/, fetch new /p/ links as single posts

--tab posts scrolls the main grid (/<user>/), collects /p/ and /reel/ links, and dedupes against the whole archive like the reels tab does. Unlike reels it stops early: the grid is newest-first, so after 2 scroll rounds in a row that turned up nothing unarchived (--max-known-rounds, round 1 exempt because pinned posts sit on top) there is nothing newer to find. First run, heejin: 3 scrolls, 36 posts seen, 30 already held, 6 new.

Same caveats as reels: it drives the signed-in Chrome on :9222, so it is by hand, not on a timer. Once a gallery-dl release fixes the listing, run-profiles.sh covers posts again and this goes back to being a fallback.

Deploy scripts atomically, never in place. bash reads a script as it runs, so overwriting gdl-cron.sh under a running sync makes bash resume at the same byte offset in the new text. On 10-22 an scp over it mid-run made the wrapper die after the sync with line 163: status: unbound variable. The fetch and publish had finished, but the exit line and the Healthchecks ping were skipped. Copy to a temp name and mv it into place (a new inode; the running bash keeps the old one), or check pgrep -af gdl- first.

A --min-interval trap: a run that returns No results still counts as a fetch, so the next run skips the source for 20 h. Re-run after a bad result with --min-interval 0 (not --force, which also drops the probe cache).

Guards — recreate these after a fresh clone

Neither guard is versioned, so a new clone has no protection at all:

git config remote.github.push refs/heads/main:refs/heads/main

cat > .git/hooks/pre-push <<'HOOK'
#!/bin/sh
remote_url="$2"
case "$remote_url" in *github.com*) ;; *) exit 0 ;; esac
while read -r _ _ remote_ref _; do
  [ -z "$remote_ref" ] && continue
  case "$remote_ref" in
    refs/heads/main|refs/tags/*) ;;
    *) echo "pre-push: refusing to push '$remote_ref' to GitHub." >&2; exit 1 ;;
  esac
done
exit 0
HOOK
chmod +x .git/hooks/pre-push

Decide on the remote ref, not the local one: a delete push sends (delete) as the local ref, and an earlier version of this hook rejected every deletion because of it.

What lives here

path	what it is
scripts/gdl-sync.py	the gallery-dl fetcher; replaced JD2 for the ARTMS profiles
scripts/gdl-upgrade.sh	pipx upgrade gallery-dl + yt-dlp; run by gdl-cron.sh and gdl-upgrade.timer
scripts/instagram-chrome-scrape.py	list a profile's reels tab or posts grid by scrolling the signed-in Chrome over CDP
scripts/chrome-sync.sh	scrape + fetch new posts for one profile; reels-sync.sh / posts-sync.sh set the tab
scripts/gdl-cron.sh	unattended wrapper: stories | profiles | full-sweep
scripts/systemd/	the timers actually installed on the fetch host
scripts/test_gdl_sync.py	its tests
scripts/check-session.py	preflight: is there a usable Instagram sessionid in the Chrome cookie DB? (exit 1 = no)
scripts/stories-streak.py	postflight: all stories sources empty for several days running?
scripts/test_session_guards.py	tests for the two guards above (offline)
scripts/jd2-sync.ts	JDownloader .crawljob generator, still used elsewhere
docs/gallery-dl.md	the measurements behind every option in the fetcher — read before changing pacing
docs/jdownloader.md	the older JD2 flow
docs/artms-instagram-accounts.txt	the profile list passed to --urls-file

Commands

# fetch: always --dry-run first; it prints the plan and the publish step
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \
    --staging <dir> --publish <user>@<nas>:<archives> \
    --archive-db <db> --urls-file artms_account_links.txt --abort 50 --dry-run

# crawljobs (no npm script — package.json is kept identical to main)
npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run

Run the fetcher from the host whose public IP matches the browser the cookie came from. --abort 50 is the routine setting; omit it for a full sweep that also catches edited carousels.

Why there is no CLAUDE.md entry for any of this

CLAUDE.md, README.md and package.json are kept byte-identical to main so that merging main into tooling never conflicts. The earlier attempt put tooling notes in CLAUDE.md and a jd2 script in package.json; because main had deleted those lines, every merge re-applied the deletion. Keep branch-specific documentation in this file, which main does not have.

Running it by hand

Everything happens on mattellite — the fetch host whose public IP matches the browser the cookie came from. Running it anywhere else is what session-hijack detection looks for.

ssh mattellite
~/gdl/gdl-cron.sh profiles    # or: stories | full-sweep

That is the whole thing: it wipes staging, fetches, and publishes straight to the NAS. To drive gdl-sync.py directly instead — always --dry-run first, which prints the plan and the exact rsync that would touch the archive:

cd ~/gdl
PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
    --index    https://instaarchive.ergosteur.com \
    --staging  ~/gdl/staging-manual \
    --publish  agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
    --archive-db ~/gdl/artms.db \
    --urls-file  ~/gdl/artms_account_links.txt \
    --sleep-request 12 20 --sleep 5 10 --rate 500K \
    --abort 50 --dry-run          # swap for --execute when the plan looks right

Do not omit the pacing flags. gdl-sync.py's own defaults are 6-10s / 3-6s / 1M, which is roughly half the caution the wrapper applies. A hand-run that leaves them off is less careful than the automation — which is backwards, and was true of this very example until 2026-08-22. Prefer gdl-cron.sh; it carries them for you.

PATH matters: gallery-dl is a pipx install in ~/.local/bin, which is not on cron's PATH and not on a non-login shell's either.

Useful variations:

--profile 0ct0ber19 --profile kimxxlip   # instead of --urls-file
--only stories                            # one surface
--max-sources 4                           # hard ceiling on what one run touches
--force                                   # ignore --min-interval AND the probe
                                          # cache; almost never what you want

Long runs

A sync runs for minutes to hours at this pacing, so detach it rather than holding an ssh session open:

cd ~/gdl && setsid nohup ./gdl-cron.sh profiles > /dev/null 2>&1 &
tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)

Do not kill it with pkill -f <pattern> over ssh. The pattern matches your own ssh command line, so you kill your own shell and the sync survives — this happened twice on 2026-08-22. Use pkill -x chrome style exact-name matches, or kill the pid: pgrep -f 'only posts,reels' | tail -1.

Nothing reaches the archive until the run finishes: gdl-sync.py publishes once, at the end, so a run killed midway leaves the archive untouched.

The three modes

Renamed on 2026-08-22. full was misleading — it is the abort-limited run — and sweep did not convey that it is the exhaustive one. The wrapper rejects the old names with a pointer rather than a bare error.

mode	cadence	cost	why
stories	daily	~6 requests	stories expire in 24h and cannot be backfilled; this is the only run that loses content if skipped
profiles	weekly, randomized day	~40-60 requests	every surface, --abort 50 — stops enumerating a profile once it reaches content already held. Catches everything new
full-sweep	rarely, by hand	~420 requests	no abort; walks every profile to the end. The only run that notices posts edited after we archived them (test case 15)

The skip-archive means an infrequent profiles run costs barely more than a frequent one — it only fetches what is new. Frequency buys freshness, not completeness, except for stories.

Pacing is deliberately slower than gdl-sync.py's own defaults. All three modes run at --sleep-request 12 20 --sleep 5 10 --rate 500K, against defaults of 6-10 / 3-6 / 1M. These are the values the 2026-08-22 runs used by hand after the scraping warning, and they produced 0 400s and 0 429s. An archive sync has no deadline: being slow is free, being restricted is not. Override with GDL_SLEEP_REQUEST, GDL_SLEEP, GDL_RATE — to raise them, not lower them.

stories also runs at --min-interval 8 rather than the 20h default, because at 20h the daily timer silently did nothing whenever a manual run had happened the previous afternoon. The floor exists to stop an aborted restart re-enumerating profiles, which is a minutes-to-hours concern; a stories fetch is one request per profile, so 8h permits at worst about twelve requests in a day instead of six. A stories run that skips every source now exits 75 and prints a warning rather than reporting success.

Scheduling — installed on mattellite

systemd user timers, running as matt, with lingering enabled so they fire without a login session:

loginctl show-user matt --property=Linger      # Linger=yes
systemctl --user list-timers 'gdl-sync@*'

unit	schedule	next fire (as installed)
gdl-sync@stories.timer	daily 09:00	09:36:45 — the delay is the randomisation working
gdl-sync@profiles.timer	weekly (OnCalendar=weekly) + RandomizedDelaySec=6d — a random day each week, not pinned	2026-10-24 06:44 (first weekly fire after the 10-14 install)
gdl-sync@full-sweep.timer	7th of Jan/Apr/Jul/Oct, 04:00	04:42:39

Unit files are version-controlled in scripts/systemd/ and installed to ~/.config/systemd/user/. One templated service, gdl-sync@.service, takes the mode as its instance name and runs run-%i.sh (the fixed-mode wrappers around gdl-cron.sh — they carry the Healthchecks URL; see "Healthchecks.io"). Before 2026-10-14 it ran gdl-cron.sh %i directly, which skipped every ping. The profiles timer was monthly (3rd, 04:00) until 2026-10-14, when the repo's weekly/random-day unit (committed 09-14 but never installed) was put on the host. Timer installs: copy the unit to ~/.config/systemd/user/, systemctl --user daemon-reload, restart the timer. Persistent=true makes that restart fire a catch-up run immediately if a scheduled elapse was missed (it did, 2026-10-14: a real profiles run started the moment the timer was restarted). Touch ~/.local/share/systemd/timers/stamp-gdl-sync@<mode>.timer first if a run has just happened by hand and you do not want one.

Three settings are load-bearing:

• RandomizedDelaySec=45m — a job firing at exactly 09:00 daily is obviously a machine, and the entire safety model is about not looking like one. This is why the table above shows 09:36 rather than 09:00.
• Persistent=true — catch up a run missed because the host was off. cron silently skips, and a skipped stories run is content gone for good.
• TimeoutStartSec=infinity — a sweep can run for hours at this pacing. The default 90s would kill it mid-fetch.

Operating them:

export XDG_RUNTIME_DIR=/run/user/$(id -u)      # needed over non-interactive ssh
systemctl --user start   gdl-sync@stories.service   # run one now
systemctl --user status  gdl-sync@profiles.timer
journalctl --user -u 'gdl-sync@*' -n 50
systemctl --user disable --now gdl-sync@full-sweep.timer   # stop one

systemctl --user fails with "Failed to connect to bus" over ssh unless XDG_RUNTIME_DIR is set. Note also that month names are not valid in OnCalendar's date field — Jan,Apr,Jul,Oct-07 is rejected outright, hence *-01,04,07,10-07. Check any change with systemd-analyze calendar '<expr>' before installing it.

cron, if you ever prefer it

17 9  *  *  *        sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
43 4  3  *  *        sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh profiles
11 4  7  1,4,7,10 *  sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full-sweep

cron runs /bin/sh, so $RANDOM does not exist — hence shuf. And % in a crontab line means newline unless escaped, so avoid it entirely. cron has no equivalent of Persistent=true.

Verifying a run

The first unattended run is 2026-08-21, around 09:36 (09:00 plus the randomised delay). Nothing below costs an Instagram request — every check is against the journal, local logs, or our own viewer's API.

# 1. did it run, and did it exit 0?
ssh mattellite
export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user list-timers 'gdl-sync@*'          # LAST/PASSED columns
journalctl --user -u 'gdl-sync@stories.service' --since yesterday --no-pager

# 2. what did it actually fetch? (sidecars vastly outnumber media -- count media)
ls -1t ~/gdl/logs | head -3
grep -E '^(==>|    FAILED|done;)' ~/gdl/logs/stories-*.log | tail -20
grep -ic 429 ~/gdl/logs/stories-*.log            # MUST be 0 -- see below

A 429 from scontent-*.cdninstagram.com ends the session, it is not a pacing knob to tune. The warning order last time was CDN 429 → 400 on the highlights endpoint → suspension. If a run logs one, disable the timers and stop for the day:

systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer

Then confirm the archive actually grew, from the workstation:

# 3. did the publish land? compare against yesterday's counts
for u in 0ct0ber19 kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
  n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
  printf '%-18s %s\n' "$u" "$n"
done

Counts after the 2026-08-22 runs, to diff against:

profile	files
0ct0ber19	3160
official_artms	6760
cher_ryppo	3056
kimxxlip	3021
zindoriyam	2256
withaseul	1760

A stories-only run adds few files and often none — profiles frequently have no active story. "0 new" is a normal result, not a failure. fileCount in /api/archives is stale by design; use the per-profile /files listing.

2026-08-21 — scraping warning, automation stopped

Instagram flagged the account. Not suspended: an interstitial at /accounts/scraping_warning/ reading "We suspect automated behaviour on your account". It was dismissed in the browser and the account is healthy — feed loads, still signed in. All three timers are disabled. Do not re-enable them without deciding the cadence question below.

How it unfolded, because each step misled in a different way:

1. 09:13 the daily timer fired, exited 0 in one second, logged done; 0 step(s) failed — and fetched nothing. Yesterday's manual run was 18.9–19.2h earlier, just under the --min-interval 20 floor, so all six sources were skipped. A silent no-op on the one surface that cannot be backfilled, reported as success.
2. 20:28 chrome-devtools.service was OOM-killed (5.1 GB peak, ~1w3d CPU). Unrelated to the above, and it does not break fetching — gallery-dl reads the cookie file, not a live browser — but it meant no browser was running to notice anything was wrong.
3. 23:19 a manual recovery run passed the floor (33h) and every source failed with 400 Bad Request on /api/v1/feed/reels_media/?reel_ids=…. Six identical failures across six profiles is not a per-profile fault.
4. Cookies were exported and checked before assuming a block: sessionid 77 chars, printable, colon-delimited, 360 days to expiry; ds_user_id present. Decryption was fine, so the fault was server-side. This check costs no Instagram requests and should always come first.
5. The browser then showed the interstitial. The 400s were the challenge state, not a ban.

What has to change before automation is re-enabled

• The --min-interval floor silently defeats the daily job. Any manual run in the preceding 20h makes the scheduled one a no-op. The floor exists to stop an aborted restart re-enumerating profiles — a minutes-to-hours concern — and a stories fetch is one request per profile. stories should use something like --min-interval 8, not 20.
• A skipped stories run must be loud. 0 to sync, 6 skipped currently exits 0 and looks identical to success. On this surface a skip is a real loss, and it should be visible in the journal without reading the log.
• Reconsider the daily cadence itself. A job hitting story endpoints for six profiles every morning is the most machine-like thing here, randomised delay or not, and it is what was flagged. Every-few-days, or on-demand, may be the honest answer even though stories will be missed.
• chrome-devtools.service has Restart=no and died silently for three hours. It needs Restart=on-failure and probably a MemoryMax=, or it will be dead the next time the cookie needs refreshing.

2026-08-22 — caught up by hand, cleanly

Both surfaces were fetched manually the next day, on the owner's call, at roughly double the configured caution: --sleep-request 12 20, --sleep 5 10, --rate 500K, versus the defaults of 6-10 / 3-6 / 1M.

run	result
stories, 6 profiles	16 media, +21 files, 0 400s, 0 429s
posts+reels, 12 sources, --abort 50	26 media, +64 files, 0 400s, 0 429s

So the 400s really were the challenge state and nothing more: once the interstitial was dismissed in the browser, the same endpoints served normally. The stories that looked lost — official_artms, 0ct0ber19, cher_ryppo, kimxxlip, zindoriyam — were all captured before expiry.

This does not retire the warning. Two hand-paced runs a day later are not evidence that the previous cadence was safe; they are evidence that the account still works. What actually changed the request cost is --abort 50: twelve sources across six profiles, official_artms included at 1829 posts and 781 reels, finished in minutes for a few dozen requests where the old behaviour would have spent ~400.

Note the gap this leaves: the scheduled profiles mode still uses the default 6-10s pacing, not the 12-20s used here. Reconcile that before re-enabling the timers, or the automation will be less careful than the hand runs that followed a warning.

When a run fails

Worked through on 2026-08-22. Do these in order — the first two cost no Instagram requests, and the third costs one page view.

1. Read the error, not the exit code

L=~/gdl/logs/$(ls -1t ~/gdl/logs | head -1); grep -vE '^(profiles|surfaces|sources|pacing|staging|publish|read )' "$L" | head -40

Two results are not failures, despite how they look:

looks like	actually
rsync error: some files/attrs were not transferred (code 23) and done; 1 step(s) failed	the chown to rslsync failing because the ssh user is not root. Data landed. Confirm by re-running the rsync with --dry-run: an empty file list means everything arrived
a source reporting No results	that profile simply has no active story / no highlights

2. Check the cookies before concluding you are blocked

First, ~/gdl/check-session.py — it reads the same cookie DB gdl does and says whether a sessionid exists and when it expires (never prints values). Do not trust the open browser tab to tell you: it keeps showing a logged-in page after the session has died; reload it.

Then, free, and it separates a local fault from a server-side one:

PATH=$HOME/.local/bin:$PATH gallery-dl \
  --cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools" \
  --cookies-export /tmp/ck.txt
grep instagram /tmp/ck.txt | awk '{print $6, length($7)}'   # names + value lengths
rm -f /tmp/ck.txt

A healthy sessionid is ~77 chars, printable ASCII, colon-delimited, and unexpired; ds_user_id should be an 11-digit number. Garbage or non-printable values mean Chrome's cookie decryption failed locally — not that the account is in trouble. Never print the values into a transcript or a bug report.

3. Look at the account in the browser

The session lives in a dedicated Chrome on mattellite, on display :1 with CDP on 9222, run by a systemd unit:

export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user status chrome-devtools.service
systemctl --user start  chrome-devtools.service    # it has Restart=no

Then VNC to :1 and open instagram.com. What you are looking for:

• /accounts/scraping_warning/ — "We suspect automated behaviour on your account". This is what 400 Bad Request on /api/v1/feed/reels_media/ actually means; the session is in a challenge state, not banned. Dismissing it in the browser restores API access immediately, verified 2026-08-22.
• a checkpoint or login page — the session is gone; re-log in the browser.
• a normal feed — the fault is elsewhere.

Do not dismiss a warning and immediately resume. The dismissal fixes the symptom. The behaviour that caused it is the thing to change.

4. Stop, if it was a 429 or a repeated 400

systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer

The documented escalation is CDN 429 → 400 on a stories/highlights endpoint → suspension. It has now run twice, and both times the 400 was the last warning before something worse.

Reels: the API is blocked, scrape by scrolling instead

Fixed upstream 2026-09-14 — read this note before the rest of the section. gallery-dl 1.32.12 (released 2026-09-12; we were on 1.32.9) resolved it: its changelog cites "posts & reels extractors... content retrieval failures", and upgrading (pipx upgrade gallery-dl) confirmed it -- a full run-profiles.sh across all six profiles came back with zero HTTP redirect errors on posts or reels, first time since 2026-08-26. reels-scrape.py (now instagram-chrome-scrape.py)/reels-sync.sh/ run-reels-all.sh are no longer needed for routine use; left in place as a fallback (and because the whole-archive dedup fix they picked up along the way, see below, is independently useful). If this regresses, check the gallery-dl changelog for a newer release before re-reaching for the scroll workaround -- it may already be fixed there too.

The rest of this section is kept as-is: the diagnosis, the workaround, and what it taught along the way are all still accurate history, just no longer the operative fix.

As of 2026-08-26/27, gallery-dl's dedicated reels extractor failed on every profile with HTTP redirect to home page — confirmed hours apart, with a freshly-warmed session and a correct X-IG-WWW-Claim header (that was the first suspect; ruled out by tracing the raw HTTP exchange). It was not a scraping-warning interstitial — the reels tab loaded completely normally in a real, already-signed-in browser — so this was Meta blocking the specific /api/v1/clips/user/ endpoint gallery-dl calls, not an account-health issue. Posts, highlights and stories were unaffected at the time; only reels broke (posts later broke too, some time before 2026-09-13, in the same way — same fix resolved both).

scripts/reels-scrape.py (renamed instagram-chrome-scrape.py on 2026-10-22, when it gained --tab posts) works around it by never calling that endpoint: it drives the same signed-in Chrome via its loopback CDP port (:9222, already exposed for MCP automation), scrolls the reels tab like a person would, and scrapes /reel/<code>/ links out of the rendered page. It only finds shortcodes — nothing is downloaded until the fetch step — and dedupes against the whole archive, not just the profile being scraped (see why below), so re-running it costs nothing for reels already held anywhere.

The easy way: reels-sync.sh

ssh mattellite
~/gdl/reels-sync.sh zindoriyam
# or a full URL: ~/gdl/reels-sync.sh https://www.instagram.com/zindoriyam/

Runs both steps (scrape, then fetch + publish whatever's new) with the same hand-paced settings gdl-cron.sh uses, logs to ~/gdl/logs/reels-<profile>-*, and exits 0 with "no new reels" printed when a profile is already caught up — gdl-sync.py never even gets invoked in that case. Override pacing the same way as gdl-cron.sh (GDL_SLEEP_REQUEST, GDL_SLEEP, GDL_RATE), plus GDL_SCROLL_PAUSE and GDL_MAX_IDLE_ROUNDS for the scrape step. Staging (~/gdl/staging-reels-<profile>) and the scraped URL list (~/gdl/<profile>-reels.txt) are wiped at the START of the next run, not after — left behind for inspection, same as gdl-cron.sh's staging-*.

Verified end to end against zindoriyam on 2026-08-27: 26 reels found on the page, 10 new ones fetched and published cleanly on the first run.

Not wired into gdl-cron.sh or the timers. Both scripts drive your actual browser session rather than firing a background request, and are slower by design (real scrolling, not an API call) — both good reasons not to run this unattended without deciding that deliberately. Today it is a per-profile, by-hand tool only.

For all six ARTMS profiles in one go, ~/gdl/run-reels-all.sh just calls reels-sync.sh once per profile in artms_account_links.txt, same reasoning as above — real scrolling, so still by hand. Run 2026-09-03: official_artms alone has 609 real reels and took ~14 minutes of scrolling; the whole six-profile run took about 15 minutes total, 2 new reels found.

Where a shared reel lands

A reel found on one profile's /reels/ page is not necessarily owned by that profile — reposts and collabs between tracked accounts show up there too. It always gets filed under its true owner, per Instagram's own metadata on the post, never under whichever profile's page you happened to scrape it from — gdl-sync.py's directory template for a scraped reel is {username} filled in from that metadata, the same mechanism highlights already used for their own directory. So:

• Scrape order doesn't matter. Run reels-sync.sh on 0ct0ber19 or zindoriyam first, whichever — a reel they share lands in the same place either way, and running it on the other one afterward just sees that shortcode as already archived (dedup checks every profile, not only the one being scraped) and skips it.
• It never gets duplicated into both accounts' directories, and it never gets misattributed to the profile you scraped instead of who actually posted it.

This is also literally why the whole-archive dedup fix above exists: a zindoriyam-scraped shortcode that turned out to belong to 0ct0ber19 was filed under 0ct0ber19/, not zindoriyam/ — dedup that only checked zindoriyam's own directory would never have found it there.

What it's doing, if you want to run the two steps separately

PROFILE=someuser

# 1. Scrape by scrolling; dedupe against the archive; write new URLs to a file.
#    Needs gallery-dl's own pipx venv python -- that's where websocket-client
#    (the one extra dependency this needs) got injected.
~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/instagram-chrome-scrape.py \
  --profile "$PROFILE" \
  --index https://instaarchive.ergosteur.com \
  --out ~/gdl/"$PROFILE"-reels.txt

# 2. Fetch whatever's new -- dry run first, same as any other gdl-sync.py call.
cd ~/gdl && PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
  --publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
  --staging ~/gdl/staging-reels-scraped \
  --sleep-request 12 20 --sleep 5 10 --rate 500K \
  --post-urls-file ~/gdl/"$PROFILE"-reels.txt --dry-run
# ...then swap --dry-run for --execute once the plan looks right.

Notes:

• instagram-chrome-scrape.py fails fast if Chrome/CDP isn't up (curl -s http://localhost:9222/json/version to check first).
• If it finds nothing new, gdl-sync.py --post-urls-file refuses to run ("no usable URLs") rather than doing nothing quietly — expected when a profile is already caught up.
• --max-idle-rounds (default 3) and --scroll-pause MIN MAX (default 2.0 3.5) are worth raising for an unusually large or slow-loading reels tab; the default stops once 3 consecutive scrolls find nothing new.

Why dedup checks the whole archive, not just the scraped profile

First version deduped only against the scraped profile's own directories. Re-running it on zindoriyam minutes after a successful fetch found "9 new" reels again — all reposts/collabs originally by 0ct0ber19 and official_artms (also tracked profiles). The old, broken direct-reels-tab fetch had left orphaned sidecar-only remnants (.json/.txt, no media) for them, misfiled under zindoriyam - reels/ with the true owner's name baked into the filename stem — index_existing() correctly does not count a sidecar-only entry as "held", so they looked new. gdl-sync.py re-fetched them, filed them correctly under {username} (the post's true owner, per its own metadata) — where they already existed from that profile's own regular sync — and rsync --ignore-existing silently skipped every one, so nothing was lost or duplicated. But 9 Instagram requests were spent finding that out. Since a shortcode is globally unique, the scraper now checks every archived profile's listing, not just the one being scraped — all local requests to the viewer's own API, never to instagram.com, so checking all 16 profiles costs nothing on the budget that actually matters. Confirmed fixed the same day: re-running against zindoriyam immediately afterward found "26 already archived, 0 new" and exited clean.

What changed on 2026-08-20

One session, three separate pieces of work. Recorded because the reasons are not recoverable from the diffs.

The sync run. First incremental fetch in four days: 184 new media, 299 files published, 0 failures, 0 CDN 429s. 20 story items, which are the part that could not have been recovered later. Cost about half what it would have, because the archive DB was already seeded and the state file was primed by hand so no probe passes ran.

--abort 50. The skip-archive suppresses downloads, which spends the CDN; it does nothing about the listing pass, which spends instagram.com and scales with how big a profile is rather than how much is new. Measured from sidecar write times mid-run: 3 new posts took ~100s each, the other 2272 were written in one second. Enumerating cher_ryppo fell from 2151 posts to 7.

The repo split. main is public and now carries none of the fetching tooling, no host details, and no real account names — its entire history was rewritten, the GitHub repo deleted and recreated to clear force-push residue, and 22 container images pruned from ghcr because the server bundle had been shipping source comments naming real accounts. This branch holds everything that was removed. See the caution at the top.

Automation. mattellite got a key on the NAS, closing the last manual step, and three systemd timers now run the sync unattended.

Outstanding

State as of 2026-08-20, after the sync run and the repo split. Nothing here is broken; these are decisions not yet made and cleanups not yet done.

Update 2026-10-14: the "Fetching" notes below predate the timers being enabled — all three gdl-sync@* timers are installed and enabled on mattellite (see "Scheduling" and "2026-10-14"). Newer open work: the video-upgrade pass (docs/video-upgrade-plan.md — measured and decided, not built), and watching the fresh scraping account through its first weekly profiles run (~2026-10-23).

Fetching

• The timers are DISABLED after the 2026-08-21 scraping warning; see that section. Their one unattended firing did the wrong thing — it skipped every source on the 20h floor and reported success — so re-enabling should wait on the four fixes listed there, not just on the account settling.

• mattellite's ~/.ssh/id_ed25519.pub is in the NAS's authorized_keys for agentapi (added 2026-08-20, alongside the workstation's existing key), so the fetch host publishes straight to the archive and no sshpass step is needed. That key is what makes the timers work — remove it and every scheduled run will fetch successfully and then fail at publish.

• 5.4 GB of stale staging on mattellite (~/gdl, 44 GB free): staging and out at 2.2 GB each from 2026-08-17, staging-0820 / out-0820 at 446 MB each, plus staging-full (78 MB) and staging-stories (66 MB) from 2026-08-22. All of it was verified published, so all of it is safe to delete. Staging is wiped per-run by gdl-cron.sh, but the out-* publish targets and anything created by a direct gdl-sync.py call are never cleaned up.

• Stories currently depend on someone remembering. The daily timer exists but is disabled, so the one surface that cannot be backfilled has no automation behind it. Every day nobody runs gdl-cron.sh stories is a day of stories gone. That is the central unresolved tension: the cadence that protects stories is also the most machine-like pattern here.

• The scheduled modes have not been reconciled with the pacing used by hand. Done 2026-08-22: all modes now pass --sleep-request 12 20 --sleep 5 10 --rate 500K, stories uses --min-interval 8, and a fully-skipped stories run exits 75 with a warning instead of looking like a success. The timers are still disabled — enabling them is a separate decision about cadence, not about pacing.

• --abort 50 is opt-in. gdl-cron.sh profiles passes it and the manual runs used it; full-sweep deliberately does not. It stops noticing edited carousels (test case 15), which only a full enumeration finds — which is what full-sweep is for. No cadence for it was ever agreed, and at ~420 requests it is the riskiest thing on the schedule; prefer running it by hand.

• The seeded flags in <db>.state.json were hand-written, reconstructed from the 2026-08-17 log rather than derived from the archive DB. They assert "the skip-archive already knows this source". If artms.db is ever rebuilt, moved or lost, clear the state file too — otherwise those sources will never re-seed and a fetch into empty staging re-downloads everything.

• ~/gdl/gdl-sync.py on the fetch host is a copy, not a checkout. It currently matches this branch (96e5694e…), but nothing keeps them in sync; scp it after any change and re-check the hash.

• The 2026-08-20 run is split across two logs — artms-run3.log (12 sources, no abort) and artms-run4.log (12 sources, --abort 50) — because it was stopped midway to pick up the new flag.

Repo and infrastructure

• The pre-rewrite-* branches on gitea hold the unredacted history — real account names, the fetch host's IP, and the tooling, as it was before the rewrite. They are deliberate backups. Decide whether they expire; the pre-push hook does cover them (it allows only main and tags to GitHub).
• The pre-rewrite-full.bundle backup is in a session scratchpad and will be deleted with it. If a durable backup outside gitea is wanted, move it now.
• Only one container image exists. 22 versions were pruned, so rolling back to an older release means checking out its tag from gitea and pushing that tag to GitHub to rebuild it — the old images are gone, not archived.
• CI warns that the Node 20 actions are deprecated. actions/checkout@v4, docker/login-action@v3, docker/metadata-action@v5 and docker/build-push-action@v5 are being forced onto Node 24. They work today; bump when convenient.
• review-fixes on gitea is a stale v1.3.0-era branch, never merged, published only because the whole local repo was pushed. Probably deletable.
• GitHub Actions run history was lost when the repo was recreated. Cosmetic.