instaarchive-viewer/TOOLING.md

# Tooling branch

> [!CAUTION]
> ## This branch must not be pushed to GitHub
>
> `tooling` is the only branch that still contains the archive-fetching
> scripts and their docs, and those name things `main` was rewritten to
> remove:
>
> - the fetch host's **public IP** (`docs/gallery-dl.md`)
> - the browser profile the session cookie is read from
> - the NAS archive path
> - the **list of Instagram accounts being archived**
>
> On 2026-08-20 `main`'s entire history was rewritten with `git filter-repo`,
> the GitHub repo was deleted and recreated, and 22 container images were
> pruned from ghcr — all to get exactly this material out of public view.
> **One push of this branch to GitHub undoes all of it.**
>
> A second cleanup would be harder than the first: after a force-push the old
> commits stayed reachable by raw SHA, and only deleting the repository
> outright removed them.

## Quick reference — getting new content onto the site

Timers are **disabled**, so this is manual today. Everything runs on
`mattellite`; it fetches, publishes to the NAS itself, and the site picks it up
with no deploy or restart.

```sh
ssh mattellite

~/gdl/gdl-cron.sh stories     # ~1 min.  DO THIS OFTEN - stories die in 24h
~/gdl/gdl-cron.sh profiles    # ~5 min.  posts, reels, highlights, stories
```

`profiles` covers everything, so it is the one to run if you only run one — but
it is no substitute for `stories`, because a story posted and expired between
two `profiles` runs is simply gone. Both are safe to run back-to-back; the
`--min-interval` floor makes a repeat a no-op rather than a re-fetch.

Then check <https://instaarchive.ergosteur.com> — new posts appear without a
redeploy. The archive index re-reads a directory when its mtime changes.

**Reading the output.** These two look alarming and are fine:

- `rsync error: ... (code 23)` / `done; 1 step(s) failed` — the `chown` to
  `rslsync` failing because the ssh user is not root. Files landed.
- `No results` for a profile — it has no active story right now.

**These mean stop**, and are covered in "When a run fails":

- `429` from `scontent-*.cdninstagram.com`
- `400 Bad Request` on `/api/v1/feed/reels_media/` — the account is behind a
  scraping-warning interstitial; check the browser before anything else.

Long runs: `setsid nohup ~/gdl/gdl-cron.sh profiles >/dev/null 2>&1 &` and
`tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`. Nothing reaches the
archive until a run finishes, so killing one midway is safe.

## Safety net: a global gallery-dl config

`~/.config/gallery-dl/config.json` on `mattellite` exists so that a **plain
`gallery-dl <url>` typed by hand** — for a quick manual check, outside
`gdl-sync.py` entirely — still gets the hand-paced caution settings instead of
gallery-dl's own faster defaults. It is loaded automatically; nothing needs to
reference it. `gdl-sync.py`'s own `--sleep-request`/`--sleep`/`--rate` flags
still override it as normal — this is only a floor for when nobody passed any.

```json
{
    "extractor": {
        "instagram": {
            "api": "rest",
            "cookies": ["chrome", "/home/matt/.config/google-chrome-devtools"],
            "sleep-request": [12.0, 20.0],
            "sleep": [5.0, 10.0],
            "sleep-429": 120.0,
            "retries": 8,
            "videos": true
        }
    },
    "downloader": {
        "http": {
            "rate": "500K",
            "retries": 8
        }
    }
}
```

Every value here mirrors `gdl-cron.sh`'s own hand-paced defaults (see its
`SLEEP_REQUEST`/`SLEEP`/`RATE` comments) and `build_config()` in
`gdl-sync.py` — `api: rest` matters most: the graphql backend issues one
request PER POST for every video and carousel, the pattern that got this
account banned once already. `cookies` here is the config-file equivalent of
`--cookies-from-browser`, so a bare `gallery-dl <url>` is already
authenticated as the archive account, not anonymous.

Verified with `gallery_dl.config.load()` + `config.get(...)` (zero live
requests) — every value above loads correctly with no `--config` flag passed.

This file is **not tracked in the repo** — like `artms.db` and the state
files, it is host-local runtime config, and it embeds the same real
Chrome-profile path already documented above. Recreate it by hand (or from
this section) after a fresh `mattellite` setup.

## Remotes

| remote | what goes there |
|---|---|
| `origin` → gitea | everything: `main`, `tooling`, tags, backups |
| `github` | **`main` and the current release tag only** — it exists to run the CI/CD image build |

The other 22 release tags stay on gitea. Pushing them all to GitHub triggers
one container build per tag, because each tag carries its own workflow file.

## Guards — recreate these after a fresh clone

Neither guard is versioned, so a new clone has **no protection at all**:

```sh
git config remote.github.push refs/heads/main:refs/heads/main

cat > .git/hooks/pre-push <<'HOOK'
#!/bin/sh
remote_url="$2"
case "$remote_url" in *github.com*) ;; *) exit 0 ;; esac
while read -r _ _ remote_ref _; do
  [ -z "$remote_ref" ] && continue
  case "$remote_ref" in
    refs/heads/main|refs/tags/*) ;;
    *) echo "pre-push: refusing to push '$remote_ref' to GitHub." >&2; exit 1 ;;
  esac
done
exit 0
HOOK
chmod +x .git/hooks/pre-push
```

Decide on the **remote** ref, not the local one: a delete push sends
`(delete)` as the local ref, and an earlier version of this hook rejected
every deletion because of it.

## What lives here

| path | what it is |
|---|---|
| `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
| `scripts/systemd/` | the timers actually installed on the fetch host |
| `scripts/test_gdl_sync.py` | its tests |
| `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
| `docs/gallery-dl.md` | the measurements behind every option in the fetcher — **read before changing pacing** |
| `docs/jdownloader.md` | the older JD2 flow |
| `docs/artms-instagram-accounts.txt` | the profile list passed to `--urls-file` |

## Commands

```sh
# fetch: always --dry-run first; it prints the plan and the publish step
./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \
    --staging <dir> --publish <user>@<nas>:<archives> \
    --archive-db <db> --urls-file artms_account_links.txt --abort 50 --dry-run

# crawljobs (no npm script — package.json is kept identical to main)
npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run
```

Run the fetcher from the host whose public IP matches the browser the cookie
came from. `--abort 50` is the routine setting; omit it for a full sweep that
also catches edited carousels.

## Why there is no CLAUDE.md entry for any of this

`CLAUDE.md`, `README.md` and `package.json` are kept **byte-identical** to
`main` so that merging `main` into `tooling` never conflicts. The earlier
attempt put tooling notes in `CLAUDE.md` and a `jd2` script in `package.json`;
because `main` had *deleted* those lines, every merge re-applied the deletion.
Keep branch-specific documentation in this file, which `main` does not have.

## Running it by hand

Everything happens on **`mattellite`** — the fetch host whose public IP matches
the browser the cookie came from. Running it anywhere else is what
session-hijack detection looks for.

```sh
ssh mattellite
~/gdl/gdl-cron.sh profiles    # or: stories | full-sweep
```

That is the whole thing: it wipes staging, fetches, and publishes straight to
the NAS. To drive `gdl-sync.py` directly instead — always `--dry-run` first,
which prints the plan and the exact rsync that would touch the archive:

```sh
cd ~/gdl
PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
    --index    https://instaarchive.ergosteur.com \
    --staging  ~/gdl/staging-manual \
    --publish  agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
    --archive-db ~/gdl/artms.db \
    --urls-file  ~/gdl/artms_account_links.txt \
    --sleep-request 12 20 --sleep 5 10 --rate 500K \
    --abort 50 --dry-run          # swap for --execute when the plan looks right
```

**Do not omit the pacing flags.** `gdl-sync.py`'s own defaults are 6-10s / 3-6s
/ 1M, which is roughly half the caution the wrapper applies. A hand-run that
leaves them off is *less* careful than the automation — which is backwards, and
was true of this very example until 2026-08-22. Prefer `gdl-cron.sh`; it
carries them for you.

`PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not
on cron's PATH and not on a non-login shell's either.

Useful variations:

```sh
--profile 0ct0ber19 --profile kimxxlip   # instead of --urls-file
--only stories                            # one surface
--max-sources 4                           # hard ceiling on what one run touches
--force                                   # ignore --min-interval AND the probe
                                          # cache; almost never what you want
```

### Long runs

A sync runs for minutes to hours at this pacing, so detach it rather than
holding an ssh session open:

```sh
cd ~/gdl && setsid nohup ./gdl-cron.sh profiles > /dev/null 2>&1 &
tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)
```

**Do not kill it with `pkill -f <pattern>` over ssh.** The pattern matches your
own `ssh` command line, so you kill your own shell and the sync survives — this
happened twice on 2026-08-22. Use `pkill -x chrome` style exact-name matches,
or kill the pid: `pgrep -f 'only posts,reels' | tail -1`.

Nothing reaches the archive until the run finishes: `gdl-sync.py` publishes
once, at the end, so a run killed midway leaves the archive untouched.

### The three modes

Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run —
and `sweep` did not convey that it is the exhaustive one. The wrapper rejects
the old names with a pointer rather than a bare error.

| mode | cadence | cost | why |
|---|---|---|---|
| `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped |
| `profiles` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating a profile once it reaches content already held. Catches everything **new** |
| `full-sweep` | rarely, by hand | **~420 requests** | no abort; walks every profile to the end. The only run that notices posts **edited** after we archived them (test case 15) |

The skip-archive means an infrequent `profiles` run costs barely more than a frequent
one — it only fetches what is new. Frequency buys freshness, not completeness,
except for stories.

**Pacing is deliberately slower than `gdl-sync.py`'s own defaults.** All three
modes run at `--sleep-request 12 20 --sleep 5 10 --rate 500K`, against defaults
of 6-10 / 3-6 / 1M. These are the values the 2026-08-22 runs used by hand after
the scraping warning, and they produced 0 400s and 0 429s. An archive sync has
no deadline: being slow is free, being restricted is not. Override with
`GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE` — to raise them, not lower them.

`stories` also runs at `--min-interval 8` rather than the 20h default, because
at 20h the daily timer silently did nothing whenever a manual run had happened
the previous afternoon. The floor exists to stop an *aborted restart*
re-enumerating profiles, which is a minutes-to-hours concern; a stories fetch
is one request per profile, so 8h permits at worst about twelve requests in a
day instead of six. **A stories run that skips every source now exits 75 and
prints a warning** rather than reporting success.

## Scheduling — installed on `mattellite`

systemd **user** timers, running as `matt`, with lingering enabled so they fire
without a login session:

```sh
loginctl show-user matt --property=Linger      # Linger=yes
systemctl --user list-timers 'gdl-sync@*'
```

| unit | schedule | next fire (as installed) |
|---|---|---|
| `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working |
| `gdl-sync@profiles.timer` | 3rd of each month, 04:00 | 04:37:44 |
| `gdl-sync@full-sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |

Unit files are version-controlled in `scripts/systemd/` and installed to
`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
the mode as its instance name and runs `gdl-cron.sh %i`.

Three settings are load-bearing:

- **`RandomizedDelaySec=45m`** — a job firing at exactly 09:00 daily is
  obviously a machine, and the entire safety model is about not looking like
  one. This is why the table above shows 09:36 rather than 09:00.
- **`Persistent=true`** — catch up a run missed because the host was off.
  cron silently skips, and a skipped `stories` run is content gone for good.
- **`TimeoutStartSec=infinity`** — a sweep can run for hours at this pacing.
  The default 90s would kill it mid-fetch.

Operating them:

```sh
export XDG_RUNTIME_DIR=/run/user/$(id -u)      # needed over non-interactive ssh
systemctl --user start   gdl-sync@stories.service   # run one now
systemctl --user status  gdl-sync@profiles.timer
journalctl --user -u 'gdl-sync@*' -n 50
systemctl --user disable --now gdl-sync@full-sweep.timer   # stop one
```

`systemctl --user` fails with "Failed to connect to bus" over ssh unless
`XDG_RUNTIME_DIR` is set. Note also that **month names are not valid in
`OnCalendar`'s date field** — `Jan,Apr,Jul,Oct-07` is rejected outright, hence
`*-01,04,07,10-07`. Check any change with `systemd-analyze calendar '<expr>'`
before installing it.

### cron, if you ever prefer it

```cron
17 9  *  *  *        sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
43 4  3  *  *        sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh profiles
11 4  7  1,4,7,10 *  sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full-sweep
```

cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a
crontab line means newline unless escaped, so avoid it entirely. cron has no
equivalent of `Persistent=true`.

## Verifying a run

The first unattended run is **2026-08-21, around 09:36** (09:00 plus the
randomised delay). Nothing below costs an Instagram request — every check is
against the journal, local logs, or our own viewer's API.

```sh
# 1. did it run, and did it exit 0?
ssh mattellite
export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user list-timers 'gdl-sync@*'          # LAST/PASSED columns
journalctl --user -u 'gdl-sync@stories.service' --since yesterday --no-pager

# 2. what did it actually fetch? (sidecars vastly outnumber media -- count media)
ls -1t ~/gdl/logs | head -3
grep -E '^(==>|    FAILED|done;)' ~/gdl/logs/stories-*.log | tail -20
grep -ic 429 ~/gdl/logs/stories-*.log            # MUST be 0 -- see below
```

**A `429` from `scontent-*.cdninstagram.com` ends the session, it is not a
pacing knob to tune.** The warning order last time was CDN 429 → `400` on the
highlights endpoint → suspension. If a run logs one, disable the timers and
stop for the day:

```sh
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
```

Then confirm the archive actually grew, from the workstation:

```sh
# 3. did the publish land? compare against yesterday's counts
for u in 0ct0ber19 kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
  n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
  printf '%-18s %s\n' "$u" "$n"
done
```

Counts after the 2026-08-22 runs, to diff against:

| profile | files |
|---|---:|
| 0ct0ber19 | 3160 |
| official_artms | 6760 |
| cher_ryppo | 3056 |
| kimxxlip | 3021 |
| zindoriyam | 2256 |
| withaseul | 1760 |

A stories-only run adds few files and often **none** — profiles frequently have
no active story. "0 new" is a normal result, not a failure. `fileCount` in
`/api/archives` is stale by design; use the per-profile `/files` listing.

## 2026-08-21 — scraping warning, automation stopped

**Instagram flagged the account.** Not suspended: an interstitial at
`/accounts/scraping_warning/` reading *"We suspect automated behaviour on your
account"*. It was dismissed in the browser and the account is healthy — feed
loads, still signed in. **All three timers are disabled.** Do not re-enable
them without deciding the cadence question below.

How it unfolded, because each step misled in a different way:

1. **09:13** the daily timer fired, exited 0 in one second, logged
   `done; 0 step(s) failed` — and fetched nothing. Yesterday's manual run was
   18.9–19.2h earlier, just under the `--min-interval 20` floor, so all six
   sources were skipped. **A silent no-op on the one surface that cannot be
   backfilled, reported as success.**
2. **20:28** `chrome-devtools.service` was OOM-killed (5.1 GB peak, ~1w3d CPU).
   Unrelated to the above, and it does not break fetching — gallery-dl reads
   the cookie *file*, not a live browser — but it meant no browser was running
   to notice anything was wrong.
3. **23:19** a manual recovery run passed the floor (33h) and every source
   failed with `400 Bad Request` on
   `/api/v1/feed/reels_media/?reel_ids=…`. Six identical failures across six
   profiles is not a per-profile fault.
4. Cookies were **exported and checked before assuming a block**:
   `sessionid` 77 chars, printable, colon-delimited, 360 days to expiry;
   `ds_user_id` present. Decryption was fine, so the fault was server-side.
   This check costs no Instagram requests and should always come first.
5. The browser then showed the interstitial. The 400s were the challenge
   state, not a ban.

### What has to change before automation is re-enabled

- **The `--min-interval` floor silently defeats the daily job.** Any manual run
  in the preceding 20h makes the scheduled one a no-op. The floor exists to
  stop an *aborted restart* re-enumerating profiles — a minutes-to-hours
  concern — and a stories fetch is one request per profile. `stories` should
  use something like `--min-interval 8`, not 20.
- **A skipped stories run must be loud.** `0 to sync, 6 skipped` currently
  exits 0 and looks identical to success. On this surface a skip is a real
  loss, and it should be visible in the journal without reading the log.
- **Reconsider the daily cadence itself.** A job hitting story endpoints for
  six profiles every morning is the most machine-like thing here, randomised
  delay or not, and it is what was flagged. Every-few-days, or on-demand, may
  be the honest answer even though stories will be missed.
- **`chrome-devtools.service` has `Restart=no`** and died silently for three
  hours. It needs `Restart=on-failure` and probably a `MemoryMax=`, or it will
  be dead the next time the cookie needs refreshing.

### 2026-08-22 — caught up by hand, cleanly

Both surfaces were fetched manually the next day, on the owner's call, at
**roughly double the configured caution**: `--sleep-request 12 20`,
`--sleep 5 10`, `--rate 500K`, versus the defaults of 6-10 / 3-6 / 1M.

| run | result |
|---|---|
| stories, 6 profiles | 16 media, +21 files, **0 400s, 0 429s** |
| posts+reels, 12 sources, `--abort 50` | 26 media, +64 files, **0 400s, 0 429s** |

So the 400s really were the challenge state and nothing more: once the
interstitial was dismissed in the browser, the same endpoints served normally.
The stories that looked lost — `official_artms`, `0ct0ber19`, `cher_ryppo`,
`kimxxlip`, `zindoriyam` — were all captured before expiry.

**This does not retire the warning.** Two hand-paced runs a day later are not
evidence that the previous cadence was safe; they are evidence that the
account still works. What actually changed the request cost is `--abort 50`:
twelve sources across six profiles, `official_artms` included at 1829 posts
and 781 reels, finished in minutes for a few dozen requests where the old
behaviour would have spent ~400.

Note the gap this leaves: **the scheduled `profiles` mode still uses the default
6-10s pacing**, not the 12-20s used here. Reconcile that before re-enabling
the timers, or the automation will be less careful than the hand runs that
followed a warning.

## When a run fails

Worked through on 2026-08-22. Do these **in order** — the first two cost no
Instagram requests, and the third costs one page view.

### 1. Read the error, not the exit code

```sh
L=~/gdl/logs/$(ls -1t ~/gdl/logs | head -1); grep -vE '^(profiles|surfaces|sources|pacing|staging|publish|read )' "$L" | head -40
```

Two results are **not** failures, despite how they look:

| looks like | actually |
|---|---|
| `rsync error: some files/attrs were not transferred (code 23)` and `done; 1 step(s) failed` | the `chown` to `rslsync` failing because the ssh user is not root. Data landed. Confirm by re-running the rsync with `--dry-run`: an empty file list means everything arrived |
| a source reporting `No results` | that profile simply has no active story / no highlights |

### 2. Check the cookies before concluding you are blocked

Free, and it separates a local fault from a server-side one:

```sh
PATH=$HOME/.local/bin:$PATH gallery-dl \
  --cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools" \
  --cookies-export /tmp/ck.txt
grep instagram /tmp/ck.txt | awk '{print $6, length($7)}'   # names + value lengths
rm -f /tmp/ck.txt
```

A healthy `sessionid` is ~77 chars, printable ASCII, colon-delimited, and
unexpired; `ds_user_id` should be an 11-digit number. Garbage or non-printable
values mean Chrome's cookie decryption failed locally — not that the account is
in trouble. **Never print the values into a transcript or a bug report.**

### 3. Look at the account in the browser

The session lives in a dedicated Chrome on `mattellite`, on display `:1` with
CDP on 9222, run by a systemd unit:

```sh
export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user status chrome-devtools.service
systemctl --user start  chrome-devtools.service    # it has Restart=no
```

Then VNC to `:1` and open `instagram.com`. What you are looking for:

- **`/accounts/scraping_warning/`** — *"We suspect automated behaviour on your
  account"*. This is what `400 Bad Request` on `/api/v1/feed/reels_media/`
  actually means; the session is in a challenge state, not banned. Dismissing
  it in the browser restores API access immediately, verified 2026-08-22.
- a checkpoint or login page — the session is gone; re-log in the browser.
- a normal feed — the fault is elsewhere.

**Do not dismiss a warning and immediately resume.** The dismissal fixes the
symptom. The behaviour that caused it is the thing to change.

### 4. Stop, if it was a 429 or a repeated 400

```sh
systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
```

The documented escalation is CDN 429 → 400 on a stories/highlights endpoint →
suspension. It has now run twice, and both times the 400 was the last warning
before something worse.

## Reels: the API is blocked, scrape by scrolling instead

As of 2026-08-26/27, `gallery-dl`'s dedicated reels extractor fails on every
profile with `HTTP redirect to home page` — confirmed hours apart, with a
freshly-warmed session and a correct `X-IG-WWW-Claim` header (that was the
first suspect; ruled out by tracing the raw HTTP exchange). It is not a
scraping-warning interstitial — the reels tab loads completely normally in a
real, already-signed-in browser — so this is Meta blocking the specific
`/api/v1/clips/user/` endpoint gallery-dl calls, not an account-health issue.
Posts, highlights and stories are unaffected; only reels breaks.

`scripts/reels-scrape.py` works around it by never calling that endpoint:
it drives the same signed-in Chrome via its loopback CDP port (`:9222`,
already exposed for MCP automation), scrolls the reels tab like a person
would, and scrapes `/reel/<code>/` links out of the rendered page. It only
finds shortcodes — nothing is downloaded until the fetch step — and dedupes
against the **whole archive**, not just the profile being scraped (see why
below), so re-running it costs nothing for reels already held anywhere.

### The easy way: `reels-sync.sh`

```sh
ssh mattellite
~/gdl/reels-sync.sh zindoriyam
# or a full URL: ~/gdl/reels-sync.sh https://www.instagram.com/zindoriyam/
```

Runs both steps (scrape, then fetch + publish whatever's new) with the same
hand-paced settings `gdl-cron.sh` uses, logs to `~/gdl/logs/reels-<profile>-*`,
and exits 0 with "no new reels" printed when a profile is already caught up
— `gdl-sync.py` never even gets invoked in that case. Override pacing the
same way as `gdl-cron.sh` (`GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE`), plus
`GDL_SCROLL_PAUSE` and `GDL_MAX_IDLE_ROUNDS` for the scrape step. Staging
(`~/gdl/staging-reels-<profile>`) and the scraped URL list
(`~/gdl/<profile>-reels.txt`) are wiped at the START of the next run, not
after — left behind for inspection, same as `gdl-cron.sh`'s `staging-*`.

Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the
page, 10 new ones fetched and published cleanly on the first run.

**Not wired into `gdl-cron.sh` or the timers.** Both scripts drive your
actual browser session rather than firing a background request, and are
slower by design (real scrolling, not an API call) — both good reasons not
to run this unattended without deciding that deliberately. Today it is a
per-profile, by-hand tool only.

### What it's doing, if you want to run the two steps separately

```sh
PROFILE=someuser

# 1. Scrape by scrolling; dedupe against the archive; write new URLs to a file.
#    Needs gallery-dl's own pipx venv python -- that's where websocket-client
#    (the one extra dependency this needs) got injected.
~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/reels-scrape.py \
  --profile "$PROFILE" \
  --index https://instaarchive.ergosteur.com \
  --out ~/gdl/"$PROFILE"-reels.txt

# 2. Fetch whatever's new -- dry run first, same as any other gdl-sync.py call.
cd ~/gdl && PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
  --publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
  --staging ~/gdl/staging-reels-scraped \
  --sleep-request 12 20 --sleep 5 10 --rate 500K \
  --post-urls-file ~/gdl/"$PROFILE"-reels.txt --dry-run
# ...then swap --dry-run for --execute once the plan looks right.
```

Notes:

- `reels-scrape.py` fails fast if Chrome/CDP isn't up
  (`curl -s http://localhost:9222/json/version` to check first).
- If it finds nothing new, `gdl-sync.py --post-urls-file` refuses to run
  ("no usable URLs") rather than doing nothing quietly — expected when a
  profile is already caught up.
- `--max-idle-rounds` (default 3) and `--scroll-pause MIN MAX` (default
  `2.0 3.5`) are worth raising for an unusually large or slow-loading reels
  tab; the default stops once 3 consecutive scrolls find nothing new.

### Why dedup checks the whole archive, not just the scraped profile

First version deduped only against the scraped profile's own directories.
Re-running it on `zindoriyam` minutes after a successful fetch found "9 new"
reels again — all reposts/collabs originally by `0ct0ber19` and
`official_artms` (also tracked profiles). The *old*, broken direct-reels-tab
fetch had left orphaned sidecar-only remnants (`.json`/`.txt`, no media) for
them, misfiled under `zindoriyam - reels/` with the true owner's name baked
into the filename stem — `index_existing()` correctly does not count a
sidecar-only entry as "held", so they looked new. `gdl-sync.py` re-fetched
them, filed them correctly under `{username}` (the post's *true* owner, per
its own metadata) — where they already existed from that profile's own
regular sync — and `rsync --ignore-existing` silently skipped every one, so
nothing was lost or duplicated. But 9 Instagram requests were spent finding
that out. Since a shortcode is globally unique, `reels-scrape.py` now checks
every archived profile's listing, not just the one being scraped — all local
requests to the viewer's own API, never to `instagram.com`, so checking all
16 profiles costs nothing on the budget that actually matters. Confirmed
fixed the same day: re-running against `zindoriyam` immediately afterward
found "26 already archived, 0 new" and exited clean.

## What changed on 2026-08-20

One session, three separate pieces of work. Recorded because the reasons are
not recoverable from the diffs.

**The sync run.** First incremental fetch in four days: 184 new media, 299
files published, 0 failures, 0 CDN 429s. 20 story items, which are the part
that could not have been recovered later. Cost about half what it would have,
because the archive DB was already seeded and the state file was primed by hand
so no probe passes ran.

**`--abort 50`.** The skip-archive suppresses *downloads*, which spends the
CDN; it does nothing about the *listing pass*, which spends `instagram.com` and
scales with how big a profile is rather than how much is new. Measured from
sidecar write times mid-run: 3 new posts took ~100s each, the other 2272 were
written in one second. Enumerating `cher_ryppo` fell from 2151 posts to 7.

**The repo split.** `main` is public and now carries none of the fetching
tooling, no host details, and no real account names — its entire history was
rewritten, the GitHub repo deleted and recreated to clear force-push residue,
and 22 container images pruned from ghcr because the server bundle had been
shipping source comments naming real accounts. This branch holds everything
that was removed. See the caution at the top.

**Automation.** mattellite got a key on the NAS, closing the last manual step,
and three systemd timers now run the sync unattended.

## Outstanding

State as of 2026-08-20, after the sync run and the repo split. Nothing here is
broken; these are decisions not yet made and cleanups not yet done.

### Fetching

- **The timers are DISABLED** after the 2026-08-21 scraping warning; see that
  section. Their one unattended firing did the wrong thing — it skipped every
  source on the 20h floor and reported success — so re-enabling should wait on
  the four fixes listed there, not just on the account settling.
- `mattellite`'s `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys` for
  `agentapi` (added 2026-08-20, alongside the workstation's existing key), so
  the fetch host publishes straight to the archive and no `sshpass` step is
  needed. **That key is what makes the timers work** — remove it and every
  scheduled run will fetch successfully and then fail at publish.

- **5.4 GB of stale staging on `mattellite`** (`~/gdl`, 44 GB free): `staging`
  and `out` at 2.2 GB each from 2026-08-17, `staging-0820` / `out-0820` at
  446 MB each, plus `staging-full` (78 MB) and `staging-stories` (66 MB) from
  2026-08-22. All of it was verified published, so all of it is safe to delete.
  Staging is wiped per-run by `gdl-cron.sh`, but the `out-*` publish targets and
  anything created by a direct `gdl-sync.py` call are never cleaned up.
- **Stories currently depend on someone remembering.** The daily timer exists
  but is disabled, so the one surface that cannot be backfilled has no
  automation behind it. Every day nobody runs `gdl-cron.sh stories` is a day
  of stories gone. That is the central unresolved tension: the cadence that
  protects stories is also the most machine-like pattern here.
- ~~The scheduled modes have not been reconciled with the pacing used by
  hand.~~ **Done 2026-08-22**: all modes now pass `--sleep-request 12 20
  --sleep 5 10 --rate 500K`, `stories` uses `--min-interval 8`, and a
  fully-skipped stories run exits 75 with a warning instead of looking like a
  success. The timers are still **disabled** — enabling them is a separate
  decision about cadence, not about pacing.
- **`--abort 50` is opt-in.** `gdl-cron.sh profiles` passes it and the manual
  runs used it; `full-sweep` deliberately does not. It stops noticing **edited
  carousels** (test case 15), which only a full enumeration finds — which is
  what `full-sweep` is for. No cadence for it was ever agreed, and at ~420
  requests it is the riskiest thing on the schedule; prefer running it by hand.
- **The `seeded` flags in `<db>.state.json` were hand-written**, reconstructed
  from the 2026-08-17 log rather than derived from the archive DB. They assert
  "the skip-archive already knows this source". If `artms.db` is ever rebuilt,
  moved or lost, **clear the state file too** — otherwise those sources will
  never re-seed and a fetch into empty staging re-downloads everything.
- **`~/gdl/gdl-sync.py` on the fetch host is a copy, not a checkout.** It
  currently matches this branch (`96e5694e…`), but nothing keeps them in sync;
  `scp` it after any change and re-check the hash.
- The 2026-08-20 run is split across two logs — `artms-run3.log` (12 sources,
  no abort) and `artms-run4.log` (12 sources, `--abort 50`) — because it was
  stopped midway to pick up the new flag.

### Repo and infrastructure

- **The `pre-rewrite-*` branches on gitea hold the unredacted history** — real
  account names, the fetch host's IP, and the tooling, as it was before the
  rewrite. They are deliberate backups. Decide whether they expire; the
  `pre-push` hook does cover them (it allows only `main` and tags to GitHub).
- **The `pre-rewrite-full.bundle` backup is in a session scratchpad** and will
  be deleted with it. If a durable backup outside gitea is wanted, move it now.
- **Only one container image exists.** 22 versions were pruned, so rolling back
  to an older release means checking out its tag from gitea and pushing that
  tag to GitHub to rebuild it — the old images are gone, not archived.
- **CI warns that the Node 20 actions are deprecated.** `actions/checkout@v4`,
  `docker/login-action@v3`, `docker/metadata-action@v5` and
  `docker/build-push-action@v5` are being forced onto Node 24. They work today;
  bump when convenient.
- **`review-fixes`** on gitea is a stale v1.3.0-era branch, never merged,
  published only because the whole local repo was pushed. Probably deletable.
- GitHub Actions run history was lost when the repo was recreated. Cosmetic.